(on Wikipedia)
https://web.archive.org/web/20010604045759/http://www.robotstxt.org:80/wc/robots.html
http://example.com/robots.txt (stored at the root of a website)
-
- robots.txt Validator
- the big crawl — More tips
-
https://web.archive.org/web/20201023165946/http://tool.motoricerca.info/robots-checker.phtml
Notes from my MediaWiki days:
Blocking via robots.txt is an optional affair. Use some sort of mod_rewrite rule to actually block.
User-agent: Openbot Disallow: / User-agent: msnbot Disallow: / User-agent: Slurp # Crawl-delay: 20 Disallow: / # For people who have their wiki in /, prevent all honourable bots from bothering with edit pages, search pages, etc. User-agent: * Disallow: /index.php Disallow /Special: Disallow /Special%3A
ported