http://example.com/robots.txt (stored at the root of a website)
Notes from my MediaWiki days:
Blocking via robots.txt is an optional affair. Use some sort of mod_rewrite rule to actually block.
User-agent: Openbot Disallow: / User-agent: msnbot Disallow: / User-agent: Slurp # Crawl-delay: 20 Disallow: / # For people who have their wiki in /, prevent all honourable bots from bothering with edit pages, search pages, etc. User-agent: * Disallow: /index.php Disallow /Special: Disallow /Special%3A