User-agent: * Disallow: /cgi-bin/ Disallow: /privatedir/ Disallow: /_vti_pvt/ Disallow: /_private/ Disallow: /_vti_inf/ Disallow: /form2mail/ Disallow: /unused/ #Rules for Google User-agent: googlebot Crawl-delay: 60 #Rules for MSN Search User-agent: msnbot Crawl-delay: 60 #Rules for Yahoo User-agent: yahoo-slurp Crawl-delay: 60 #Rules for Ask/Teoma User-agent: teoma Crawl-delay: 60 #Rules for GigaBlast User-agent: gigabot Crawl-delay: 60 #Rules for Scrub The Web User-agent: scrubby Crawl-delay: 60 #Rules for DMOZ Checker User-agent: robozilla Crawl-delay: 60 #Rules for Nutch User-agent: nutch Crawl-delay: 60 #Rules for Alexa/Wayback User-agent: ia_archiver Crawl-delay: 60 #Rules for Baidu User-agent: baiduspider Crawl-delay: 60 #Rules for Google Image User-agent: googlebot-image Crawl-delay: 60 #Rules for Yahoo MM User-agent: yahoo-mmcrawler Crawl-delay: 60 #Rules for MSN PicSearch User-agent: psbot Crawl-delay: 60 #Rules for SingingFish User-agent: asterias Crawl-delay: 60 #Rules for Yahoo Blogs User-agent: yahoo-blogs/v3.9 Crawl-delay: 60 #Rules for All Other Engines User-agent: * Crawl-delay: 60