User-agent: * #direct all spiders Disallow: /mychat/ Disallow: /cgi/ Disallow: /zips/ User-agent: Roverbot Disallow: / User-agent: Acme.Spider Disallow: / User-agent: WebCopy Disallow: / User-agent: Teleport Pro/1.2 Disallow: / User-agent: Offline Explorer/1. Disallow: / User-Agent: WebStripper/1.41 Disallow: / User-Agent: WebZIP/4.0 Disallow: /