# # ROBOTS.TXT - Robot exlusion file. This is a standard ruleset for robots, # spiders, web bots, and crawlers that adhere to the robots # exlusion standard. Using this file you can tell specific # robots what they can or cannot scan on your website. The # rules provided below are a sampling that apply to most of # our users. You can change these rules as needed to match # your wishes. # # For further details and explanations of the robot exclusion # standard, visit http://www.robotstxt.org. # # ver 1.1 12-22-2002 WMC Added: additional image wildcards # Rules for ALL robots to follow: # User-agent: * Disallow: /test.html Disallow: /admin/ Disallow: /cgi-bin/ Disallow: /images/ Disallow: /cgi-img/ Disallow: /*.gif$ # Wildcards may not be recognized by all robots Disallow: /*.jpg$ Disallow: /*.png$ Disallow: /*.psd$ Disallow: /*.bmp$ Disallow: /*.dat$ Disallow: /*.inc$ Disallow: /*.css$ Disallow: /*.js$ # Rules specifically for the Google image indexer # User-agent: Googlebot-Image Disallow: / # Disallow the entire site. Helps protect your images. # Rules specifically for EmailSiphon # User-agent: EmailSiphon Disallow: / # Disallow the entire site. Protects you from unwanted UCE.