# # Module : robots.txt # Date : 9602.05 # Author : Dave Pratt - MSCI # Purpose : For those web crawlers which adhere to the standard, this # document provides a means of excluding robots from portions # of your file space. For more information on the standard # see "http://www.ch.ic.ac.uk/chemical_mime.html". The # presence of this file should knock down the error.log file # considerably. # At present, all crawler agents are allowed in and none # should traverse the tmp, icons, or lynx directories. # # User-agent: * Disallow: /