robots.txt question
bobmin@nidlink.com
Wed, 6 Dec 2000 11:52:52 -0800
Does anyone know if the robots.txt file can support multiple user
agents. I've managed to identify a persistent spam bot thats been
nailing our domain pretty hard lately, and would like to disallow the
bot entirely (assuming of course, that their claim to conforming to
robots protocol is correct). Unfortunately I already have a robots.txt
file that is setup to inhibit the other spiders from indexing certain
files/directories in our site. ie.
user-agent: *
disallow: /cgi-bin
etc...
Now I need to add something like this to the file, but I'm not sure that
the protocol will support it.
user-agent: zeus
disallow: /
user-agent: *
disallow: /cgi-bin
etc...
Will this work? Or am I stuck with this blasted bot? The robots
specification isn't very clear on this subject, and the examples are
too limited. Anyone have any ideas? Thoughts? (other than being
mean and sic'ing my own spider back on them *grin*).
Thanks
Bob Minnick, Northern Webs
^------------------------------------------------------------------------^
| Bob Minnick, President, Northern Webs |
| 159 Trapps Drive |
| Sagle, ID 83860 |
| 208.265.0474 |
| http://www.northernwebs.com (homepage) |
| http://www.northernwebs.com/set/setsimjr.html (meta medic) |
^------------------------------------------------------------------------^