Re: robots.txt question

• Messages sorted by: [ date ][ thread ][ subject ][ author ]
• Next message: Alex Rhomberg: "HTTP:Cookies with URI::html (incl patch suggestion)"
• Previous message: Justin Case: "Re: problems with LWP::UserAgent"
• Maybe in reply to: bobmin@nidlink.com: "robots.txt question"

* <bobmin@nidlink.com> wrote:
| Now I need to add something like this to the file, but I'm not
| sure that the protocol will support it.
|
| user-agent: zeus
| disallow: /
|
| user-agent: *
| disallow: /cgi-bin
| etc...
|
|  Will this work?

This should work if the robots conforms to the robots exclusion standard.
--
Björn Höhrmann ^ mailto:bjoern@hoehrmann.de ^ http://www.bjoernsworld.de
am Badedeich 7 ° Telefon: +49(0)4667/981ASK ° http://bjoern.hoehrmann.de
25899 Dagebüll # PGP Pub. KeyID: 0xA4357E78 # http://learn.to/quote [!]e
  "Everything we see or seem is but a dream within a dream" - E.A. Poe

• Next message: Alex Rhomberg: "HTTP:Cookies with URI::html (incl patch suggestion)"
• Previous message: Justin Case: "Re: problems with LWP::UserAgent"
• Maybe in reply to: bobmin@nidlink.com: "robots.txt question"