Re: New response code
BearHeart / Bill Weinman (bearheart@bearnet.com)
Mon, 19 Feb 1996 12:06:44 -0600
At 08:36 am 2/19/96 -0800, Shel Kaphan spake:
>Some applications would generate pages differently if they are being
>probed by a robot. For instance, in applications that use URL
>So, I'd like to propose that robots be allowed to identify themselves
>as such by including a simple header line in requests, which ought to
>be passed along to CGI programs. The header could just be "robot: true"
I don't see that this is necessarily an issue for HTTP. If you
are interested in pursuing it, I would suggest that you take it up
with the Robots mailing list. Here's their subscription info:
"To subscribe to this list, send a mail message to
robots-request@webcrawler.com, with the word subscribe
on the first line of the body."
If all those folks find they have a pressing need for a new
header field in HTTP, you may then have the necessary leverage over
here. OTOH, I think your problem could easily be solved if the
robot authors would agree to put the word "ROBOT" somewhere in their
User-Agent headers. It seems just as effective as creating a new
header field, and you would need their cooperation either way.
+--------------------------------------------------------------------------+
| BearHeart / Bill Weinman | BearHeart@bearnet.com | http://www.bearnet.com/
| Author of The CGI Book -- http://www.bearnet.com/cgibook/