Re: comments in robots.txt - bug in RobotRules.pm??
David L. Sifry (david@sifry.com)
Wed, 29 Jan 1997 08:30:43 -0800
Martijn Koster wrote:
>
> Ehr... why? If you do a GET with relevant If-modified-since and Accept
> headers you achieve the same with a single transaction.
I haven't found a server yet that discriminates by client Accept:
header. All the servers I've tested look at the GET request only and
then send the data whether or not it fits the profile you ask for. With
limited bandwidth, doing a HEAD first saves me a LOT of network
congestion, because my spider only indexes documents of type text/html
and text/plain.
I've tried using the Accept: header in the GET on a number of well used
servers, including Netscape-Enterprise, NCSA, Cern, and Apache
(different versions).
Dave
--
Dave Sifry http://www.sifry.com
President, Sifry Consulting (408) 471-0667 (voice)
david@sifry.com (408) 471-0666 (fax)