Re: comments in robots.txt - bug in RobotRules.pm??

Martijn Koster (m.koster@webcrawler.com)
Wed, 29 Jan 1997 06:16:02 -0800 (PST)


At 11:51 PM -0800 1/28/97, David L. Sifry wrote:

>I agree with you.  I do a HEAD request before any GET to check MIME type
>and timestamp,

Ehr... why? If you do a GET with relevant If-modified-since and Accept
headers you achieve the same with a single transaction.


-- Martijn

Email: m.koster@webcrawler.com
WWW: http://info.webcrawler.com/mak/mak.html