Re: comments in robots.txt - bug in RobotRules.pm??
Gisle Aas (aas@bergen.sn.no)
26 Jan 1997 14:29:37 +0000
Andrew Daviel <andrew@andrew.triumf.ca> writes:
> I've got robots.txt like:
> User-agent: *
> Disallow: /cgi-bin/
> # Disallow: /cgi-pub/
> Disallow: /try/
>
> where /try/ is not parsed (i.e. the robot is hitting it). Seems to me
> that this is a bug....
True. It will be fixed in the next release.
> Also, I think in LWP4 that getting robots.txt was a "freebie" (not counted
> against waiting time), but in LWP5 it isn't free. Since $ua->host_wait
> returns zero for an unvisited site, this is somewhat irritating and
> I've changed it (if host_wait > 0, go somewhere else first).
I don't follow you. Care to explain it once more?
Regards,
Gisle