Re: possible RobotUA bug?
Martijn Koster (m.koster@webcrawler.com)
Fri, 3 Jan 1997 09:55:47 -0800 (PST)
At 3:55 AM 12/31/96, Clinton Wong wrote:
>I'm using LWP 5.05 and it appears that some undesired behavior happens
>with LWP::RobotUA when attempting to access a web server that
>is on a port other than 80. It appears that RobotUA will
>request robots.txt every time it does a request for a resource on the web
>server. If I use the same code on a web server on port 80, it works fine:
>it requests a robots.txt and never requests it again.
It must be something other than just the difference in port;
I cannot reproduce your problem when simply using a different port.
>Maybe this isn't a bug... but I couldn't find any mention of this behavior
>in the robot exclusion standards.
It should work the same as port 80, ie request http://bar:8080/robots.txt once.
>I also happen to be using a proxy server between the LWP client and the
>web server, but I doubt that it has any influence in RobotUA's strange
>behavior.
One never knows; it could be that the proxy keeps saying the document
is expired.
Are any of your machines publicly accessible, so I can try to reproduce?
Can you add a "use LWP::Debug ('+')" and show the traces?
Can you show your code?
-- Martijn
Email: m.koster@webcrawler.com
WWW: http://info.webcrawler.com/mak/mak.html