possible RobotUA bug?

• Messages sorted by: [ date ][ thread ][ subject ][ author ]
• Previous message: Martijn Koster: "Re: libwww-5.02 problem"

Hi,

I'm using LWP 5.05 and it appears that some undesired behavior happens
with LWP::RobotUA when attempting to access a web server that
is on a port other than 80.  It appears that RobotUA will 
request robots.txt every time it does a request for a resource on the web 
server.  If I use the same code on a web server on port 80, it works fine:
it requests a robots.txt and never requests it again.

Example:

http://foo.com/robots.txt
http://foo.com/1.html
http://foo.com/2.html
http://foo.com/3.html

http://bar.com:8008/robots.txt
http://bar.com:8008/1.html
http://bar.com:8008/robots.txt
http://bar.com:8008/2.html
http://bar.com:8008/robots.txt
http://bar.com:8008/3.html

Maybe this isn't a bug... but I couldn't find any mention of this behavior
in the robot exclusion standards.  I also happen to be using a proxy
server between the LWP client and the web server, but I doubt that it has
any influence in RobotUA's strange behavior.

Is this a bug?  Or a feature?  Can you duplicate this behavior
on your system?  Comments appreciated.

Regards,
Clinton

• Previous message: Martijn Koster: "Re: libwww-5.02 problem"