Re: broken pipe during spidering
Gisle Aas (gisle@aas.no)
09 Jun 1998 13:24:28 +0200
lynx@some.other.pages.de writes:
> i was just spidering to check some urls and
> met a site which is sufficiently ill-configured
> to make LWP break out with a 'Broken pipe'
>
> the site is http://sbox.tu-graz.ac.at/
>
> a telnet looks like this:
>
> > telnet sbox.tu-graz.ac.at 80
> Trying 129.27.193.129...
> telnet: connect to address 129.27.193.129: Connection refused
> telnet: connect to address 129.27.193.130: Connection refused
> telnet: connect to address 129.27.193.131: Connection refused
> telnet: connect to address 129.27.193.132: Connection refused
> telnet: Unable to connect to remote host: Connection refused
>
> i circumvented this by hand and probably don't need a
> fix for myself now, but i thought you might be interested..
> if my version of LWP is old and yours can handle this
> please forget this mail of mine.
This happens when connect(2) itself succeeds but the server closes the
connection before LWP manage to syswrite() the request itself. Adding
$SIG{PIPE} = 'IGNORE' to your script avoid this problem.
We have not decided if it is a good idea that LWP assigns to
$SIG{PIPE} by itself. Any more thoughts on that?
Regards,
Gisle