Workaround for LWP/Solaris missing EOF bug
Phil Mitchell (philip_mitchell@harvard.edu)
Tue, 30 Jan 2001 15:48:28 -0500
On Solaris, an LWP request sometimes fails (ie. reports a server error)
when the server is fine, b/c of some interaction between the OS and the
server response. (See below for more info.) In fact, LWP is receiving data,
but the data is improperly terminated and LWP times out. The workaround
that I have found is to use Telnet.pm to open a connection (say, to port
80), and do the GET by hand. Telnet.pm will also time out with these cases,
but even when it does, you can get the response by looking at input_log().
Then you will need to parse it by hand.
Seems like it would be nice if LWP exposed its input log similarly -- or
maybe it does and I just don't know about it...
More about the problem:
There is an interaction between Solaris and certain web servers (notably
Microsoft-IIS/4.0) where the response to a GET comes back improperly
terminated. You can observe this independently of LWP by doing a "telnet
www.ndu.edu 80",
and then "GET http://www.ndu.edu/ndu/inss/strforum/forum83.html"
You will see telnet receive the response (it's a 404), but then (usually
but not always) fail to terminate. As far as I know, you will only see this
on Solaris (I'm on Solaris 7, have heard similar problems w/ 6); I've tried
same under Linux and haven't seen this problem. I have seen this on a
number of other sites.
The problem seems to be that LWP reports a server error (500) b/c it times
out while waiting for the missing EOF terminator. I've turned on LWP::Debug
and see it hang, apparently in the LWP::Protocol::http->request subroutine,
or possible in collect(), though I'm not sure. You can see that LWP
received the response, so it ought to do more than just report a timeout.