Authenticating large, dynamic posts
steve.kilbane@ind.alstom.com
Sat, 29 May 1999 10:51:16 +0100
I'm posting large files (15MB upwards) with UserAgent, on an NT 4 box. The
normal posts were hanging, consuming all CPU time, and I suspect that NT was
having problems with the memory management, so I've been trying to use the
DYNAMIC_FILE_UPLOAD option. The problem is that the post is to a server that
requires authentication.
The request() function in LWP::Protocol::http gets off about four or five chunks
of the file, before the server responds with an Authentication required message,
which also closes the TCP connection. $socket->syswrite() barfs on the next
chunk.
I tried working around this, by just skipping out of the loop on syswrite()
error, in the hope that the next part of the function would pick up the
response, and realise auth was needed, but the sysread() for fetching the
response also failed ("Invalid argument") - I had at least expected any incoming
data to still be available.
My apologies if this has all been sorted in other places; for internal reasons,
my perl is quite old (5.004_01), and I'm working with the Nov'98 release of
libwww, but I couldn't see anything that addressed this issue in the recent
releases.
It looks to me like there are two things that need to be done to fix this:
- the chunking loop needs to check for incoming responses before fetching and
sending more data;
- the closure needs to be able to handle a rewind to the start, or perhaps an
instruction that no more data is required. Or perhaps http.pm needs to buffer
everything it's read so far. yuk.
I've been wondering about intermediate solutions as well as more correct ones.
It seems that if you know that authentication is going to be demanded, then
trying to send a big file is a bad idea; it would make more sense to send, say,
1 byte, just for the look of the thing, to give you a chance to go around the
authentication loop.
Comments appreciated.
steve