Bug when trying to access NCBI

Brian Karlak (bkarlak@lmi.net)
Thu, 24 Sep 1998 17:10:36 -0800 

LibWWWers -

Please excuse me if this is a previously reported bug; I am new to the list
and couldn't find any list archives on the web.

Anyways, I am building a simple spider to automatically access & parse the
BLAST server at NCBI.  The server I am connecting to is a CGI that perofrms
some rather time-consuming calculations on the other side.  When accessing
the server through a browser, it usually prints out a "Your request is
being processed . . ." message and then pauses for 15-30 seconds until the
search results come back.

When I try to access the pages via libwww, however, the request is always
truncated after the "Your request is being processed . . ." message and the
results are not returned.  The code I am using to access the page is:

  my $req = new HTTP::Request "$method" => "$url";
  $req->header('Accept' => 'text/html');
  $req->content_type('application/x-www-form-urlencoded');
  $req->content($self->getArgString());
  my $res = $ua->request($req);

where $url and $self->getArgString() pass the appropriate values to access
the page.  As you can see, I am reading directly into $res without using a
filehandle or subroutine reference to process the information as it is
returned from the server.

In trying to track down what was happening, I noted that $buf in the
anonymous subroutined passed to Protocol::collect at line 236 of
LWP/Protocol/http.pm was truncated and in trying to trace what was
happening I decided to hack the $size variable by adding the line

$size = 100000 unless ($size);

after line 57 in http.pm.  Amazingly enough, this worked, although I won't
pretend to have a clue as to what's really going on in all the socket code.
My guess is that some of the socket reading code is timing out since it
can fill its buffer but no text is waiting for it afterwards.  Or
something.

So, anyways, what's up?  I think this is a bug.  The workaround that would
make sense for me (not counting the crock I've described above) is to pass
the appropriate $size variable to UserAgent::request() but this seems to
require I use a subroutine reference and I can't get that to work either (I
try passing an anonymous subroutine but my "use strict" pukes when I try to
access $buf).

Thank you for any and all input . . .

zenkat