RE: timeout problem

Philippe CROCHAT (pcrochat@sqli.com)
Mon, 10 Jan 2000 16:35:44 +0100 

unfortunatly, nothing is printed on the screen, with the modification =
on
Parallel/Protocol/http.pm, are you sure that i should not modifie
"LWP/Protocol/http.pm instead" ?,
this is just a question.

oups, i got a broken pipe an other time, just right now.

I can't test my thousand urls with the simple LWP library because i =
don't
catch any broken pipe and must wait at least one day before all is =
finished.
The problem of the timeout however remain the same beetwen LWP and
LWP::Parallel.=20
Perhaps it comes from a lower layer ?=20

-----Message d'origine-----
De: Marc Langheinrich [mailto:langhein@inf.ethz.ch]
Date: lundi 10 janvier 2000 16:14
=C0: Philippe CROCHAT
Cc: LWP Mailing List
Objet: Re: timeout problem


On Mon, 10 Jan 2000, Philippe CROCHAT wrote:
> wahoo, i think that i should need some little more help on that =
point.
> Could you indicate me for example where i shall modifie your source =
in
order
> to make it work properly with my huge number of urls.
Mmh, so the first step to figure out what's wrong would be to =
double-check
where the program hangs. Maybe you could add a little "print" into the =
"sub
connect" routine in "Parallel/Protocol/http.pm" to get a little =
indicator
whenever PUA connects to a new site. For example like:

 sub connect {=20
    my ($self, $host, $port, $timeout) =3D @_;
    # this method inherited from LWP::Protocol::http
    print "["
    my $socket =3D $self->_new_socket($host, $port, $timeout);
    print "]"
    $socket;
 }

This way you'd be able to tell if it hangs during connecting or while
actually downloading data. Once we have that figured out we could then =
try
to find out which host it is that's causing the problem, and why our
$timeout variable gets ignored! Which version of IO::Socket are you =
using?


> no, in fact it's happening all the time. I never succeed in fetching =
all
my
> urls (around 2000) in one pass with your library.
Hmm, maybe you could try fetching those 2000 URL's just once using =
regular
LWP and see if there's a "broken pipe" as well? Or try to find out =
which
line issues the "Broken pipe" and insert some print right in front of =
it in
order to find out the host the program tries to connect to or download
from...


-m
--=20
Marc Langheinrich
marclang@cs.washington.edu