Parallel User Agent *GET* speed accelerate?

• Messages sorted by: [ date ][ thread ][ subject ][ author ]
• Next message: Marc Langheinrich: "Re: Parallel Spidering tool..."
• Previous message: Michael Belt: "Saving and depositing cookies"

Thanks To marc's example, I got PUA to work for my URL Spider.

The problem is, it doesn't seem to work in parallel through the loop. My
old SOCKET and lynx methods seemed just as fast.=20

I'm trying to read 887 URLS and grab the titles. I thought PUA would read
through a bunch at a time, but the way I have it set up, it just reads one
at a time.

Below my code is a sampling of my results with the amount of seconds to
retrieve each. Is there something missing in the way my script is written,
or is this performance the best I can expect?

Any help in creating a timeout gate would also be appreciated.

Thanks

---- CODE ----

use LWP::Parallel::UserAgent;

   $ntime =3D time();

foreach $line(@udata){
(@fld) =3D split(/\t/,$line);
$url =3D $fld[1];
#####################
   $pua =3D new LWP::Parallel::UserAgent;
   $rest =3D $pua->register(HTTP::Request->new(GET =3D> $url));

my $entries =3D $pua->wait;
foreach (keys %$entries) {
   $res =3D $entries->{$_}->response;
   $pagebody =3D ($res->content) ? $res->content : $url;
   $pagebody =3D~ s/<[^>]*>//g;
   $pagebody =3D~ s/\s+/ /g;
   (@words) =3D split(/ /,$pagebody);
   $bod =3D @words;
   $num++;
   $otime =3D $ntime;
   $ntime =3D time();
   $dif =3D $ntime - $otime;
   $title =3D ($res->title) ? $res->title : $url;
   $curcont =3D "$bod:$title";
   print "\n$dif seconds\n<$num><A HREF=3D\"" . $url . "\">" . $curcont .
"</A>\n";
}
}

---- RESULTS ----
75 seconds
<90><A HREF=3D"http://www.gallaudet.edu/~rgpricke/coda/">
169:http://www.gallaudet.edu/~rgpricke/

3 seconds
<91><A HREF=3D"http://www.dpa.org.sg/DF/">
1002:Project HIIT: Internet for the Hearing Impaired</A>

1 seconds
<92><A HREF=3D"http://www.hearinghealthmag.com/">
8:Hearing Health Magazine</A>

4 seconds
<93><A HREF=3D"http://www.hsdc.org/">
392:http://www.hsdc.org/</A>

0 seconds
<94><A HREF=3D"http://www.healthtouch.com/level1/leaflets/ninds/ninds017.htm=
">
1093:Healthtouch - Dizziness From Brain & Nerve Damage & Systemic
Illnesses</A>

1 seconds
<95><A HREF=3D"http://users.aol.com/inventmd/bppv.html">
2857:Canalith Repositioning</A>

0 seconds
<96><A HREF=3D"http://www.bp.sissa.it/cochlea/index.html">
161:Cochlea's Home Page</A>

108 seconds
<97><A=
 HREF=3D"http://curriculum.qed.qld.gov.au/lisc/articles/hi/hiinfo46.htm">
940:LIU information sheet on hearing impairment: The signing student in the
regular</A>

106 seconds
<98><A=
 HREF=3D"http://curriculum.qed.qld.gov.au/lisc/articles/hi/hiinfo45.htm">
1063:LIU information sheet on hearing impairment: Hearing loss and self
esteem</A>

----- END ----




	Andrew Deal
=B0`=B0=BA=A4=F8,=B8=B8,=F8=A4=BA=B0`=B0=BA=A4=F8,=B8=B8,=F8=A4=BA=B0`=B0=BA=
=A4=F8,=B8=B8,=F8=A4=BA=B0`=B0=BA=A4=F8,=B8=B8,=F8=A4=BA=B0`=B0=BA=A4=F8,=B8=
=B8,=F8=A4=BA=B0`
CGI Productions         FULL SERVICE INTERACTIVE ONLINE DEVELOPMENT =20
310-829-1557   mailto:deal@cgipro.com  -  Perl, JavaScript, Commerce
=AB=A4=BB=86=AB=A4=BB=A7=AB=A4=BB=86=AB=A4=BB=A7=AB=A4=BB=86=AB=A4=BB=A7=AB=
=A4=BB=86=AB=A4=BB=A7=AB=A4=BB=86=AB=A4=BB=A7=AB=A4=BB=86=AB=A4=BB=A7=AB=A4=
=BB=86=AB=A4=BB=A7=AB=A4=BB=86=AB=A4=BB=A7=AB=A4=BB=86

• Next message: Marc Langheinrich: "Re: Parallel Spidering tool..."
• Previous message: Michael Belt: "Saving and depositing cookies"