Problems with Parallel User Agent

• Messages sorted by: [ date ][ thread ][ subject ][ author ]
• Next message: Aaron Aw: "HELP: HTTP proxy authentication"
• Previous message: BAZLEY Sebastian: "RE: Questions with HTTP Get and LWP"

Hello !

I have problem with PUA 2.42. Why script below returns 'Broken pipe' (on
Linux RedHat with newest version of libraries from CPAN). First part
generates list of webservers visited by polish infoseek...


require LWP::Parallel::RobotUA;
use HTTP::Request;
require WWW::RobotRules::AnyDBM_File;

 require LWP::UserAgent;
 $ua = new LWP::UserAgent;

 $root = "http://infoseek.icm.edu.pl/root.html";

 $request = new HTTP::Request('GET', $root);
 $response = $ua->request($request); # or


$content = $response->content;
$content =~
s/(<\s*(a|frame)\b[^>]+\b(?:href|src)\s*=\s*)(["']?)([^>\s]+)\3/push(@urls,$
4)/gie;
$protocol = 'http://';

my $rules = new WWW::RobotRules::AnyDBM_File 'Spy','cache';
my $pua = new LWP::Parallel::RobotUA ("Spy",'kelcon@search.pl',$rules);


$pua->duplicates(1);  # ignore duplicates
$pua->timeout   (5);  # in seconds                       
$pua->delay     (2);  # Powinno byc                      
$pua->redirect  (1);  # follow redirects
$pua->max_req   (1);  # max parallel requests per server 
$pua->max_hosts(30);  # max parallel servers accessed    


foreach (@urls)
  {
  if ((/$protocol/) && ($ile < 300))
    {
    $pua->register(HTTP::Request->new('GET',$_));
    }
  }

my $entries = $pua->wait();


Pozdrawiam,
  Lukasz Keler +48 604 640 124

• Next message: Aaron Aw: "HELP: HTTP proxy authentication"
• Previous message: BAZLEY Sebastian: "RE: Questions with HTTP Get and LWP"