"100 levels deep in subroutine calls" with multiple GETs and LWP::UserAgent

David L. Sifry (david@sifry.com)
Mon, 09 Sep 1996 20:33:28 -0700 

I'm no expert in libwww, but I'm getting stuck on an error I can't find
the cause of:

I create an instance of LWP::UserAgent (I tried RobotUA first, but the
problem presents itself in either case) in order to pull down some URLs
for my web spider.  

I then use a HTTP::Request object to pull down the URL and store it in
my UserAgent object.  The purpose behind this is not to use the
UserAgent class at all, but to use the RobotUA subclass so that I can
make sure I don't hit one site too many times and so that I can maintain
a RobotRules database, but that's besides the point for now...

Anyway, whether I have a UserAgent object or a RobotUA object, I can do
about 40-50 GETs with that object and then perl fails me, telling me
that I am "100 levels deep in subroutine calls".  I don't know the
internals of LWP well enough to go much further than that, but here is a
small program that I worte up that showcases this behavior.  Run it
under the perl debugger or you'll just SEGV fault out when perl reaches
100 subroutine calls.

My setup:

Linux 2.0.6, perl5.003, libwww-perl-5.01.  About the only thing that's
non standard is that the perl is socksified.

Here's the program I wrote that fails on iteration 44:

----------------------------------------------------------------------
#!/usr/bin/perl -w
use strict;

use URI::URL;

require LWP::UserAgent;
my $ua = new LWP::UserAgent "test/0.1" , 'david\@sifry.com';
my $res;

my $url = "http://www.aptltd.com/index.html";
my $i;

for ($i=0;$i<50;$i++) {
    $res = $ua->request(HTTP::Request->new(GET => $url));
    print "GET: ", $i, " " , $res->request->url,"\n";
}
----------------------------------------------------------------------
Is this a bug or am I doing something stupid?  I've tried it with
different URLs, to no avail.  Try it with a bad URL - it fails even
faster!  What's going wrong?  Is my request syntax incorrect?  TIA for
your help!  

Dave
-- 
Dave Sifry
david@sifry.com, sifry@aptltd.com
(408) 471-0667 (voice)
(408) 471-0666 (fax)