Re: HEAD and GET fail on http://www.suck.com/suckreviews/nph-suckblast.cgi

Gisle Aas (aas@bergen.sn.no)
Tue, 23 Apr 1996 15:40:00 +0200 

In message <199604230847.CAA01247@nelson.santafe.edu>, Nelson Minar writes:
> $ HEAD http://www.suck.com/suckreviews/nph-suckblast.cgi
> Unable to determine scheme for '1' at /usr/local/lib/perl5/site_perl/auto/URI
/URL/_generic/abs.al line 12

I guess this is the HTTP::Response bug in the base() method.  This
patch is a fix for that:

Index: Response.pm
===================================================================
RCS file: /f/stovner/utvikling/CVSROOT/aas/perl/mods/libwww-perl/lib/HTTP/Response.pm,v
retrieving revision 1.16
retrieving revision 1.18
diff -u -r1.16 -r1.18
@@ -115,12 +121,11 @@
        # Look for the <BASE HREF='...'> tag
        # XXX: Should really use the HTML::Parse module to get this
        # right. The <BASE> tag could be commented out, which we are
-       # not able to handle here.
-       $self->{'_content'} =~ /<\s*base\s+href=([^\s>]+)/i;
-       $base = $1;
-       if ($base) {
+       # not able to handle with this simple regexp.
+       if ($self->{'_content'} =~ /<\s*base\s+href=([^\s>]+)/i) {
+           $base = $1;
            $base =~ s/^(["'])(.*)\1$/$2/;  #" get rid of any quoting
-            return $base;
+           return $base;
        }
     }
     $base = $self->header('Base') unless $base;

> I confess I don't really understand what's going on here (I've been
> hacking too long tonight.) That URL does a redirect to another URL,
> but I can't determine why the redirect fails. That's the only URL I've
> seen (so far) that causes a problem.
> 
> 
> Btw, I just started playing with libwww-perl a couple of days ago.
> Great stuff! I'm writing some code for agents that live in the Web,
> this is a really nice basis for it.

Good to hear that you like it.

> Two other random questions:
> 
> Is there a way to only load the first, say, 4k of a document? I want
> to try to make a quick guess as to what the content of a page is and
> bail out after just a little bit of read. Is there a callback trick?

You might try to set up a callback using:

   $ua->request($req, sub { ..... });

and then die within that subroutine when you have seen enough.

> Dumb perl question - is there an easy way to remove duplicates in a list?

Perhaps:  @list = grep(!$seen{$_}++, @list);

Regards,
Gisle