Re: URI and spidering unique docs

• Messages sorted by: [ date ][ thread ][ subject ][ author ]
• Next message: Paul Maine Jr.: "LWP Header Question"
• Previous message: Stephen R. Wilcoxon: "Re: URI and spidering unique docs"
• In reply to: Bill Moseley: "URI and spidering unique docs"

Bill Moseley <moseley@hank.org> writes:

> Oh my, I'm writing yet another spider, for some reason.
> 
> I'd like to only spider documents one time.  So I'm using a hash of
> URI->canonical keys.
> 
> Although I realize these *could* be two different docs, they are not on our
> server:
>      http://localhost/path/to/my/file.html
>      http://localhost/path/to/../to/my/file.html
> 
> Any (URI?) tricks to seeing those as the same document?

Untested extra canonicalization:

my $p = $uri->path;
$p =~ s,/\./,/,;
$p =~ s,[^/]*/../,/;
$uri->path($p);

Regards,
Gisle

• Next message: Paul Maine Jr.: "LWP Header Question"
• Previous message: Stephen R. Wilcoxon: "Re: URI and spidering unique docs"
• In reply to: Bill Moseley: "URI and spidering unique docs"