Re: How to 'solve' 401 HTTP-errors?

• Messages sorted by: [ date ][ thread ][ subject ][ author ]
• Next message: SalomeH@tend.net: "sending a file attachment using Net::SMTP"
• Previous message: Bjorn Hermans: "How to 'solve' 401 HTTP-errors?"
• In reply to: Bjorn Hermans: "How to 'solve' 401 HTTP-errors?"
• Next in thread: Bjorn Hermans: "Re: How to 'solve' 401 HTTP-errors?"

>>>>> "Bjorn" == Bjorn Hermans <bjorn.hermans@capgemini.nl> writes:

Bjorn> I have built a Web crawler based on libwww 5.36. My problem is that I
Bjorn> run into pages that generate a HTTP "401 Access Denied" error. I cannot
Bjorn> figure out why I'm getting this error?
Bjorn> Servers that generate this error do not (seem to) have a robots.txt
Bjorn> which disallows me access to the server (or parts of it). And when I
Bjorn> request such an URL (try http://www.leoadaly.com/ to see what I mean) 
Bjorn> via my Web browser, all goes fine.

It might be a really dumb-idea'ed server that is looking at browser
type.

For examples of webcrawlers that use LWP, look at my link verifier
scripts in <url:http://www.stonehenge.com/merlyn/WebTechniques/>.

-- 
Name: Randal L. Schwartz / Stonehenge Consulting Services (503)777-0095
Keywords: Perl training, UNIX[tm] consulting, video production, skiing, flying
Email: <merlyn@stonehenge.com> Snail: (Call) PGP-Key: (finger merlyn@teleport.com)
Web: <A HREF="http://www.stonehenge.com/merlyn/">My Home Page!</A>
Quote: "I'm telling you, if I could have five lines in my .sig, I would!" -- me

• Next message: SalomeH@tend.net: "sending a file attachment using Net::SMTP"
• Previous message: Bjorn Hermans: "How to 'solve' 401 HTTP-errors?"
• In reply to: Bjorn Hermans: "How to 'solve' 401 HTTP-errors?"
• Next in thread: Bjorn Hermans: "Re: How to 'solve' 401 HTTP-errors?"