How to 'solve' 401 HTTP-errors?
Bjorn Hermans (bjorn.hermans@capgemini.nl)
Mon, 08 Feb 1999 12:35:46 +0100
Hi,
I have built a Web crawler based on libwww 5.36. My problem is that I
run into pages that generate a HTTP "401 Access Denied" error. I cannot
figure out why I'm getting this error?
Servers that generate this error do not (seem to) have a robots.txt
which disallows me access to the server (or parts of it). And when I
request such an URL (try http://www.leoadaly.com/ to see what I mean)
via my Web browser, all goes fine.
My second guess was that I needed to get and set some cookies, but when
I request the URL mentioned above, my browser does not report getting
any...
So, what can I do to prevent these 401 errors from happening? (i.e. what
is causing them??)
Or is this a known bug in libwww? (I doubt that)
Many thanks & greetings!
Bjorn.