Re: 2 RobotUA problems
Otis Gospodnetic (otisg@panther.middlebury.edu)
Sat, 7 Dec 1996 14:02:38 -0500 (EST)
> > I mailed this list how on one machine a script that uses RobotUA keeps
> > asking for robots.txt over and over, even though I use AnyDBM_File.
> > It turned out that the problem is not in the script, nor in that
> > machine where machien was running nor the fault of the HTTP server the
> > script was accessing. Accessing other Web servers didn't show this
> > problem, and accessing the same Web server from a different machien
> > also didn't cause multiple fetches of robots.txt.
> > So, I guess the problem happens only when I access that one particular
> > server from that one particular machine.
> > I figured out that when the script fetches robots.txt it assigns the
> > rules a fresh_until date in past (less seconds than the amount of
> > seconds at the moment of fetching).
> > eg:
> > No visits: 2
> > Last visit: 849908242
> > Fresh until: 849908022
> > So it keeps refetching robots.txt and it is always about 219 seconds
> > too 'young'
> > I looked at RobotRules.pm but I haven't figured out how this could be
> > fixed nor what's causing this.
> > Maybe somebody else knows what's going on ?
>
> Maybe the clock on this machine is not very accurate. Perhaps it
> thinks it is in another timezone?
well, that is true, this machine's clock is off by about 1 hour (it's 1 hour
fast). I thought of that, but I then thought that incorrect time can't be a
problem because everything works fine when getting documents from other
machines (in different time zones). After a bit of thinking I think I
understand why incorrect time causes problem when fetching documents from
servers in the same time zone and not while requesting files from other
servers in different time zones.
So, I guess, yes, that's it.
Lots of redundant words here, heh...
Thanks
Otis