LWP:RobotUA: Persistent robots.txt objects or databases

• Messages sorted by: [ date ][ thread ][ subject ][ author ]
• Next message: Jason S. Priebe: "Re: LWP-5.04 LinkExtor"
• Previous message: Peter Seibel: "Re: Exception handlers [was: Traps for module writers]"

Has anyone out there implemented a client/server type database for the
RobotUA class?  I'm thinking along these lines:

A robots.txt database that held recently loaded robots.txt files with a
server program that managed requests of the form: 

client: GET hostname		server: OK (sends robots.txt file)
			or	server: NOFILE (if no robots.txt exists)
and

client: PUT hostname EXPIRES date (robots.txt file)	server: OK

Doing the client/server stuff is easy - what I'd like to do is interface
it into the RobotUA class, or make it a subclass of RobotUA,
PersistentRobotUA or something like that.

Having multiple client processes each with a PersistentRobotUA object
would then further reduce load on the net as they could all share the
same robots.txt database.

Any help or pointers would be appreciated!

Dave
-- 
Dave Sifry 				President, Sifry Consulting
david@sifry.com				http://www.sifry.com
(408) 471-0667 (voice)			(408) 471-0666 (fax)

• Next message: Jason S. Priebe: "Re: LWP-5.04 LinkExtor"
• Previous message: Peter Seibel: "Re: Exception handlers [was: Traps for module writers]"