Roy T. Fielding,
Maintaining Distributed Hypertext Infostructures:
Welcome to MOMspider's Web.
At the present time, the World-Wide Web is experiencing phenomenal growth. Our ability to sustain that growth will depend a great deal upon the manageability, and thus the maintainability, of the infostructures which make up the web. Existing maintenance methods are inadequate to support large infostructures, particularly when they span distributed sites or multiple owners. Failure to address this maintenance problem could result in the collapse of large portions of the Web and considerably reduce its usefulness for serious applications.
The Multi-Owner Maintenance spider (MOMspider) alleviates this maintenance problem by automating those tasks which are most tiresome for human maintainers. MOMspider can periodically traverse a list of webs, check each web for any changes which may require its owner's attention, and build a special index document that lists out the attributes and connections of the web in a form that can itself be traversed as a hypertext document. Moreover, it does so in an efficient manner by sharing information across maintenance tasks, minimizing the use of network bandwidth, and allowing complex restrictions to be placed on its operation.
This paper has described the requirements for automated support of distributed hypertext maintenance and how those requirements have influenced MOMspider's design. The MOMspider program, including source code, will be made freely available at the following distribution sites:http://www.ics.uci.edu/WebSoft/MOMspider/
ftp://liege.ics.uci.edu/pub/arcadia/MOMspider/
The future applications for maintenance tools like MOMspider will depend on the availabilty of document metainformation. For instance, the index that MOMspider generates is not significantly different than that used to index sites for search engines like ALIWEB [Koster94b] and Archie [ED92] -- all that needs to be added is the specific metainformation that is useful to those engines. Similarly, since MOMspider already does the work of traversing an infostructure, it should be possible to generate graphical depictions of that structure for use by alternative modes of web navigation.
As the Web continues to grow, more applications will be found for programs that can properly traverse infostructures without becoming a burden on limited network and server resources. It is hoped that MOMspider will serve as an example of how such a web-roaming robot should be designed.
[Continue to Acknowledgements or Up to Contents]