Autonomy Agentware does web site searches based upon a specific
topic and creates copies of relevant ones on the users hard disc. It
searches by following the most relevant links first, and will
backtrack after encountering a number of irrelevant pages in a row
(the number is set by the 'Activity' control in the Agent Configure
screen). You can read the manual at
http://www.agentware.com/manual/automan.htm
I hope this is of interest. While it is not exactly an indexing
robot, there may be enough overlap to be worth looking at.
Regards,
Nick Dearnaley,
AutoNomy.
autonomy@stjohns.co.uk