What do you mean by "poorly written agent spending a lot of time
parsing the output"? Is the output simply long, or is it devious in a
way? How can it be, with the current simple specifications?
Can you give an example of such a /robots.txt file?
-- Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia --------------------------------+-------------------------------- "What is the sound of Perl? Is it not the sound of a wall that people have stopped banging their heads against?" -- Larry Wall _________________________________________________ This messages was sent by the robots mailing list. To unsubscribe, send mail to robots-request@webcrawler.com with the word "unsubscribe" in the body. For more info see http://info.webcrawler.com/mak/projects/robots/robots.html