> >I'm not sure that Visit-Time is the best way to throttle robots.
> >Request-Rate also sounds too difficult for Robots to implement unless
> >it was very simple like 1000 URLs / day, but 10 URLs / minute is too
> >fine-grained. Then, I'd question its usefulness.
>
> Agreed.
>
I don't think it's that difficult. Something like (in some pigdin
computer like language):
var docs;
var rate;
var docrate;
string rateline;
rateline = getrateline();
docs = atoi(part of rateline before "/");
rate = atoi(part of rateline after "/");
switch(suffix on rateline)
{
case 'S': case 's': break;
case 'M': case 'm': rate *= 60 ; break;
case 'H': case 'h': rate *= 3600 ; break;
case 'D': case 'd': rate *= 86400 ; break;
default: error(); break;
}
docrate = rate / docs;
This will give you the amount of time to wait before requesting a new
document from the server. But anyway ...
-spc (What about Visit-time?)
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html