Re: [SLUG] Search engine traffic dominates

[Mary Gardiner <mary@xxxxxxxxxxxx>]

On Thu, Oct 23, 2008, Rev Simon Rumble wrote:
> You might want to look into the Crawl-delay extension to the robots.txt 
> standard, which can limit by robot:
> http://en.wikipedia.org/wiki/Robots.txt#Crawl-delay_directive

There's also the Sitemaps protocol, in which you can suggest how
frequently content changes: http://en.wikipedia.org/wiki/Sitemaps
It doesn't seem to come with any guarentees that the robots will respect
that as a refresh frequency though.

-Mary