[chat] Re: [SLUG] apache access log -- "GET /robots.txt HTTP/1.0" 404 284

[Peter Hardy <peterhardy@xxxxxxxxxxxxxx>]

...and then Matt Allen said:
> That one is a search engine spidering your site.

I was interested to note that wget also accesses robots.txt, although I
don't know if it actually acts on the contents.  Anybody?

Peter
peterhardy@xxxxxxxxxxxxxx
-- 
        You don't organise a mob, Nanny.  A mob is something that
        happens spontaneously.

        There's seventy-nine Oggs in these parts.  Spontaneous it is, then.
                -- Agnes and Nanny Ogg
                   (Terry Pratchett, Carpe Jugulum)

Attachment: pgpbEOMLGH9gG.pgp
Description: PGP signature