Tugger the SLUGger!SLUG Mailing List Archives

Re: [SLUG] apache access log -- "GET /robots.txt HTTP/1.0" 404 284


That one is a search engine spidering your site.

robots.txt can contain rules that tells the search engine what to look
at and what not to look at.

google it.

Matta

On Tue, Sep 04, 2001 at 12:18:46PM +1000, Xiaolu Zhang wrote:
> how about this one, does it mean anything ?
> 
> xiaolu
> 
> 199.172.149.144 - - [03/Sep/2001:19:22:14 +1000] "GET /robots.txt HTTP/1.0"
> 404 284
> 216.239.46.76 - - [03/Sep/2001:20:53:13 +1000] "GET /robots.txt HTTP/1.0"
> 404 284
> 208.219.77.29 - - [04/Sep/2001:05:43:10 +1000] "GET /robots.txt HTTP/1.1"
> 404 296
> 216.35.116.91 - - [04/Sep/2001:10:36:31 +1000] "GET /robots.txt HTTP/1.0"
> 404 284

-- 
Matt Allen
Technical Director
Investigation Marketplace
0413 777 771
matt@xxxxxxxxxxxxxxxxxxxxxxxxxxxx