Re: [SLUG] apache access log -- "GET /robots.txt HTTP/1.0" 404 284

[Tony Green <tgreen@xxxxxxxxxxxxxx>]

* This one time, at band camp, Xiaolu Zhang said:
> how about this one, does it mean anything ?
> 
> xiaolu
> 
> 199.172.149.144 - - [03/Sep/2001:19:22:14 +1000] "GET /robots.txt HTTP/1.0"
> 404 284
> 216.239.46.76 - - [03/Sep/2001:20:53:13 +1000] "GET /robots.txt HTTP/1.0"
> 404 284
> 208.219.77.29 - - [04/Sep/2001:05:43:10 +1000] "GET /robots.txt HTTP/1.1"
> 404 296
> 216.35.116.91 - - [04/Sep/2001:10:36:31 +1000] "GET /robots.txt HTTP/1.0"
> 404 284

Can I suggest this as a good starting point
http://www.google.com/search?q=robots.txt

In general, people on the mailing list will be more willing to help if
you show willing to help yourself - Google is your friend :-).

The method used to exclude robots from a server is to create a file on the
server which specifies an access policy for robots. This file must be
accessible via HTTP on the local URL "/robots.txt". 


Greeno
-- 
Greeno <tgreen@xxxxxxxxxxxxxx>
GnuPG Key :  1024D/B5657C8B 
Key fingerprint = 9ED8 59CC C161 B857 462E  51E6 7DFB 465B B565 7C8B

Imagine working in a secure environment and finding the string 
_NSAKEY in the OS binaries without a good explanation
    -Alan Cox 04/05/2001