- To: Xiaolu Zhang <xiaolu@xxxxxxxxxxxxxxxxxxx>
- Subject: Re: [SLUG] apache access log -- "GET /robots.txt HTTP/1.0" 404 284
- From: Matt Allen <matt@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>
- Date: Tue Sep 4 12:28:01 2001
- Cc: "Slug (E-mail)" <slug@xxxxxxxxxxx>
- User-agent: Mutt/1.2.5i
That one is a search engine spidering your site.
robots.txt can contain rules that tells the search engine what to look
at and what not to look at.
google it.
Matta
On Tue, Sep 04, 2001 at 12:18:46PM +1000, Xiaolu Zhang wrote:
> how about this one, does it mean anything ?
>
> xiaolu
>
> 199.172.149.144 - - [03/Sep/2001:19:22:14 +1000] "GET /robots.txt HTTP/1.0"
> 404 284
> 216.239.46.76 - - [03/Sep/2001:20:53:13 +1000] "GET /robots.txt HTTP/1.0"
> 404 284
> 208.219.77.29 - - [04/Sep/2001:05:43:10 +1000] "GET /robots.txt HTTP/1.1"
> 404 296
> 216.35.116.91 - - [04/Sep/2001:10:36:31 +1000] "GET /robots.txt HTTP/1.0"
> 404 284
--
Matt Allen
Technical Director
Investigation Marketplace
0413 777 771
matt@xxxxxxxxxxxxxxxxxxxxxxxxxxxx
SLUG Mailing List Archives