SLUG Mailing List Archives

Re: [chat] Re: [SLUG] apache access log -- "GET /robots.txt HTTP/1.0" 404 284

To: Peter Hardy <peterhardy@xxxxxxxxxxxxxx>, Silly Linux Users Group <slug-chat@xxxxxxxxxxx>
Subject: Re: [chat] Re: [SLUG] apache access log -- "GET /robots.txt HTTP/1.0" 404 284
From: Matt Hope <matth@xxxxxxxxxxxxxxx>
Date: Wed Sep 5 11:42:02 2001

On Tue, 04 Sep 2001, Peter Hardy <peterhardy@xxxxxxxxxxxxxx> wrote...

: Does wget respect the contents of robots.txt?

Yes, but there is an option to disable this behavour.

/dopey

Attachment: pgpOSIfNmLNXa.pgp
Description: PGP signature

References:
- Re: [chat] Re: [SLUG] apache access log -- "GET /robots.txt HTTP/1.0" 404 284
  - From: Andre Pang
- Re: [chat] Re: [SLUG] apache access log -- "GET /robots.txt HTTP/1.0" 404 284
  - From: Peter Hardy

Messages sorted by: [ date | thread ]
Prev by Date: Re: [chat] Re: [SLUG] apache access log -- "GET /robots.txt HTTP/1.0" 404 284
Next by Date: [chat] Re: [SLUG] Best Mailing List software?
Previous by thread: Re: [chat] Re: [SLUG] apache access log -- "GET /robots.txt HTTP/1.0" 404 284
Next by thread: Re: [chat] Re: [SLUG] apache access log -- "GET /robots.txt HTTP/1.0" 404 284