- To: Silly Linux Users Group <slug-chat@xxxxxxxxxxx>
- Subject: Re: [chat] Re: [SLUG] apache access log -- "GET /robots.txt HTTP/1.0" 404 284
- From: Jamie Wilkinson <jaq@xxxxxxxxxxxxxx>
- Date: Thu Sep 6 10:29:03 2001
- Reply-by: Sun Sep 9 10:24:07 EST 2001
- Reply-to: Silly Linux Users Group <slug-chat@xxxxxxxxxxx>
- User-agent: Mutt/1.3.20i
This one time, at band camp, Peter Hardy wrote:
>Does wget respect the contents of robots.txt?
You snipped the part of Andre's reply where he said yes. And as others have
said, you can tell wget to ignore robots.txt too.
--
jaq@xxxxxxxxxxxxxx http://spacepants.org/jaq.gpg
<Balial> This port may thing it's fortified, butt I seem to be mounting
a pretty good assault
SLUG Mailing List Archives