Re: [chat] Re: [SLUG] apache access log -- "GET /robots.txt HTTP/1.0" 404 284

[Terry Collins <terryc@xxxxxxxxxx>]

Andre Pang wrote:
>  i think that all web search engines respect the
> robots.txt file.

I missed your humour tag. Unfortunately, many search engines ignore
robots.txt, which is why there was a discussion on poison and similar
revenge tactics in the archives. A good example of ones that ignore it
are the ones harvesting email addresses.


-- 
   Terry Collins {:-)}}} Ph(02) 4627 2186 Fax(02) 4628 7861  
   email: terryc@xxxxxxxxxx  www: http://www.woa.com.au  
   Wombat Outdoor Adventures <Bicycles, Books, Computers, GIS>

 "People without trees are like fish without clean water"