[SLUG] htdig or other searchers

[David <david@xxxxxxxxxxxxx>]

I've just attempted to rebuild an htdig database after an extensive 
rebuild of a web site. htdig is only finding 20 documents, although there 
are actually about 300 linked html documents in the site.

Does htdig go to the server (thereby parsing includes?). The site 
navigates perfectly from a browser.

Any suggestions about where I'm going wrong would be appreciated.

Second question... is there anything around that is better than htdig?


many thanks

David.