Web crawler for pubic HTTP fileservers