Re: Web crawling library proposal

[Paulino Calderon <paulino () calderonpale com>]

Hi list,

Thank you all for the feedback. You've pointed out some very important 
issues. I've started working on the following changes:
* OO design.
* crawl() must return a iterator.
* Allow multiple crawlers to run at the same time.
* Remove anchors from links by default instead of having to set the 
variable httpspider.ignoreParams.
* Queue will be removed.
* Implement a better uri filter.

Cheers!

--
Paulino Calderón Pale
Web: http://calderonpale.com
Twitter: http://www.twitter.com/paulinocaIderon

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/