First of all, have you checked scrapy (http://scrapy.org) ? if not, you should, it’s a very powerful framework, but in my case and unfortunately i have found some drawbacks with Scrapy which lead me to create Crawly, which are:
Because i love micro-frameworks (Flask VS Django) and because i believe that
Inside every large, complex program is a small, elegant program that does the same thing, correctly -- Tony Hoare
And because i wanted to fix all the problems listed above without having to dig in Scrapy, and when i estimated the cost of digging into scrapy and the cost of me creating a new crawler library and what i will gain, well guess what ?!
IMHO, a crawler library should (not in order of importance):