scrapydo

I use code: ERROR:scrapy.core.scraper:Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/scrapy/utils/defer.py", line 102, in iter_errback yield next(it) File "/usr/local/lib/python3.6/site-packages/scrapy/spidermiddlewares/offsite.py", line 30, in process_spider_output for x in result: File "/usr/local/lib/python3.6/site-packages/scrapy/spidermiddlewares/referer.py", line 339, in return (_set_referer(r) for r in result or ()) File "/usr/local/lib/python3.6/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in return (r for r in result or () if _filter(r)) File "/usr/local/lib/python3.6/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in return (r for r in result or () if _filter(r)) File "/Users/hjtian/github/courier-spiders/crawler/crawler/spiders/aftership_spider.py", line 62, in parse raise TypeError TypeError

I'm trying scrapydo and can't even get started: import scrapydo scrapydo.setup() url = "http://google.com" scrapydo.fetch(url) Traceback (most recent call last): File "", line 1, in File "/Users/john/anaconda/envs/py3/lib/python3.4/site-packages/scrapydo/api.py", line 51, in fetch return crawler.spider.response AttributeError: 'DefaultSpider' object has no attribute 'response'

Hi ! I have a question about the run_spider() method and how to use it as a generator. I am using scrapydo inside a django project, where a user can type a keyword in a form, submit and get the results from three different spiders, called by scrapydo.run_spider(). When there are many results (100+), the page is hanging for a while before displaying the results. I would like to display each item as soon as it's scraped, using StreamingHttpResponse() and a generator. The thing is, although the parse() method from spider is yielding items, I can't manage to yield items using the scrapydo.run_spider() method. I have tried to add a new signal handler with the 'item_passed' signal, but it's still not working. Here's what i tried I have added a 'yield_items' flag in the function _run_spider_in_reactor, which calls a signal handler to yield every item with the item_passed signal Then in my test view in view.py I iterate through every item returned by run_spider() and I yield every item. when I call the view on the browser using http://localhost:8000/streamer, the page hangs for 5 seconds and then displays all the results at once, just like with a regular HttpResponse. Do you have any idea of what I am doing wrong and how I could make it work ? Thanks a lot!