dirbot-mysql
Scrapy project based on dirbot to show how to use Twisted's adbapi to store the scraped data in MySQL.
by rmaxPython
Commit activity
Last 12 weeks · 0 commits
Community health
1 of 6 standards met
28
✓README○License○Contributing○Code of Conduct○Issue Template○PR Template
Recent PRs & issues
Quiet · 3 discussions · Last activity 12y ago
See all on GitHub →
scrapy list Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.5/bin/scrapy", line 11, in sys.exit(execute()) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/cmdline.py", line 108, in execute settings = get_project_settings() File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/utils/project.py", line 60, in get_project_settings settings.setmodule(settings_module_path, priority='project') File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/settings/init.py", line 285, in setmodule self.set(key, getattr(module, key), priority) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/settings/init.py", line 260, in set self.attributes[name].set(value, priority) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/settings/init.py", line 55, in set value = BaseSettings(value, priority=priority) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/settings/init.py", line 91, in init self.update(values, priority) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy/settings/init.py", line 317, in update for name, value in six.iteritems(values): File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/six.py", line 581, in iteritems return iter(d.items(**kw)) AttributeError: 'list' object has no attribute 'items'
I am using your code snippet and getting next error: Maybe you know how can i solve this?
Not so much of an issue, but more a question about how to best begin getting data into the environment. I'm looking at getting a scraper / pipeline set up to begin populating the DB. From looking at the schemas, it looks like roxy_ui is for scraped data whereas memex_uncharted is more for rules and context. Is this correct? Any advice would be much appreciated!
Recent fixes
I was trying the MySQL pipeline and It wasn't working for me. In the process_item you pass as first parameter the item and second the spider, and scrapy(at least 0.14) expect : item, spider. http://doc.scrapy.org/en/0.14/topics/item-pipeline.html#process_item def process_item(self, item, spider): instead of def process_item(self, spider, item): Anyway, thanks for share this code.
Hi, its not working anymore. Traceback (most recent call last): File "./scrapy-ctl.py", line 7, in execute() File "/usr/lib/python2.6/site-packages/scrapy/cmdline.py", line 143, in execute _run_print_help(parser, _run_command, cmd, args, opts) File "/usr/lib/python2.6/site-packages/scrapy/cmdline.py", line 88, in _run_print_help func(_a, _kw) File "/usr/lib/python2.6/site-packages/scrapy/cmdline.py", line 150, in _run_command cmd.run(args, opts) File "/usr/lib/python2.6/site-packages/scrapy/commands/crawl.py", line 46, in run spider = self.crawler.spiders.create(spname, opts.spargs) File "/usr/lib/python2.6/site-packages/scrapy/command.py", line 34, in crawler self._crawler.configure() File "/usr/lib/python2.6/site-packages/scrapy/crawler.py", line 44, in configure self.engine = ExecutionEngine(self, self._spider_closed) File "/usr/lib/python2.6/site-packages/scrapy/core/engine.py", line 63, in __init__ self.scraper = Scraper(crawler) File "/usr/lib/python2.6/site-packages/scrapy/core/scraper.py", line 66, in init* self.itemproc = itemproc_cls.from_crawler(crawler) File "/usr/lib/python2.6/site-packages/scrapy/middleware.py", line 50, in from_crawler return cls.from_settings(crawler.settings, crawler) File "/usr/lib/python2.6/site-packages/scrapy/middleware.py", line 29, in from_settings mwcls = load_object(clspath) File "/usr/lib/python2.6/site-packages/scrapy/utils/misc.py", line 40, in load_object raise ImportError, "Error loading object '%s': %s" % (path, e) ImportError: Error loading object 'googledir.pipelines.FilterWordsPipeline': No module named exceptions
change "def process_item(self, spider,item)" to "def process_item(self, item, spider)" otherwise the pipeline won't work.
Structured data for AI agents
Repository: rmax/dirbot-mysql. Description: Scrapy project based on dirbot to show how to use Twisted's adbapi to store the scraped data in MySQL. Stars: 117, Forks: 52. Primary language: Python. Languages: Python (100%). Open PRs: 0, open issues: 3. Last activity: 12y ago. Community health: 28%. Top contributors: pablohoffman, rmax, dangra, akshar-raaj, stvnwlsn, stav.