scrapy-redis

Implements #310 Changes now accepts (unchanged behavior) or (priority-ordered) Multi-key mode: consumes from highest-priority non-empty queue first Drains current queue for throughput; optional for periodic preemption Idle detection considers all configured keys Keys are templated, deduplicated, and validated Full test coverage with (no real Redis required) Closes #310

Summary Adds comprehensive developer/agent guide to the repository, covering everything needed for efficient contribution: AGENTS.md — Main guide at repo root covering: Project architecture (RedisMixin, RedisSpider, RedisCrawlSpider, Scheduler, etc.) Multi-key mode documentation (PR #311 feature) Quick start, testing with FakeRedisServer, tox matrix Pre-commit hooks, Sphinx docs build CI/CD workflows and debugging PR workflow (branch naming, commit conventions, co-author credit, squash merge) Security practices and dependency management Links to supplementary docs docs/coding-standards.rst — Python coding standards: Style guide (PEP 8, 120 chars), linting config Type hints for public APIs, NumPy/Google docstrings Best practices (isinstance guards, backward compatibility, logging, context managers) docs/security-guide.rst** — Security guide: Bandit scanning, Dependabot, dependency security Redis TLS/SSL and authentication configuration Input validation and request fingerprinting Known concerns (pickle serializer, Redis exposure, multi-tenant isolation) Closes #310

Hi, I’d like to propose optional support for prioritized multiple Redis input keys in . Currently, assumes a single . In some production use cases, work is split across multiple Redis queues with different priorities, and users need to process the highest-priority non-empty queue first without running multiple spiders or maintaining custom overrides. Proposed behavior: allow to be either a string or an ordered list of strings preserve current behavior when it is a string in list mode, consume from the highest-priority non-empty queue optionally add a setting so the spider can periodically check whether a higher-priority queue has received new items This would be fully backward compatible and opt-in for multi-key users only. If this direction fits the project, I’d be happy to prepare a PR.