Core Laravel Package

Product Decisions This Supports

• Build vs. buy for data acquisition. Roach is the "build, but not from scratch" middle path. If a roadmap item needs data pulled from third-party sites (catalog ingestion, competitor price monitoring, content aggregation, lead enrichment, SEO/QA audits), Roach replaces an ad-hoc Guzzle-plus-regex loop with a structured, Scrapy-style architecture: Spiders → Scheduler → Downloader + middlewares → Spider middleware → Item pipelines → Extensions. The decision it lets you make is "one sprint to a working prototype" instead of "one sprint to a bespoke crawler framework."
• Keeping data collection inside the PHP/Laravel stack. This is a TCO decision, not a technical taste one: no second runtime, no separate deploy pipeline, no separate on-call, no new hiring profile. That materially changes the cost side of the build-vs-buy comparison against a managed scraping vendor or a Python/Scrapy microservice.
• Centralizing crawling policy once. Concurrency limits, per-domain delays and auto-throttling, retries/backoff, User-Agent and robots.txt handling, URL deduplication, and item validation/persistence are all first-class extension points. These are policy decisions you want made in one place and reused across every crawler rather than re-litigated per script.
• Roadmap sequencing and estimation. A data-ingestion epic can be prototyped and validated in days, so it can be funded as a spike before headcount is committed. It also makes a "shared internal scraping toolkit" platform play plausible — one team's middleware benefits every team's spiders.
• Scope boundaries. It does not support decisions that depend on browser rendering, anti-bot evasion, or very high-volume distributed crawling. Those push the decision toward managed scraping vendors or a Python/Playwright service, and it's better to know that at scoping time.

When to Consider This Package

Adopt when:

• The stack is PHP/Laravel and there is no Python service or Scrapy expertise on the team.
• Workloads are I/O-bound, HTML-first, structured crawling at modest-to-moderate volume — thousands to low millions of pages, scheduled or batch, not real-time.
• You need repeatable pipelines (extract → validate → normalize → persist) and multiple crawlers sharing middleware, not a single throwaway script.
• You want scheduling, throttling, retries, and pipeline semantics without inventing and maintaining them.
• You can absorb the vendor risk: ~1.4k stars, thin maintainer bench, last release March 2025. Treat it as a dependency, not a platform — pin versions, wrap the public surface behind your own interfaces, and budget for the possibility of forking.

Look elsewhere when:

• Targets are JS-rendered SPAs or gated by bot detection. Roach ships no headless browser; bolting one on forfeits most of the downloader's throughput advantage, at which point a managed vendor (Zyte, ScrapingBee, Bright Data) or a Python/Playwright worker is cheaper.
• You need large-scale distributed crawling. Scrapy plus its ecosystem has a decade of operational knowledge and plugins; PHP's concurrency model (no true async runtime) is a weaker foundation at the high end.
• Parsing is CPU-heavy at volume — PHP workers blocked on DOM work won't overlap.
• You only need one or two simple fetches; a plain HTTP client has fewer moving parts.
• Legal/compliance posture is unsettled (ToS, robots, personal data). No library solves that — fix policy first.
• You require vendor support, SLAs, or a security-response commitment.

How to Pitch It (Stakeholders)

For executives: Roach is an open-source PHP framework for web scraping, modeled on Python's Scrapy. It lets our existing PHP team build structured, scheduled data-collection jobs — competitor pricing, catalog feeds, content and SEO monitoring — without adding a language, a service, or headcount. It supplies the plumbing we'd otherwise write and maintain ourselves: concurrency, retries, throttling, deduplication, and validated data pipelines. The first crawler is a days-long prototype, not a quarter-long platform build. The cost is a Composer dependency plus engineering ownership.

For engineering: It's framework-agnostic but built on Laravel/Symfony components. You define Spiders and Item Pipelines; middlewares handle request/response concerns; the engine handles scheduling and concurrent downloads via Guzzle promises. Familiar if anyone has touched Scrapy. Because it's a mid-size community project with a thin maintainer bench, we pin the version, keep it behind our own abstraction layer, and avoid spreading its types across the codebase.

Risk framing for both: Mid-size community, thin maintainer bench → treat it as a library we could fork if abandoned. No headless browser → not a solution for JS-heavy targets. No opinion on legality or robots.txt → that policy is ours. Recommend a bounded pilot on one low-risk target before it appears in a roadmap commitment.