Core Laravel Package

Technical Evaluation

Architecture fit. Roach is a framework-agnostic scraping engine modeled closely on Python's Scrapy, not a Laravel-native package. Its primitives are Spider (crawl logic + parse method), Request/Response, Item (typed result DTO), and three middleware chains (request, response, item) plus Pipeline and Extension hooks. It sits on Symfony components (Console, EventDispatcher, DomCrawler, CssSelector) and Guzzle for transport, with its own lightweight service container. For a Laravel app that means embedding a parallel runtime rather than composer require-and-go. The abstractions map cleanly onto concepts a TPM already reasons about: Spider ↔ queued job / Artisan command, Item ↔ DTO or Eloquent model, Pipeline ↔ job chain, Extension ↔ service-provider-style cross-cutting hook. The mapping is manual but legible to any Laravel engineer.

Integration feasibility. Low-to-medium effort. Guzzle is already transitively present in a Laravel app (via the HTTP client), so the transport layer is compatible. There are two realistic integration surfaces: (1) wrap engine invocation in an Artisan command or queued job that passes a spider class plus run config, then persist results through a custom pipeline; (2) bridge containers so pipelined code can resolve Eloquent, queues, Storage disks, and config() inside Roach. Confirm first whether an official or community Laravel bridge exists before writing glue—it materially changes the estimate.

Technical risk.

• Runtime model. Concurrency is Guzzle promise / curl_multi-based, not a full event loop. Runs must happen in CLI workers, never in FPM request cycles. Long-run memory growth requires measurement and worker recycling.
• Dependency drift. Symfony and Guzzle version constraints must reconcile with the host Laravel version; pin and test on upgrade.
• No JS rendering. HTTP + DOM only. Any source requiring a headless browser is out of scope and forces a second toolchain.
• Persistence and retries are yours. No ORM, no scheduler, no durable queue semantics beyond what you configure in pipelines and middleware.
• Bus factor. ~1.4k stars, active releases (March 2025), community-supported. Acceptable for internal tooling; risky as a dependency under a revenue-critical data path without an owned fallback.
• Compliance surface. robots.txt, ToS, rate limiting, and PII handling are product risks the TPM owns—the library provides no policy layer.

Key questions.

• What is the target volume and freshness SLA per source? This drives concurrency, worker count, and storage design.
• Which sources, and do we have documented permission / robots.txt clearance?
• Do any sources require JavaScript execution? If yes, this package covers only part of the need.
• Where do items land—Postgres via Eloquent, object storage, queue, webhook?
• Does a maintained Laravel wrapper already exist that we should adopt instead of building glue?
• Who owns the "target site changed, parser broke" runbook and on what cadence?

Integration Approach

Stack fit. PHP 8.1+ on Laravel 10/11 is a natural host. Roach's Symfony dependencies typically coexist with Laravel's vendored versions; the shared Guzzle transport removes the most common conflict point. Add symfony/css-selector for CSS-based extraction. The one hard architectural boundary: no browser automation—pair with a separate headless-browser service only if a source genuinely demands it, and treat that as a distinct workstream.

Migration path.

• Phase 0 — spike (~1 week). Install in a sandbox, port one existing scraper or build one new spider, run via CLI, persist items to a scraped_items table through a custom pipeline. Proves container bridge plus persistence end-to-end.
• Phase 1 — productionize. Wrap invocation in scrape:run {spider} and/or a queued job; externalize concurrency, delay, user-agent, and proxy settings into config; add a pipeline that dispatches Laravel jobs for downstream processing.
• Phase 2 — operate. Supervision (Supervisor), structured logging, per-spider metrics (requests, items, error rate, duration), and a dead-letter path for failed items.
• Phase 3 — scale. Horizontal sharding by domain or seed list, per-domain rate-limit counters in Redis, dedupe cache, and backpressure on downstream writes.

Compatibility. Pin PHP, Symfony, and Guzzle constraints against the host Laravel version before Phase 0 completes. Isolate runs in dedicated processes if Roach installs global error or shutdown handlers—those will collide with queue-worker lifecycles.

Sequencing. Prove one spider end-to-end before abstracting middleware or pipelines. Build persistence second, observability third, scaling last. Do not parallelize spider development across sources until the operational runbook exists, or you will accumulate unowned parsers.

Operational Impact

Maintenance. Spider code is the real liability: every target-site redesign breaks selectors, often silently. Budget recurring parser upkeep per source as a standing line item. Library upgrades should be quarterly, pinned, and tested against a regression spider. Align Roach's upgrade cadence with Laravel's.

Support. No commercial backing; community issues and PRs only. Establish a single internal owner for spiders, the runner, and the runbook. For revenue-critical sources, pre-negotiate a managed-vendor fallback (Apify, Zyte, ScrapingBee) so a breakage has a known escape hatch.

Scaling.

• Vertical: raise concurrency in config, bounded by memory and target-site tolerance—not CPU.
• Horizontal: multiple workers partitioned by domain or seed list; Redis for dedupe and rate-limit counters.
• Storage: batch item writes; avoid per-item database round-trips at volume.
• Throughput is governed by politeness settings at least as much as by hardware—capacity planning must start from what the target site will tolerate.

Failure modes.

• HTTP errors, timeouts, 403s and CAPTCHAs → retry/backoff middleware plus alerting on error-rate spikes.
• Parser drift → silent data corruption is the worst case; add schema validation and canary assertions on expected item counts and required fields.
• Memory exhaustion in long runs → cap requests/items per process and recycle workers.
• Stuck or zombie runs → run-level timeouts and heartbeats.
• Duplicate ingests → idempotency keys in the persistence pipeline.
• Compliance drift → robots.txt and ToS changes; maintain a source allowlist and a kill switch.

Ramp-up. A Laravel engineer new to scraping reaches productivity in roughly 2–5 days; the Scrapy mental model transfers directly. The larger cost is the surrounding scaffolding—container bridge, pipeline design, supervision, logging, metrics—typically 2–3 weeks to a stable first production spider and closer to a quarter to a multi-source, observable, on-call-ready pipeline.

TPM recommendation. Roach is a credible, well-shaped engine for PHP teams already on Laravel, with an architecture that maps onto patterns the team already knows. It is neither a turnkey Laravel package nor a browser automation tool. Leverage it as an embedded engine behind Artisan and queue workers, invest early in persistence and observability, and treat parser maintenance and compliance as first-class product costs. The modest opportunity score (11.57) is consistent with the call: build on it where scraping is core to the product, and evaluate a managed vendor where it is incidental.