Core Laravel Package

Getting Started

Roach is a Scrapy-style scraping engine for PHP built on ReactPHP and Symfony components. You define spiders (what to crawl), emit requests and items, and the engine handles scheduling, concurrency, retries, and dispatch to handlers.

1. Install and publish config:

composer require roach-php/core
php vendor/bin/roach install   # publishes config/roach.php

If there's no install command in your version, copy vendor/roach-php/core/config/roach.php into config/roach.php manually and make sure it's merged into your app config.

2. Look first at config/roach.php. Every knob lives there: concurrency, log_level, default middlewares, pipelines, handlers, and a spiders map keyed by spider class where you can override per-spider middleware/pipelines/start URLs. That file is your mental model of the runtime.

3. A minimum viable spider:

use Roach\Spider\Spider;
use Roach\Response;
use Roach\Http\Request;

class BlogSpider extends Spider
{
    public function startUrls(): array
    {
        return ['https://example.com/blog'];
    }

    public function parse(Response $response): \Generator
    {
        foreach ($response->getBody()->filter('a.post-link') as $link) {
            yield Request::get($link->getAttribute('href'));
        }

        yield [
            'title' => $response->getBody()->filter('h1')->text(),
            'url'   => $response->getRequest()->getUri(),
        ];
    }
}

4. Run it:

use Roach\Roach;

Roach::startSpider(BlogSpider::class);

Your first useful milestone is: one spider, one URL, items printed via ClosureHandler or LogHandler (set 'handlers' => [ClosureHandler::class => fn ($item) => dump($item)]). Once you see output, wire in a real pipeline.

Implementation Patterns

Generators are the contract. parse() is a generator, not a function returning data. Every yield is interpreted by the engine:

• yield Request::get($url) → new request pushed onto the scheduler.
• yield $array or yield $item → passes through pipelines and handlers.
• Nothing returned from return is used; return only ends the generator.

Because it's lazy, you can build pagination and drill-down logic in a natural loop without buffering everything in memory.

Splitting parsing across methods. Use meta to carry context (a category name, a detail-page ID) so a single spider can handle both list and detail pages:

public function parse(Response $response): \Generator
{
    if ($response->getRequest()->getMeta('type') === 'detail') {
        yield $this->parseDetail($response);
        return;
    }

    foreach ($response->getBody()->filter('.card') as $card) {
        yield Request::get($card->filter('a')->attr('href'))
            ->addMeta('type', 'detail')
            ->addMeta('category', $card->filter('.tag')->text());
    }
}

Use Item classes when schemas get real. Array literals are fine for prototypes; once you have >3 fields or multiple spiders sharing a shape, define an Item with Field + extractor definitions (ArrayExtractor, JsonExtractor, RegexExtractor, or a callback). That gives you one place to fix selectors and lets extractors handle missing/typed values consistently instead of scattering ?? null through spiders.

Middleware is where cross-cutting behavior belongs. Do not put retries, headers, user agents, or error-page detection inside parse(). Compose middlewares in config, and override per spider:

// config/roach.php
'spiders' => [
    BlogSpider::class => [
        'middlewares' => [
            SetUserAgent::class => 'MyCrawler/1.0 (+https://example.com/bot)',
            RetryRequests::class => 3,
            AddRequestHeader::class => ['Accept-Language' => 'en'],
        ],
    ],
],

Pipelines for persistence. A pipeline receives the item and returns it (or null to drop it). This is the right place for dedupe, DB upserts, and enrichment—not parse(). Pair with a handler for terminal side effects (emit to queue, log, webhook).

Extensions for lifecycle work. If you need a DB connection opened once, a progress bar, a counter, or graceful shutdown on SpiderFinished, write an extension rather than a spider base class. Extensions hook into engine events and stay out of the hot path.

Events for observability. Subscribe to request-sent / response-received / request-failed / item-scraped events on the dispatcher to emit metrics (per-domain request counts, failure rates, item throughput). This is cheaper than logging from parse().

Testing. Instantiate the spider and call parse() with a hand-built Response wrapping a fixture body and a synthetic Request. Iterate the generator and assert on yielded items/requests. Keep fixtures in tests/fixtures/—HTTP calls during tests will be slow and flaky.

Gotchas and Tips

Blocking calls kill throughput. The engine runs on a ReactPHP loop with concurrency requests in flight. Any synchronous sleep(), blocking DB driver, or non-async HTTP client inside parse() or a pipeline stalls all concurrent requests. Use async-friendly clients (or short transactions) in pipelines.

Middleware order matters and is non-obvious. Middlewares wrap the request/response cycle; a retry middleware placed before an error-handling middleware sees different results than after. Read the resolved config (config('roach.middlewares') merged with per-spider overrides) before debugging "why isn't this retried."

parse() silently doing nothing is almost always one of: forgot yield, used return $data, or the selector matched zero elements. Dump $response->getBody()->html() early and confirm you're on the page you think you are (anti-bot pages return 200 + a captcha).

Response status is not an exception by default. HTTP 4xx/5xx responses are still delivered to parse() unless you add HandleErrorResponses. Guard with a status check or install the middleware; otherwise you'll "parse" error pages into empty items.

Concurrency and politeness are a tension. High concurrency with no per-domain delay is the fastest way to get blocked. Combine a modest concurrency (2–5) with a delay middleware for production crawls; scale concurrency only on domains you control or with rotating proxies.

Memory grows with in-flight requests and item buffering. Keep item payloads small, avoid holding onto the full Response in closures, and stream to storage via pipelines rather than collecting items in an array inside the spider.

Extension points are the escape hatch. If a middleware/pipeline doesn't exist for your need (auth token refresh, rotating proxies, custom retry backoff), implement the interface rather than hacking the spider. Extensions register through config, so the same spider stays reusable across projects.

Config caching. In Laravel-style apps, config:cache will freeze roach.php; a spider's per-class overrides added at runtime won't apply. Keep spider config static and read environment values through env() at config-load time.

Debugging slow runs. Log RequestFailed events with the exception message — DNS failures, TLS errors, and timeouts surface there, not in parse(). Start with log_level at debug, one spider, one URL, then scale up.