Getting Started

This bundle is the Symfony integration layer for roach-php/core, a Scrapy-style async crawler built on ReactPHP and Symfony's BrowserKit/DomCrawler. Adoption is tiny (10 stars, no dependents) and the last tagged release was October 2022, so treat it as a thin convenience wrapper you may end up extending.

Minimal setup:

Install core and the bundle:

composer require nelexa/roach-php-bundle

NeLexa\RoachPhpBundle\RoachPhpBundle::class => ['all' => true],

Look first at the bundle's DependencyInjection/ folder and Resources/config/ to confirm the extension alias (typically roach_php), the tagged service IDs, and which console command runs a spider. This is the fastest way to understand what it actually wires up, since docs are sparse.

Define a settings config. The bundle exposes roach's settings array under the extension alias — check the Configuration.php tree for exact keys before guessing:

# config/packages/roach_php.yaml
roach_php:
    settings:
        DOWNLOAD_DELAY: 1
        CONCURRENT_REQUESTS: 8
        USER_AGENT: 'MyCrawler/1.0 (+https://example.com)'

Create your first spider by extending RoachPHP\Spider\AbstractSpider:

final class BlogSpider extends AbstractSpider
{
    public function name(): string { return 'blog'; }
    public function startUrls(): array { return ['https://example.com/blog']; }

    public function parse(ResponseInterface $response): Generator
    {
        yield $response->css('h2 a')->each(fn ($a) => $this->item(['title' => $a->text()]));
        yield $this->request('GET', $this->absoluteUrl($response, 'a.next')); // follow pagination
    }
}

Run it via the console command provided by the bundle (bin/console list roach will show the exact name). First use case: a single-site listing scraper where you follow pagination and emit items to a saving pipeline.

Implementation Patterns

Spiders as services, not new. Tag your spiders (or rely on autoconfiguration if the bundle implements AutoconfigureTag — verify in RoachPhpExtension). Then resolve them from the container in a command/controller so settings, middlewares, and pipelines configured in YAML actually apply.

Items as typed DTOs. Extend AbstractItem (or use Item with __set) for anything you'll persist. Define explicit public properties for the fields you scrape; this gives you clean serialization into pipelines and avoids mystery keys.

Pipelines for persistence. This is where most day-to-day value lives: a pipeline implementing ItemPipelineInterface::processItem(ItemInterface $item): ItemInterface that inserts to Doctrine, pushes to a queue, or writes NDJSON. Register as a service and tag it (roach.item_pipeline or whatever the bundle's tag constant is — confirm in the extension).

final class DoctrinePersistPipeline implements ItemPipelineInterface
{
    public function __construct(private EntityManagerInterface $em) {}

    public function processItem(ItemInterface $item): ItemInterface
    {
        $this->em->persist(Article::fromItem($item));
        return $item;
    }
}

Because items are processed per-item, wrap in a transaction if you batch-flush; otherwise rely on EntityManager flushing at crawl end (fragile) or flush per item (slow).

Middlewares for cross-cutting concerns. Use a downloader/spider middleware to add headers, rotate proxies, or skip URLs already seen. Middlewares are the right place for retry logic and rate-limiting rather than inside parse().

Settings layering. Keep the shared defaults in roach_php.yaml, but override per run by instantiating the engine with a merged settings array when the crawl is ad-hoc (e.g., a --limit console option that sets CLOSESPIDER_PAGECOUNT). Don't hardcode crawl limits in the spider class.

Composition for multi-site scrapers. Define one spider per site, share a base class with pagination helpers, and run them sequentially from a Symfony command. Roach's async engine handles concurrency within a spider; chaining spiders gives you isolation and per-site rate limits.

Testing. Unit-test parse() directly by constructing a HtmlResponse from a fixture HTML string — no network needed, and it fits Symfony's KernelTestCase if you need the crawler stack.

Gotchas and Tips

The package is stale (Oct 2022). Pin roach-php/core explicitly and check compatibility with your Symfony version before committing. If core has moved, the bundle may still work but won't get bugfixes — be prepared to fork or replace the DI wiring with a small custom extension.
ReactPHP is single-threaded and cooperative. Any blocking code — synchronous HTTP, heavy Doctrine queries per item, sleep(), large file_get_contents — stalls the whole loop. Defer I/O to pipelines or queue-based workers, and keep parse() pure.
Settings keys are constants, not free-form strings. Validate the exact keys the bundle's Configuration accepts; typos in YAML often fail at runtime, not at cache warmup, because roach's settings are a plain array.
Item yields must be ItemInterface. Returning arrays or DTOs throws late and confusingly; wrap with $this->item([...]) or your DTO extending AbstractItem.
URL relativization bites. parse() receives response objects where links are relative. Use the response's URI when building follow-up requests ($response->getUri()->resolve($href)), not raw strings.
Cache/storage config is global per bundle config, not per spider. If you need different cache stores for dev vs. prod, override via Symfony config per-environment or inject a settings override at run time.
Debugging a stalled crawl is easiest at the middleware layer and with -vvv. Log every outgoing request in a middleware; roach's console output is sparse about why a request was dropped.
Autoconfiguration gaps. If your spider/pipeline isn't picked up, it's almost always a missing tag or a private service; run bin/console debug:container --tag=... with the bundle's tag names to confirm.
Extension points to prefer when the bundle lags: ServiceCompilerPass to add tags in bulk, a custom SpiderProvider, or overriding the bundle's service definitions in your app's services.yaml — all less invasive than forking.

Roach Php Bundle Laravel Package

Getting Started

Implementation Patterns

Gotchas and Tips