Laravel Laravel Package

Getting Started

Install and confirm the runtime pair. 3.2.0 is the line that adds Laravel 12 / PHP 8.4 support, so pin it explicitly if you're on Laravel 12:

composer require roach-php/laravel:^3.2 --with-all-dependencies
php artisan about        # Laravel + PHP versions

If resolution fails, composer why-not roach-php/laravel 3.2.0 names the conflicting constraint immediately — almost always an older laravel/framework or PHP floor.

Publish the config. Use the provider form so you don't have to guess tags:

php artisan vendor:publish --provider="RoachPHP\Laravel\RoachServiceProvider"

This gives you config/roach.php — the two things that matter day-to-day are the Guzzle client options (timeouts, headers, proxies, handler) and the extensions/middleware list. Everything tunable lives there or in the spider class.

Scaffold your first spider.

php artisan roach:spider BlogSpider     # writes app/Spiders/BlogSpider.php
php artisan list roach                  # see exactly what run/list commands this version ships

A minimal spider:

namespace App\Spiders;

use Generator;
use RoachPHP\Http\Request;
use RoachPHP\Http\Response;
use RoachPHP\Spider\BasicSpider;

class BlogSpider extends BasicSpider
{
    public array $startUrls = ['https://example.com/blog'];

    public function parse(Response $response): Generator
    {
        foreach ($response->filter('article h2 a') as $node) {
            yield Request::get($node->getAttribute('href'));
        }
    }
}

parse() yields either Request (follow it, its response comes back through parse() again) or Item/arrays (collected into the result). BasicSpider gives you $startUrls + a parseItem() hook; extend the abstract Spider when you need startRequests() or per-spider middleware.

Run it from tinker or a one-off command:

$items = Roach::startSpider(BlogSpider::class)->items();

Where to look first, in order: config/roach.php (every knob + default), the generated spider (the shape of the API you'll actually use), then the package's service provider — that tells you what's bound in the container and whether the event dispatcher is bridged to Laravel's. Skip the README's feature list; the config file is the real documentation.

Implementation Patterns

Spiders are application code, not scripts. Keep them in app/Spiders, one per site/shape, and keep them dumb: fetch, parse, yield items. No Eloquent writes, no HTTP calls to your own app, no notification side effects. Persistence belongs in an action or a job listening for scraped items — that's what makes spiders testable and re-runnable.

Never call Roach::startSpider() from a controller. It blocks the process until the crawl finishes. Always wrap it:

final class RunBlogCrawl implements ShouldQueue
{
    public int $timeout = 600;

    public function handle(): void
    {
        $result = Roach::startSpider(BlogSpider::class);
        // persist / dispatch per-item work here
    }
}

Pass the spider class string into the job (or hardcode it) — never serialise a spider instance, and never serialise SpiderResult.

Schedule the job, not the crawl.

Schedule::job(new RunBlogCrawl)->hourly()
    ->withoutOverlapping()
    ->onOneServer();

Crawls are exactly the workload withoutOverlapping() was invented for.

Compose crawls with chained requests. Yield Request objects from parse() for pagination and detail pages; carry context on the request and read it back on the response. This keeps one spider per site rather than a spider-per-page-type.

Per-spider middleware beats global config. Retries, timeouts, cookie handling and user-agent belong in the spider's middleware() so one flaky target doesn't force you to change behaviour everywhere. Reserve config/roach.php for genuinely global concerns (proxy, base client options, logging extension).

Listen to events rather than subclassing. The Laravel integration routes Roach's events through Laravel's dispatcher, so Event::listen(ItemScraped::class, ...) / SpiderFinished / failure events give you metrics, alerting and incremental persistence without touching package internals — and listeners survive minor bumps.

Persist incrementally for anything non-trivial. SpiderResult::items() holds everything in memory. For crawls over a few hundred pages, react to the item/finish events (or register a custom extension) and write as you go; keep only counters in memory.

Make crawls idempotent. Upsert on a natural key (canonical URL, external ID). Crawls get re-run, overlap, and get retried — duplicate rows are the default outcome otherwise.

Test against fixtures, not the network. Roach uses its own Guzzle client, so Http::fake() won't intercept it. Either point client.handler at a mock handler in the test environment, or run the spider against a file:///local fixture server and assert on the yielded items. Test the parse logic hard — that's where breakage actually happens.

CI matrix. PHP 8.3/8.4 × Laravel 11/12, with --prefer-lowest on one lane, and pin config.platform.php to your production version so a developer's local PHP can't satisfy a constraint your image can't.

Gotchas and Tips

Http::fake() silently does nothing. The single most common "my test hits the real internet" moment with this package. Fake at the Guzzle handler level via config/roach.php, and add a test that asserts no real requests escape.

Selector drift is your real production failure mode. Sites change markup; filter() returns an empty crawler rather than throwing, so a broken selector looks like "zero items, crawl succeeded". Assert count() on critical selectors and fail the job loudly when a page you expect to yield N links yields zero.

Relative URLs. filter()->link()/getAttribute('href') hands you /about as often as an absolute URL. Resolve through url() / UriResolver before yielding a Request, or you'll fire requests at a nonsense host.

Memory in long crawls and queue workers. SpiderResult accumulates items; queue workers don't release memory between jobs by default. Set --max-jobs / --memory on the worker and keep crawls chunked by start URL.

Config caching. php artisan config:cache bakes config/roach.php in at deploy time — a package upgrade that adds a key leaves it null until you re-cache. Deploy order: composer install --no-dev → config:cache → route:cache → view:cache. And don't call env() outside the config file; it returns null once cached.

Never vendor:publish --force over a tuned config. Publish to a scratch path, diff, keep what you want. Same discipline as any Laravel package.

PHP 8.4 deprecation noise. Implicitly nullable parameters (Type $x = null) are deprecated in 8.4 and will surface from Guzzle/DomCrawler transitively, not from this package directly. Run with error_reporting=E_ALL and grep for deprecat, and consider failOnDeprecation in PHPUnit so it doesn't become a PHP 9 wall later.

Be a good citizen. Default concurrency will hit a small site hard. Throttle via a request middleware or the Guzzle handler, honour robots.txt yourself (the package doesn't decide that for you), and set a real user agent with contact info. This is the failure mode that gets you IP-banned and your crawler's name in someone's logs.

Item payloads must be scalars/arrays. Put DTOs, not Eloquent models, into Item::from(). Models in items are slow to serialise, awkward to assert on in tests, and pull DB connections into places you don't want them.

Debugging a failing crawl: drop concurrency to 1 in the client config so stack traces map to a single request, tail storage/logs/laravel.log (the package logs through Laravel's channel), and catch the spider-failed event rather than letting it bubble into a queue retry loop. Queue retries on a crawl are usually harmful — mark the job tries = 1 and alert instead.

Extension points, most→least stable: published config → events/listeners → middleware/extensions array → container rebinding of the spider → subclassing concrete package classes. Subclassing is what breaks on 3.3.0; if you must, add a test asserting the constructor signature so a bump fails loudly.

Keep one adapter around the package. A thin App\Support\Scraper that owns every Roach:: call means a future major is a one-file change, and gives you a single place to fake in tests. Cheap now, very cheap later.