Crawler Laravel Package

Getting Started

Install (Laravel 10/11/12, PHP 8.2+):

composer require spatie/crawler

No service provider, no config file, no facade. It's a standalone library you drive from a console command or a queued job — not from a web request.

Your first crawl — a link/status audit of your own app:

// app/Console/Commands/AuditLinks.php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\CrawlResponse;

public function handle(): int
{
    Crawler::create(config('app.url'))
        ->internalOnly()
        ->depth(3)
        ->concurrency(5)
        ->onCrawled(function (string $url, CrawlResponse $response) {
            if ($response->status() >= 400) {
                $this->warn("{$response->status()} {$url}");
            }
        })
        ->onFailed(fn (string $url, $e) => $this->error("FAILED {$url}: {$e->getMessage()}"))
        ->start();

    return self::SUCCESS;
}

Run it locally against php artisan serve before pointing it at production. Depth 2–3 and internalOnly() are the two settings that keep a "quick check" from becoming a 40-minute crawl of your entire CDN.

Where to look first, in order:

1. https://spatie.be/docs/crawler — the README is a teaser; the docs site has the real API surface.
2. UPGRADING.md in the repo. v9 was a major rewrite (string URLs instead of UriInterface, CrawlResponse instead of PSR-7, CrawlProfile is now an interface, startCrawling() is gone, setBrowsershot() is gone). Virtually every blog post and Stack Overflow answer you'll find predates it. Read this file before copying anything from the internet.
3. CHANGELOG — it's unusually detailed and doubles as a feature list (matchWww(), includeSubdomains(), retry(), shouldStopCallback(), throttles, stream() all landed post-9.0).

Realistic first use cases (in increasing order of effort):

• Broken-link / 404 audit of your marketing site in a nightly scheduled command.
• Collect all internal URLs to diff against sitemap.xml (catch orphan pages and stale sitemap entries).
• Scrape a small set of pages after login for a smoke test of a third-party integration.
• Asset inventory: ->extractAll() with ResourceType to list every image/script/stylesheet referenced.

Implementation Patterns

Command with closures vs. observers

Closures (onCrawled, onFailed, onWillCrawl, onFinished) are perfect for one-off commands and scripts. As soon as you have two reasons to crawl — e.g. report 404s and collect page titles — move to observer classes and addObserver($a, $b) (variadic since 9.0). Observers are also the testable unit: instantiate one, pass a CrawlResponse, assert.

class StorePageObserver implements CrawlObserver
{
    public function crawled(string $url, CrawlResponse $response, ?CrawlUrl $foundOnUrl = null): void
    {
        if (! $response->isSuccessful() || $response->body() === '') {
            return; // non-parseable mime types arrive with an empty body
        }

        Page::updateOrCreate(['url' => $url], [
            'title' => $response->dom()->filter('title')->text(''),
            'status' => $response->status(),
            'crawled_at' => now(),
        ]);
    }

    public function crawlFailed(string $url, RequestException $e, ?CrawlUrl $foundOnUrl = null, ?TransferStatistics $stats = null): void
    {
        Page::where('url', $url)->update(['error' => $e->getMessage()]);
    }
}

The ?TransferStatistics parameter on crawlFailed() (added 9.0) is how you distinguish "connection timed out" from "server returned 500" without string-matching exception messages.

Scoping: the four knobs that matter

Crawler::create('https://example.com')
    ->internalOnly()          // same host only
    ->matchWww()              // treat www + non-www as one site
    ->includeSubdomains()     // docs.example.com counts as internal
    ->depth(3)                // 3 hops from the seed
    ->shouldCrawl(fn (string $url) => ! str_contains($url, '/logout'));

internalOnly() without matchWww() is the single most common source of "why did it only crawl one page" — you seed example.com, every link says www.example.com, and everything looks external. alwaysCrawl() / neverCrawl() are your pattern-level overrides that beat the profile, useful for whitelisting a CDN path or blacklisting /admin.

Bounded crawls: limits, time, stop callbacks

Everything long-running needs a ceiling. Use all three:

Crawler::create($url)->maxUrls(2000)->maxExecutionTime(300)   // v9: limit() / timeLimit()

Or, when the stop condition is external (a queue message, a cache flag flipped by a support engineer), shouldStopCallback() is checked before each request is scheduled — cleaner than throwing from inside onCrawled(). Check the returned FinishReason enum afterwards: Completed, CrawlLimitReached, TimeLimitReached, Interrupted are different outcomes and you want to log them differently. onFinished() hands you a CrawlProgress with urlsCrawled, urlsFailed, urlsFound, urlsPending — write those into your command output so failures aren't silent.

Politeness and reliability

->concurrency(5)                  // default is higher than you want on someone else's server
->delay(200)                      // ms between requests
->userAgent('MyApp-Audit/1.0 (+https://example.com/bot)')
->retry(3)                        // connection errors + 5xx

FixedDelayThrottle / AdaptiveThrottle exist for the case where you want the delay to react to response times. Set userAgent() before you crawl third-party sites and your own — robots.txt is fetched with your UA (fixed in 8.4.5), so the crawl respects the rules you'd be judged by.

Large crawls: jobs and queues

For anything beyond a few thousand URLs, don't run it inline:

• Wrap the command in a queued job so Horizon handles retries and logs.
• The default queue is in-memory (ArrayCrawlQueue), so a crashed worker loses state. For resumable crawls, implement CrawlQueue backed by a DB table or Redis — this is the intended extension point and it also gives you free normalization if you copy what ArrayCrawlQueue does.
• If you're on a worker with a hard timeout, maxExecutionTime() shorter than the worker timeout + SIGTERM handling gives you a clean exit instead of a killed process with a half-written table.

JavaScript-rendered sites

Rendering is driver-based in v9; spatie/browsershot is a suggest, not a dependency. So:

composer require spatie/browsershot
# plus Node, Puppeteer and a Chrome binary on the machine

Only reach for it when you've confirmed the HTML you get is empty — it's roughly two orders of magnitude slower per page and it needs binaries your production container may not have. Keep JS crawls on a dedicated queue/worker so a Puppeteer crash doesn't take down your normal jobs. CloudflareRenderer and the JavaScriptRenderer interface are there when you need a hosted or custom renderer.

Testing — this is the standout feature

it('reports 404s', function () {
    Crawler::create('https://example.com')
        ->fake([
            'https://example.com' => '<html><a href="/gone">x</a></html>',
            'https://example.com/gone' => ['status' => 404, 'body' => 'nope'],
        ])
        ->addObserver(new StorePageObserver)
        ->start();

    expect(Page::where('status', 404)->count())->toBe(1);
});

fake() turns "crawler tests" from a mocked-Guzzle nightmare into a pure fixture test. It's the reason this package is pleasant in a real codebase — your observers get genuine coverage without a network.

Gotchas and Tips

Every tutorial you'll find is for v8. UriInterface → string, ResponseInterface → CrawlResponse, startCrawling() → start(), setCrawlProfile() → internalOnly()/shouldCrawl(), setBrowsershot() gone. If a snippet doesn't compile, this is why — check UPGRADING.md rather than debugging it.

Default scheme is https as of v9. Passing example.com without a scheme used to hit HTTP; now it hits HTTPS. On an internal service with a self-signed cert, that's an instant wall of crawlFailed — use withoutVerifying() (dev only) or fix the cert.

allowedMimeTypes responses still notify observers — with an empty body. Fixed in 9.0.1, and it's a footgun: your onCrawled will fire for a PDF or a JPEG, $response->body() will be '', and ->dom() will blow up or return nothing. Always guard on body() !== '' before parsing.

Redirects are followed and tracked by default in v9 (['track_redirects' => true]). That means the URL you get in onCrawled may be post-redirect; use redirectHistory() and wasRedirected() when you're auditing redirect chains. It also means depth accounting has historically been subtle — 8.2.1 fixed depth-tree checks for redirected URLs — so if you see pages crawled beyond your depth(), you're not imagining it.

fake() is not a browser. It serves your fixtures verbatim; it won't reproduce weird encodings, malformed HTML edge cases, or JS rendering. Use it for logic, and keep one real crawl in a scheduled command or a manual pre-release check.

Long URLs and control characters. fnmatch() is skipped for URLs over FILENAME_MAX (9.3.1), and URLs with control characters are reported as malformed. If you're crawling a site that generates pathological URLs, filter them in shouldCrawl() rather than letting them into the queue.

pcntl and signals. Graceful SIGINT/SIGTERM handling is skipped when pcntl functions are disabled (9.3.2) — common in some Docker images. That's fine, but it means Ctrl-C is abrupt and your onFinished() may never run. Don't rely on onFinished() for critical persistence; write incrementally in onCrawled().

Streaming bodies. ->stream() (9.0) reduces memory for huge pages but changes how you can consume the body. Response streams are closed after processing (9.2.1) — good, but don't stash a CrawlResponse object in a property and try to read its body after start() returns. Extract what you need inside the callback.

Guzzle version. 9.4.0 added Guzzle 8 support and 9.4.1 fixed Guzzle 8 exception handling. If you're on a Laravel version pulling Guzzle 8 transitively, make sure you're on spatie/crawler >= 9.4.1 or retries will misbehave.

Custom clientOptions merge with defaults in v9 — passing options no longer silently replaces the package's own (pass null to remove one). If you're porting v8 code that set allow_redirects => false wholesale, expect different behavior.

Extension points worth knowing about before you write a workaround: a custom UrlParser (return ExtractedUrl[]), a custom CrawlQueue (resumable/DB-backed crawls), a custom JavaScriptRenderer, middleware() for Guzzle-level instrumentation, and parseSitemaps() when you'd rather seed from a sitemap than follow links. alsoExtract() / extractAll() with the ResourceType enum covers asset discovery that would otherwise be a second parser.

Operational hygiene: never seed a production crawl at / with high concurrency during business hours; the crawler is a load test you didn't schedule. Point staging first, watch CrawlProgress numbers, then promote.