Crawler Laravel Package

Technical Evaluation

Architecture fit. spatie/crawler is a standalone, framework-agnostic PHP library, not a Laravel package in the conventional sense: there is no service provider, facade, config file, or publishable migration. That is a feature, not a gap — it means the crawl engine composes cleanly into a Laravel app as a library invoked from an Artisan command, a queued Job, or a scheduled task, with no bootstrapping ceremony and no coupling to the framework's container. For a Laravel/PHP stack the fit is high: it is same-vendor as much of the Spatie ecosystem already in use, MIT-licensed, Composer-installable, and its extension points (observers, custom UrlParser, custom CrawlQueue, JavaScriptRenderer) are interface-based and injectable.

Integration feasibility. Straightforward. The API surface is fluent and small (Crawler::create()->internalOnly()->depth()->onCrawled()->start()), and the package ships two capabilities that materially de-risk adoption:

• ->fake([...]) allows deterministic unit tests of crawl logic with zero HTTP, which fits existing PHPUnit/Pest pipelines.
• ->shouldStopCallback() plus the FinishReason enum and CrawlProgress counters give us external stop control and terminal-state reporting — the two things missing from most crawler libraries and the usual cause of unkillable long-running jobs.

The main integration seam to design is persistence: the library emits results to observer callbacks and returns CrawledUrl objects; it does not store anything. Our value-add is the observer implementation that writes to our models/queue.

Technical risk.

• Execution model vs. Laravel request lifecycle. Crawling is a long-running, blocking PHP process using Guzzle promises for intra-process concurrency. It has no business running inside an HTTP request; it must run in CLI/worker context. This is a design constraint, not a defect, but it dictates the deployment topology.
• State is in-memory by default. The default queue tracks visited URLs in process memory. There is no resumable, distributed crawl out of the box; a crashed process loses progress unless we implement a persistent CrawlQueue.
• JS rendering has heavy external dependencies. The README advertises JavaScript-rendered site support, but spatie/browsershot is now a suggest (v9), not a requirement. Enabling it means Node, Puppeteer, and a headless Chrome binary in the runtime image — a significant infrastructure and image-size commitment. Default to off; treat as an explicit opt-in decision.
• Version drift. v9 is a major rewrite with substantial breaking changes (string URLs instead of UriInterface, CrawlResponse instead of PSR-7, CrawlProfile became an interface, startCrawling() removed). A large fraction of tutorials and Stack Overflow answers target v8. Pin the major and treat the UPGRADING guide as required reading.
• Legal/compliance and SSRF surface. A crawler is an outbound request engine pointed at attacker-influenceable URLs. If crawl seeds ever derive from user input, we inherit SSRF risk (internal IPs, metadata endpoints). Robots.txt handling exists but does not equal policy compliance.

Key questions before committing.

1. Is this for crawling our own properties (SEO/QA/link-integrity) or third-party sites? The answer changes the legal review, the politeness budget, and whether SSRF hardening is mandatory.
2. Do targets require JavaScript rendering? If yes, scope the Chrome/Puppeteer infrastructure cost now.
3. What is the realistic crawl envelope — domains, depth, page count, frequency? This determines whether single-process in-memory crawling is sufficient or we need a persistent queue.
4. Do we need resumability and horizontal scale, or is "rerun from scratch nightly" acceptable?
5. Who consumes the output — a database table, an event stream, a diff against a previous crawl? Persistence design is our largest piece of custom work.
6. Why is the composite score only ~52.6? The opportunity sub-score (41) suggests limited strategic differentiation. This is a solid utility, not a platform bet; size the investment accordingly.

Integration Approach

Stack fit. Native Laravel/PHP fit with a deliberately thin footprint. Requirements are PHP 8.x and Guzzle 7 or 8 (v9.4.0+ added Guzzle 8 support, v9.4.1 fixed Guzzle 8 exception handling — pin ^9.4.1 at minimum). No Laravel version lock in v9; the package does not depend on illuminate/* at all, so Laravel 11/12/13 compatibility is a non-issue, which removes the usual upgrade-treadmill worry. Existing Spatie tooling in the codebase reduces vendor sprawl and review friction.

Migration path. Greenfield — nothing to migrate. The only migration-shaped risk is a future major bump from 9.x to 10.x, which given the project's history (v9 was a full rewrite) will likely be breaking. Mitigate by isolating the package behind a thin internal service class (SiteCrawler) that owns all direct Crawler:: calls; swapping majors then touches one file. Do not let Spatie\Crawler\* types leak into controllers, models, or domain services.

Compatibility. Two friction points with the surrounding stack:

• The package uses Guzzle directly, bypassing Laravel's Http client. Existing global HTTP middleware, retry policy, and Http::fake() in tests will not apply. Use the package's fake() for crawler tests and keep Guzzle-level configuration (->middleware(), ->retry(), ->proxy()) explicitly in the wrapper.
• Observers run inside the crawl loop. Anything slow or failure-prone in the observer (DB writes, event dispatch, external calls) directly throttles the crawl. Buffer and batch-write rather than inserting per page.

Sequencing. Recommend a four-stage ramp, each stage independently shippable:

1. Spike (1 week). Prototype against one owned property using foundUrls(); validate depth, internalOnly() + includeSubdomains() + matchWww() scoping, and robots.txt behavior. Confirm the output shape we actually need.
2. CLI command (1–2 weeks). Wrap in an Artisan command with typed options, ->limit(), ->depth(), ->delay(), ->userAgent(), and a shouldStopCallback reading a stop flag (cache/DB) so an operator can halt a runaway crawl. Emit FinishReason and CrawlProgress to logs. This is the minimum viable production surface.
3. Persistence + scheduling. Implement observers writing to crawl_runs / crawled_urls tables with batched inserts; schedule via Laravel's scheduler with overlap protection (withoutOverlapping()). Add diffing against the prior run for link-integrity and redirect-drift reporting.
4. Scale-out (conditional). Only if volume demands it: implement a persistent CrawlQueue (Redis/DB) and shard by hostname across workers. Defer until stage 3 data proves the need — the package's single-process model may well suffice.

Operational Impact

Maintenance. Favourable profile: actively maintained by Spatie with a documented changelog, tagged releases, and a steady cadence of small patch releases (v9.3.x–v9.4.2). Recent entries are reassuringly boring — sitemap namespace parsing, Guzzle 8 exception handling, skipping pcntl signal registration when those functions are disabled, closing response body streams after processing. The last item matters operationally: it is an explicit memory-leak fix, evidence the maintainers are tending production concerns. Budget for routine minor upgrades each quarter; budget a real project for any major.

Support. Community-only via GitHub issues, no SLA, no vendor contract implied by the MIT license. For a non-critical internal tool that is fine. If crawl output becomes a customer-facing feature or a compliance control, the lack of commercial support must be explicitly accepted by someone with authority, or backstopped by an internal owner who can read the source — which is small and readable.

Scaling. The honest constraint: one crawl = one PHP process, concurrency inside that process via Guzzle promises, visited-URL state in memory unless replaced. Vertical scaling (more memory, more concurrency, tuned delay()) is easy; horizontal scaling requires a custom CrawlQueue and a partitioning strategy, which is real engineering we would own entirely. Practical guidance: for dozens of sites at shallow depth on a nightly cadence, a single worker suffices; for thousands of domains or deep crawls, plan for the persistent-queue workstream up front rather than discovering it under load.

Failure modes to instrument.

• Memory exhaustion on large crawls — enforce hard ->limit() caps and alert on process RSS.
• Runaway duration — ->limit(), ->depth(), an execution time limit, and the stop-flag callback; map FinishReason to distinct alert severities (CrawlLimitReached/TimeLimitReached are warnings, not successes).
• Worker termination mid-crawl — SIGINT/SIGTERM handling exists but pcntl is often unavailable in containerized or restricted environments (v9.3.2 patched a crash here). Verify signal behavior in the actual runtime image, not on a dev machine.
• Rate limiting and blocking — 429/403 storms, Cloudflare challenges. AdaptiveThrottle and retry() exist; use a descriptive, contactable userAgent() and honour robots.txt.
• Redirect loops and chains — redirectHistory()/wasRedirected() make these observable; cap redirects.
• Target-site breakage — a crawl failure must never be indistinguishable from "site has no links." Distinguish crawlFailed from empty results in reporting.
• Silent partial crawls — the worst outcome is a green run that crawled 40% of a site. Compare urlsFound vs urlsCrawled per run and alarm on significant divergence from the trailing baseline.

Ramp-up. Low for any mid-level Laravel developer — the fluent API and fake() testing hook mean a productive first command inside a day, production-grade integration inside a sprint. The genuinely non-obvious knowledge is concentrated in three places: the v9 breaking-change surface (docs lag), memory/state semantics of the default queue, and safe observer design. Capture those three in an ADR and a runbook and the bus factor is manageable.