Crawler Laravel Package

Product Decisions This Supports

• Build vs. buy for link auditing / site-integrity tooling. spatie/crawler collapses what would otherwise be a multi-week internal project (concurrent HTTP fetching, URL queueing, internal-link scoping, redirect handling, robots.txt, depth limits, throttling) into a Composer dependency. The decision becomes "do we build a crawler or wire up callbacks?" — a very different cost curve.
• Feature scoping for SEO / content-health surfaces. Concretely enables: broken-link reports, redirect-chain cleanup (redirectHistory(), wasRedirected()), orphan-page detection (foundUrls() vs. sitemap), internal linking depth analysis, and sitemap-driven discovery (parseSitemaps(), including the namespaced-sitemap fix in 9.4.2). These are shippable features, not research.
• Roadmap sequencing for QA and release safety. fake() makes crawl logic testable in CI without network calls; depth(), limit(), concurrency(), and delay() make a smoke-crawl job cheap and deterministic. That means "crawl the staging site after deploy and fail the pipeline on 5xx or broken internal links" can be scheduled as a small, low-risk roadmap item rather than an infrastructure initiative.
• Migration and launch verification. Post-replatform or CMS migration, internalOnly() + foundUrls() gives a quick inventory comparison against the pre-migration URL set — a decision-grade artifact for "are we safe to 301 and decommission?"
• Investing in JavaScript-rendered sites or not. The package supports a driver-based JavaScript renderer (Browsershot/Chrome, plus a CloudflareRenderer), so the PM decision "do we support auditing SPAs?" has a known, costed path rather than an open question. It's opt-in via suggest, so the decision is reversible per environment.
• Operational control decisions. shouldStopCallback(), FinishReason (Completed, CrawlLimitReached, TimeLimitReached, Interrupted), and AdaptiveThrottle/FixedDelayThrottle mean you can decide on crawl budgets, graceful shutdown behavior, and politeness policy as product/config choices instead of bespoke engineering.
• Use cases worth validating first: internal link/SEO audits, docs coverage checks, broken-link monitoring for customer-facing properties, competitive/market site inventories (respecting robots.txt and terms), and content-inventory feeds for search or internal tooling.

When to Consider This Package

Consider it when most of the following hold:

• Your stack is PHP/Laravel (the package carries Illuminate/Laravel support and fits existing Composer, queue, and scheduler infrastructure). If your team is Python/Node-first, the integration tax usually outweighs the API ergonomics.
• The crawl target is a single site or a bounded set of internal links, not the open web. internalOnly(), includeSubdomains(), and matchWww() are the design center.
• You need batch/CLI execution with callbacks — onCrawled, onFailed, onWillCrawl, onFinished — rather than a managed, always-on crawling service.
• You want testability as a first-class requirement (fake()) and a permissive license (MIT) with no per-page or per-crawl fees.
• You need moderate concurrency (Guzzle promise pool) and want to keep operational surface area small.
• JavaScript rendering is occasional, not the default path — Chrome/Browsershot per page is real infrastructure cost and a deployment dependency.

Look elsewhere when:

• You need a distributed, resumable, horizontally scaled crawler. This is a single-process library with an in-memory ArrayCrawlQueue; persistence, sharding, and cross-worker coordination are yours to build.
• You need continuous scheduled monitoring with alerting, history, and dashboards out of the box. That's a SaaS category (Screaming Frog CLI, Ahrefs, a hosted crawler) or a much larger internal build; this package is the engine, not the product.
• Your pages are heavily client-rendered and high-volume. You'll end up operating headless Chrome at scale — evaluate a dedicated rendering service instead.
• You need structured data extraction, NLP, or content classification at scale; crawling is the easy 20% and the extraction pipeline is the real work.
• You're mid-cycle on a major version upgrade. v9 was a breaking rewrite (string URLs, CrawlResponse, CrawlProfile as interface, startCrawling() → start(), Browsershot no longer required, UriInterface removed). Budget real migration effort if you're on v8.
• Ecosystem signals are thin for your context. Despite ~2.8k stars, the dependents count reported here is 0 — validate real-world usage and your own support path (Spatie's docs and changelog are strong; that's the mitigation).

How to Pitch It (Stakeholders)

For executives (one paragraph): We can add site-crawling capability — broken-link reports, redirect cleanup, internal-link and content inventories, post-migration URL verification — using a mature, MIT-licensed Laravel package instead of a multi-week build. It's actively maintained (frequent releases, documented, changelogged), has strong brand recognition in the PHP ecosystem (~2.8k GitHub stars), and integrates with the stack we already run. Main costs are operational rather than licensing: crawl budgets, politeness/throttling, and optional headless-Chrome infrastructure if we choose to audit JavaScript-rendered pages. It is an engine, not a monitoring product — we should scope the first release to one or two concrete reports rather than promise a platform.

For engineering (the short version): Crawler::create($url)->internalOnly()->depth(3)->foundUrls() is the whole onboarding. Under the hood it's Guzzle concurrency with a clean observer model: onCrawled(), onFailed(), onWillCrawl(), onFinished(), plus a CrawlResponse object exposing status(), body(), dom(), header(), transferStats(), and redirect history. v9 added what we'd otherwise have to write: fake() for network-free tests, FinishReason for explicit termination semantics, shouldStopCallback() for graceful external stops, retry(), stream() for memory-bounded runs, and FixedDelayThrottle/AdaptiveThrottle for politeness. Scope helpers (allowCrawl/neverCrawl, alwaysCrawl), request config (basicAuth(), token(), proxy(), cookies(), middleware()), and a JavaScriptRenderer interface mean it adapts to our auth and rendering constraints without forking. Two things to plan for: queue persistence is ours to implement if we need resumability, and the v9 upgrade is breaking — pin and migrate deliberately. Recommended first slice: a scheduled CLI job that crawls the production domain, flags non-2xx internal links and redirect chains, and writes results to our own storage.