vdb/php-spider
A configurable PHP web spider/crawler with a modular architecture for scraping and traversing sites. Define start URLs, URI filters, processors, and post-processors to control crawl depth, skip duplicate pages, and extract or analyze content with custom callbacks.
Architecture fit. vdb/php-spider is a framework-agnostic PHP crawling library, not a Laravel package. It has no service provider, facade, publishable config, or container bindings; it operates as a stateful Spider object composed with a PSR-18 HTTP client, a PSR-7 message factory, a URI queue (SplQueue/custom), and a symfony/event-dispatcher observer pipeline. A Laravel codebase therefore cannot composer require and inherit idiomatic ergonomics — it must author a thin adapter layer. The core abstraction is synchronous and long-running: $spider->crawl() blocks until the queue empties. That is philosophically at odds with Laravel's request/queue worker model but maps cleanly onto a queued job or Artisan command if the boundary is drawn correctly.
Integration feasibility. High on paper, moderate in practice. Guzzle satisfies the PSR-18/PSR-7 requirements Laravel's HTTP client already depends on, so dependency resolution is unlikely to conflict with framework constraints. The library is actively maintained (PHP 7.4/8.x era), MIT-licensed, and small enough to audit. The friction is conceptual rather than mechanical: crawler configuration (depth, politeness delays, URI filters, robots.txt handling, per-host throttling, redirect policy) must be surfaced as Laravel config and validated, and the observer system must be bridged to Laravel events so scraping results flow into queues, Eloquent models, and job dispatch rather than being trapped in an inline callback. With 0 dependents, there is no ecosystem precedent to copy — the adapter is bespoke work owned by your team.
Technical risk. The dominant risks are operational, not architectural:
memory_limit and outrun worker timeouts. Depth caps, host allowlists, and max_uris are mandatory, not optional.Spider object is not designed to be frozen and resumed. You cannot trivially serialize it into a queued job and continue next tick, which pushes you toward one-job-per-crawl or a custom checkpointing observer that persists discovered URIs to a store and rehydrates.robots.txt enforcement, user-agent identification, and allowlist/denylist filters. Shipping a spider that ignores robots.txt or hammers a target is a reputational and potentially ToS-violating failure mode.Key questions. What is the crawl's purpose — content ingestion, link validation, competitive monitoring, or internal site QA? What is the maximum expected page count per run, and does memory hold for it? Must crawling be resumable across queue workers, or is one bounded job acceptable? Is the target third-party (needs politeness/rate-limit discipline and legal review) or first-party (needs auth/session handling)? Who owns the adapter package, and will it be versioned internally?
Stack fit. Reasonable, provided the team accepts an adapter layer. The natural shape is an internal Laravel package (app/Support/Spider or a private Composer package) that: (1) registers a SpiderServiceProvider; (2) publishes config/spider.php for depth, delay, concurrency, user-agent, robots.txt toggle, and URL filters; (3) binds a SpiderFactory into the container that assembles the Spider from config plus an injected Guzzle client (so Laravel's HTTP client middleware, retries, and logging are reused); (4) bridges Spider observers to Laravel events via Event::dispatch, which is where persistence and downstream job dispatch live; and (5) exposes an Artisan command and a CrawlSiteJob for operational entry points.
Migration path. Treat this as greenfield instrumentation rather than a migration — there is no incumbent to replace and no legacy surface. The incremental path is: spike (one Artisan command, hardcoded config, one site, results dumped to storage) → harden (config file, container bindings, event bridge, robots.txt and rate-limit enforcement) → operationalize (queued job, per-host locking to prevent concurrent crawls of the same domain, structured logging, metrics) → expose (internal API or admin UI for triggering crawls and inspecting results). Each stage is independently shippable and reversible.
Compatibility. Low risk. Guzzle is already in the dependency graph of most Laravel apps; symfony/event-dispatcher ships with the framework. The binding constraints are PHP version (library tracks 7.4/8.x — verify against your framework's floor) and any PSR-18 client you substitute. Do not attempt to swap in Laravel's Http facade as the transport — it is not a PSR-18 client and the library requires one; wrap Guzzle instead and layer middleware there.
Sequencing. (1) Confirm PHP and Guzzle version alignment and lock the library to a specific minor. (2) Build the service provider, config, and factory; enforce safe defaults (depth ≤ 3, delay ≥ 1s, robots.txt on, host allowlist required). (3) Wire observers to events and define the persistence schema before writing scrape logic. (4) Add the queued job with WithoutOverlapping scoped to the target host. (5) Instrument and load-test against a realistic site size before enabling in production. (6) Only then expand to multi-target or scheduled crawls.
Maintenance. Ongoing cost is the bespoke adapter, not the library. Budget for periodic reviews of the config surface, the observer-to-event bridge, and URL filtering rules as sites change. Because the package has 0 dependents, upstream regressions are unlikely to be caught by the community — your CI must own that: pin the version, run integration tests against a fixture site (a local static server), and gate upgrades behind them. Expect occasional breakage from Guzzle/PSR major versions; the library's release cadence should be monitored quarterly.
Support. The library is a single-maintainer open-source project under MIT — no commercial SLA. Internal ownership must be explicit: one engineer or a small team accountable for the adapter, crawl configuration, and incident response. Document the safe-default policy (politeness, allowlists) so operators cannot casually disable robots.txt or rate limits. Establish a runbook covering "crawl is stuck," "crawl is hammering a host," and "crawl exhausted memory."
Scaling. Vertical scaling is the primary lever — more memory and a longer timeout for the worker executing a crawl. Horizontal scaling is achievable but requires partitioning the URL space by host and enforcing one active crawl per host via a distributed lock; the Spider itself does not coordinate across processes. Concurrency within a crawl is bounded by the underlying client and your politeness settings, so throughput is intentionally modest. If the workload grows to thousands of pages per run, this library becomes the wrong tool and a distributed crawler (or a hosted scraping service) should replace it — plan the exit boundary now by keeping crawl orchestration behind your own interface.
Failure modes. The realistic ones: memory exhaustion on unbounded queues; worker timeout on deep crawls; silent truncation when filters are too aggressive; infinite loops from redirect chains or session-token URLs (requires canonicalization and a visited-set); robots.txt misparsing; target-site rate limiting or IP bans from insufficient delay; duplicate or near-duplicate content inflating storage; and unhandled exceptions from a single bad URL aborting the entire crawl unless observers catch and log per-URI failures. Each of these needs a guard, an alert, and a documented recovery step — not a code comment.
Ramp-up. Moderate: one to two weeks for a mid-level Laravel engineer to reach productivity, most of which is spent on the adapter and event bridge rather than the library. The library's API is small and legible; the real learning curve is crawling etiquette, URL canonicalization, and the stateful/synchronous execution model that conflicts with queue-worker intuition. Pair a spike with a written internal usage guide and a hardened config template, and the knowledge concentrates in one or two people — acceptable for an internal tool, risky if it becomes business-critical without a second owner.
How can I help you explore Laravel packages today?