Crawler Laravel Package

Getting Started

1. Install with Composer (the package is framework-agnostic at its core but ships Laravel-friendly wiring):

   composer require mediamonks/crawler
   

2. Publish the config (Laravel) and set the target site before touching any code:

   php artisan vendor:publish --tag=crawler-config
   

   In config/crawler.php, the first things to change are the base URL, the list of allowed hosts, and the default client driver. Point it at a staging environment, not production, on your first run.

3. Where to look first:

   • config/crawler.php — this is the real API surface. Client selection, concurrency, timeouts, user agent, and URL filtering all live here.
   • The Clients directory (or Crawler\Client\* namespace) — each driver is a small adapter. Read the default one to understand the request/response contract before writing your own.
   • Any provided artisan command (crawler:crawl or similar) — run php artisan list to confirm the exact name; this is the fastest way to see the package working end to end.

4. First use case — a broken-link smoke test against staging: Run the crawl command against your staging URL, dump the collected URLs and status codes to stdout, and grep for 4xx/5xx. This validates the install, network config, and client driver in one pass before you invest in event listeners or storage.

Implementation Patterns

Client-swapping per environment. The main value of the package is the client abstraction. Use a lightweight HTTP client (Guzzle-style) for large, text-heavy crawls and a headless-browser client (Panther-style) only for pages where JS-rendered content matters. Keep two config profiles — fast and rendered — and select by environment rather than editing config inline.

Event-driven per-page processing. The idiomatic integration is to listen for the crawl's per-page event rather than post-processing a giant result array. Persist each visited URL, its status, and extracted meta tags to your own table (or log) inside a listener. This keeps memory flat and lets you build reports incrementally:

Event::listen(PageCrawled::class, function ($event) {
    CrawledPage::updateOrCreate(
        ['url' => $event->url],
        ['status' => $event->status, 'title' => $event->meta['title'] ?? null]
    );
});

Queue the crawl, don't run it synchronously. Wrap the crawl dispatch in a job so it survives request timeouts. Combine with a scheduler entry for nightly SEO regression checks against staging.

Link-graph expansion with filters. Configure allowed hosts and path exclusions (query-string permutations, /admin, asset extensions) before enabling recursion, or a single crawl will spiral into thousands of URLs. Most crawl blowups are filter-config mistakes, not client bugs.

CI gate. Wire the crawl into a CI job against a preview deployment: fail the build if any page returns 5xx, if canonical tags are missing, or if robots.txt disallows a route you expect to be indexed. Treat the crawler's output as test assertions.

Report pipeline. Emit a per-page DTO from your listener and push it to a CSV or a seo_pages table. That gives you a diffable artifact for "what changed since last deploy" — far more useful than a one-off console dump.

Gotchas and Tips

• Respect rate limits and robots.txt. The package crawls your site, so throttling is polite rather than mandatory — but a headless-browser client will happily saturate staging CPU. Set concurrency low (1–2) for browser clients and higher for HTTP clients.
• Absolute vs. relative links. Normalize URLs (trailing slash, fragments, query ordering) in your filter callbacks. Duplicate detection fails silently otherwise, and you'll re-crawl the same page ten times.
• Client feature parity is not guaranteed. Extracted meta tags, redirect handling, and JS-rendered content differ per driver. If a field is missing, check which client you're running before debugging your listener.
• Headless clients need a running binary. If you use a Panther/Selenium-style driver, the Chrome/Chromedriver dependency is the most common local-vs-CI failure. Pin versions and pre-install in the CI image.
• Config caching bites in production. Run php artisan config:clear after editing crawler.php; a stale cached config pointing at the wrong base URL produces confusing "0 pages crawled" results.
• Memory grows with the frontier, not the page count. For sites over a few thousand URLs, persist the queue or split by path prefix rather than holding the full visited set in memory.
• Extension points to lean on: custom client adapters (implement the client interface and register it in config), the per-page event for storage/validation, and URL filter callbacks. Most "I need it to do X" requirements are solved by a listener, not by forking.
• Low-activity package — pin it. With a small star count and limited release cadence, constrain the version in composer.json and keep a thin wrapper around the package's public classes. If the API shifts or you need to replace it, your wrapper is the only thing you rewrite.
• Debugging silently-empty crawls: log the client's raw response for the seed URL first. An SSL error, a redirect to a login page, or a user-agent block all present as "no pages found."