Crawler Laravel Package
spatie/crawler
A powerful PHP package for crawling websites, using Guzzle promises to fetch multiple URLs concurrently. It can render JavaScript via Chrome and Puppeteer, restrict crawls to internal links or depth, collect found URLs, and fake HTTP responses for testing crawl logic.
spatie/crawler is a powerful PHP package for crawling links on a website, built on Guzzle promises so multiple URLs are fetched concurrently. It can even execute JavaScript through Chrome and Puppeteer (via Browsershot), making JavaScript-rendered sites crawlable too.
Getting started is a one-liner: create a crawler, register hooks, and call start(). Collect all discovered URLs with foundUrls(), restrict the crawl with internalOnly() and depth(), or swap real requests for fixtures using fake() to test crawl logic offline.
- Concurrent crawling powered by Guzzle promises
- JavaScript rendering via headless Chrome
- Simple callbacks such as
onCrawledandshouldStopCallback - Built-in fakes for request-free testing
- Fluent filters for depth and internal-only links
- How do I install spatie/crawler in a Laravel project?
- Run `composer require spatie/crawler` in your project directory. The package integrates seamlessly with Laravel’s service container and requires no additional configuration unless using JavaScript rendering, which needs Chrome/Puppeteer installed.
- Can I crawl JavaScript-rendered sites (e.g., React/Vue SPAs) with this package?
- Yes, the crawler supports dynamic content via Puppeteer/Chrome. Enable it by calling `withBrowser()` on the crawler instance, but note this adds overhead and requires Chrome to be installed in your environment.
- How do I limit the crawl depth or scope to internal links only?
- Use `depth($maxDepth)` to restrict crawl depth and `internalOnly()` to focus on links within the same domain. For example, `Crawler::create('https://example.com')->internalOnly()->depth(3)->start()`.
- Is spatie/crawler compatible with Laravel 10+?
- Yes, the package supports Laravel 9 and 10. Check the [GitHub repo](https://github.com/spatie/crawler) for the latest version, as minor updates may introduce breaking changes for newer Laravel releases.
- How can I test crawl logic without hitting real APIs?
- Use the `fake()` method to simulate responses. Pass an associative array of URLs to their HTML content, like `->fake(['https://example.com' => '<html>...</html>'])`. This works with `foundUrls()` and `onCrawled` callbacks.
- What’s the best way to handle large-scale crawls (e.g., 10K+ pages) in Laravel?
- For scalability, pair the crawler with Laravel Queues (e.g., `shouldQueue()`) and distribute jobs across workers using Horizon or Redis. The crawler itself is single-process, so external queue systems are needed for distributed crawling.
- Does spatie/crawler respect robots.txt or rate limits?
- The package does not enforce `robots.txt` or rate limits by default. Use Guzzle middleware (e.g., `RateLimitMiddleware`) or custom observers to implement these rules, especially for external sites.
- How do I store crawled URLs or responses in a Laravel database?
- The crawler returns raw data (e.g., `foundUrls()` or `CrawlResponse`). Manually persist this to a database using Eloquent or a custom table. Example: `Url::create(['path' => $url, 'status' => $response->status()])`.
- Are there alternatives to spatie/crawler for Laravel?
- Yes, consider `symfony/panther` (headless browser testing) or `laravel-web-scraper` for simpler scraping. For distributed crawling, `spatie/crawler` paired with queues is more robust than standalone tools like `Goutte` (PHP-only, no JS).
- How do I debug failed crawls or timeouts in production?
- Use observers to log errors (e.g., `->onError(function ($url, $exception) { Log::error($exception); })`). For timeouts, adjust Guzzle’s timeout settings or implement retry logic with `shouldRetryCallback()`.
How can I help you explore Laravel packages today?