Weave Code
Crawler Laravel Package
spatie/crawler
PHP web crawler with concurrent requests via Guzzle. Crawl internal/external links, limit depth, and collect discovered URLs. Supports JavaScript-rendered sites using Chrome/Puppeteer. Includes faking responses for testing crawl logic without real HTTP calls.
Frequently asked questions about Crawler
- How do I install spatie/crawler in a Laravel project?
- Run `composer require spatie/crawler` in your project directory. The package integrates seamlessly with Laravel’s service container and requires no additional configuration unless using JavaScript rendering, which needs Chrome/Puppeteer installed.
- Can I crawl JavaScript-rendered sites (e.g., React/Vue SPAs) with this package?
- Yes, the crawler supports dynamic content via Puppeteer/Chrome. Enable it by calling `withBrowser()` on the crawler instance, but note this adds overhead and requires Chrome to be installed in your environment.
- How do I limit the crawl depth or scope to internal links only?
- Use `depth($maxDepth)` to restrict crawl depth and `internalOnly()` to focus on links within the same domain. For example, `Crawler::create('https://example.com')->internalOnly()->depth(3)->start()`.
- Is spatie/crawler compatible with Laravel 10+?
- Yes, the package supports Laravel 9 and 10. Check the [GitHub repo](https://github.com/spatie/crawler) for the latest version, as minor updates may introduce breaking changes for newer Laravel releases.
- How can I test crawl logic without hitting real APIs?
- Use the `fake()` method to simulate responses. Pass an associative array of URLs to their HTML content, like `->fake(['https://example.com' => '<html>...</html>'])`. This works with `foundUrls()` and `onCrawled` callbacks.
- What’s the best way to handle large-scale crawls (e.g., 10K+ pages) in Laravel?
- For scalability, pair the crawler with Laravel Queues (e.g., `shouldQueue()`) and distribute jobs across workers using Horizon or Redis. The crawler itself is single-process, so external queue systems are needed for distributed crawling.
- Does spatie/crawler respect robots.txt or rate limits?
- The package does not enforce `robots.txt` or rate limits by default. Use Guzzle middleware (e.g., `RateLimitMiddleware`) or custom observers to implement these rules, especially for external sites.
- How do I store crawled URLs or responses in a Laravel database?
- The crawler returns raw data (e.g., `foundUrls()` or `CrawlResponse`). Manually persist this to a database using Eloquent or a custom table. Example: `Url::create(['path' => $url, 'status' => $response->status()])`.
- Are there alternatives to spatie/crawler for Laravel?
- Yes, consider `symfony/panther` (headless browser testing) or `laravel-web-scraper` for simpler scraping. For distributed crawling, `spatie/crawler` paired with queues is more robust than standalone tools like `Goutte` (PHP-only, no JS).
- How do I debug failed crawls or timeouts in production?
- Use observers to log errors (e.g., `->onError(function ($url, $exception) { Log::error($exception); })`). For timeouts, adjust Guzzle’s timeout settings or implement retry logic with `shouldRetryCallback()`.
Weaver
How can I help you explore Laravel packages today?