- How do I integrate CrawlerDetect into Laravel middleware to block bots?
- Create a middleware class that injects the `CrawlerDetect` service and checks `isCrawler()` in the `handle()` method. Return a 403 response or redirect bots. Example: `if ($this->detector->isCrawler()) return response('Forbidden', 403);`. Bind the detector as a singleton in a service provider for DI.
- Will this work with Laravel 10/11 and PHP 8.1+?
- Yes, CrawlerDetect supports Laravel 10/11 and PHP 8.1+. It dropped PHP 7.1 support in v1.4.0, so ensure your Laravel app meets these requirements. Tested up to PHP 8.5, but Laravel’s PHP policy may lag—verify compatibility with your stack.
- Can I extend CrawlerDetect to detect custom bots or AI agents like ChatGPT?
- Absolutely. Edit the `$data` array in `src/Fixtures/Crawlers.php` to add regex patterns for new bots. Contribute updates upstream or fork the package. Use `getMatches()` to log unidentified UAs for manual review, especially for emerging crawlers like AI agents.
- How accurate is CrawlerDetect? Will it misclassify legitimate users (e.g., Puppeteer)?
- CrawlerDetect is highly accurate but not perfect. False positives can occur with headless browsers like Puppeteer. Use `getMatches()` to inspect detected bots and log unidentified UAs for review. Combine with IP-based checks (e.g., `spatie/fake-useragent-laravel`) for defense-in-depth.
- Does CrawlerDetect work with Laravel’s Request object out of the box?
- Yes, but you’ll need to manually pass the User-Agent header. For seamless integration, add a Request macro like `Request::macro('isCrawler', fn() => app(CrawlerDetect::class)->isCrawler($this->userAgent()));`. This lets you call `$request->isCrawler()` directly in controllers or middleware.
- How do I handle false positives/negatives in production?
- Log unidentified UAs using `getMatches()` and review them periodically. Contribute fixes upstream by updating `src/Fixtures/Crawlers.php` or fork the package. For critical apps, combine with additional checks (e.g., IP reputation) and monitor false positive rates in staging.
- Is CrawlerDetect performant enough for high-traffic Laravel apps?
- Yes, it’s optimized with compiled regex caching and memoization since v1.4.0. Benchmark in staging under your traffic load (e.g., 100K+ RPS) to confirm performance. For HTTP-level blocking, use middleware to avoid runtime overhead in business logic.
- Can I use CrawlerDetect to filter bots from Google Analytics?
- Yes, integrate it into middleware or a service to exclude bot traffic from analytics. Example: `if (!$request->isCrawler()) { Analytics::track($request); }`. This reduces noise in reports without affecting your Laravel app’s core logic.
- What are the alternatives to CrawlerDetect for Laravel?
- Alternatives include `bot/lite` (simpler but less maintained) and `spatie/fake-useragent-laravel` (IP-based checks). CrawlerDetect stands out for its extensive UA database, regex flexibility, and Laravel-friendly integration. For defense-in-depth, combine it with IP reputation services.
- How do I test CrawlerDetect in my Laravel app?
- Use the included test fixtures (e.g., `tests/data/user_agent/crawlers.txt`) to mock crawler UAs in PHPUnit. Test edge cases like spoofed headers or headless browsers. For middleware, mock `$request->userAgent()` and verify `isCrawler()` behavior. Example: `$request->expectsToReceive('userAgent')->andReturn('Googlebot');`