- How do I install CrawlerDetect in a Laravel project?
- Run `composer require jaybizzle/crawler-detect` in your project root. For Laravel-specific integration (e.g., middleware or facades), use the `jaybizzle/laravel-crawler-detect` package instead, which extends the core library with Laravel helpers.
- Does CrawlerDetect work with Laravel 10/11 and PHP 8.4?
- Yes, CrawlerDetect supports PHP 8.1+ and is fully compatible with Laravel 10.x and 11.x. The latest version (v1.3.0+) explicitly tests against PHP 8.4, so no additional configuration is needed for modern Laravel setups.
- Can I use CrawlerDetect to block bots from specific routes?
- Absolutely. Register the `CrawlerDetectMiddleware` in your `app/Http/Kernel.php` under the `$routeMiddleware` array, then apply it to routes like this: `Route::middleware(['crawler.detect'])->group(function () { ... })`. This lets you block or redirect bots globally or per route.
- How accurate is CrawlerDetect at identifying bots vs. false positives?
- CrawlerDetect uses a large, regularly updated regex dataset for high accuracy, but false positives can occur (e.g., headless browsers like Puppeteer). Use the `getMatches()` method to log detected bots and review edge cases. Contribute missing patterns via GitHub PRs to improve coverage.
- Will CrawlerDetect impact my Laravel app’s performance?
- No, the package is optimized for speed with O(1) regex matching (~1–5ms per request). For high-traffic apps, cache results per IP or user-agent using Laravel’s cache layer or Redis to reduce overhead further.
- Can I whitelist or blacklist specific bots (e.g., allow Googlebot but block Scrapy)?
- Yes. After detecting a bot with `isCrawler()`, use `getMatches()` to identify the bot name, then implement custom logic (e.g., `if ($bot === 'Googlebot') { return true; }`). For middleware, extend the `CrawlerDetectMiddleware` or use Laravel’s policy system.
- How do I log detected bots for analytics or security audits?
- Call `getMatches()` to retrieve the bot name and IP, then log it via Laravel’s logging system (e.g., `Log::info('Detected bot:', ['bot' => $matches, 'ip' => request()->ip()])`). For structured storage, use Laravel Queues to asynchronously write to a database or SIEM.
- Are there alternatives to CrawlerDetect for Laravel?
- Yes, alternatives include `spatie/bot-detector` (simpler, Laravel-focused) or `matthiasmullie/mini-bot-detector` (lightweight but less comprehensive). CrawlerDetect stands out for its extensive bot signature database and active maintenance, making it ideal for production-grade detection.
- How can I extend CrawlerDetect to detect custom bots not in the default list?
- Add your bot’s regex pattern to the `$data` array in `src/Fixtures/Crawlers.php` and update the raw data files (`raw/Crawlers.json`, `raw/Crawlers.txt`). Test your changes locally, then submit a PR to the repository to help the community. The README provides detailed instructions.
- Does CrawlerDetect support detecting bots in API requests (e.g., GraphQL or REST)?
- Yes, CrawlerDetect works seamlessly with Laravel APIs. Use middleware to filter bot requests before they reach your controllers, or inject the `CrawlerDetect` service into your API routes to conditionally process or reject bot traffic.