Crawler Detect Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require jaybizzle/crawler-detect
   

   Add to composer.json if using Laravel’s require-dev for testing.

2. First Use Case: Detect crawlers in a Laravel middleware or controller:

   use Jaybizzle\CrawlerDetect\CrawlerDetect;
   
   $detect = new CrawlerDetect();
   if ($detect->isCrawler()) {
       // Block, log, or redirect crawlers
       abort(403, 'Crawlers not allowed');
   }
   

3. Where to Look First:

   • src/Fixtures/Crawlers.php: Core regex patterns for detection.
   • tests/data/user_agent/: Test cases for known crawlers.
   • Laravel Integration: Use the Laravel-Crawler-Detect package for middleware/blade helpers.

Implementation Patterns

Core Workflows

1. Middleware Integration (Laravel):

   // app/Http/Middleware/DetectCrawler.php
   public function handle(Request $request, Closure $next) {
       $detect = new CrawlerDetect();
       if ($detect->isCrawler()) {
           return redirect()->route('crawler-blocked');
       }
       return $next($request);
   }
   

   Register in app/Http/Kernel.php:

   protected $middleware = [
       \App\Http\Middleware\DetectCrawler::class,
   ];
   

2. API Rate Limiting:

   $detect = new CrawlerDetect();
   if ($detect->isCrawler()) {
       return response()->json(['error' => 'Rate limit exceeded'], 429);
   }
   

3. Analytics Exclusion:

   if (!$detect->isCrawler()) {
       Analytics::trackEvent('page_view');
   }
   

4. Custom User Agent Check:

   $ua = 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)';
   if ($detect->isCrawler($ua)) {
       // Handle custom UA
   }
   

Advanced Patterns

• Cache Detection Results (for performance):

  $cacheKey = 'crawler_' . md5($request->userAgent());
  $isCrawler = Cache::remember($cacheKey, 3600, function () use ($detect) {
      return $detect->isCrawler();
  });
  

• Extend Detection Logic:

  $matches = $detect->getMatches();
  if (in_array('Googlebot', $matches)) {
      // Custom logic for Googlebot
  }
  

• Laravel Service Provider: Bind CrawlerDetect to the container for dependency injection:

  // app/Providers/AppServiceProvider.php
  public function register() {
      $this->app->singleton(CrawlerDetect::class, function () {
          return new CrawlerDetect();
      });
  }
  

Gotchas and Tips

Pitfalls

1. False Positives:

   • Some headless browsers (e.g., Puppeteer) may trigger crawler detection. Use getMatches() to verify:

     $matches = $detect->getMatches();
     if (in_array('Puppeteer', $matches)) {
         // Allow headless browsers if needed
     }
     

   • Fix: Exclude known false positives in Crawlers.php or override detection logic.

2. Performance:

   • Regex compilation is memoized, but heavy usage in loops may impact performance. Cache results for repeated checks.

3. User Agent Spoofing:

   • Crawlers can spoof UAs. Combine with IP reputation checks or behavioral analysis for higher accuracy.

4. Laravel Request Object:

   • If using Laravel, pass the Request object directly for automatic header parsing:

     $detect = new CrawlerDetect();
     $detect->setRequest($request); // Laravel Request object
     

Debugging Tips

• Log Matches:

  \Log::debug('Crawler matches:', ['matches' => $detect->getMatches()]);
  

• Test Edge Cases: Use the test fixtures in tests/data/user_agent/ to validate custom UAs.

Extension Points

1. Add Custom Crawlers: Edit src/Fixtures/Crawlers.php and add a regex pattern:

   'custom_bot' => [
       'pattern' => '/CustomBot\/1\.0/',
       'name'    => 'CustomBot',
   ],
   

   Regenerate raw files via composer run-script export.

2. Override Detection: Extend the CrawlerDetect class:

   class CustomCrawlerDetect extends CrawlerDetect {
       protected function getData() {
           $data = parent::getData();
           $data['custom_bot'] = ['pattern' => '/MyCustomPattern/', 'name' => 'MyBot'];
           return $data;
       }
   }
   

3. Laravel Blade Helper: Add a helper in app/Helpers/CrawlerDetect.php:

   if (!function_exists('is_crawler')) {
       function is_crawler($ua = null) {
           $detect = new \Jaybizzle\CrawlerDetect\CrawlerDetect();
           return $detect->isCrawler($ua);
       }
   }
   

   Use in Blade:

   @if(!is_crawler())
       <!-- Render content for humans -->
   @endif
   

Config Quirks

• Header Prioritization: The library checks HTTP_FROM before User-Agent. Override in CrawlerDetect if needed:

  public function setHeaders(array $headers) {
      $this->headers = array_merge(['HTTP_FROM' => null], $headers);
  }
  

• Case Sensitivity: Regex patterns are case-insensitive by default. Adjust if needed:

  'pattern' => '/case-sensitive-pattern/',