No Ai Bundle Laravel Package

Getting Started

Minimal Steps

1. Installation:

   composer require deuchnord/no-ai-bundle
   

   The bundle auto-registers via Symfony’s autoloader.

2. First Use Case: Test with a known AI crawler User-Agent (e.g., GPTBot):

   curl -v -H "User-Agent: GPTBot/1.1" http://your-site.test
   

   Expected: 403 Forbidden response.

3. Where to Look First:

   • Configuration: config/packages/deuchnord_no_ai.yaml (auto-generated).
   • Event Listener: src/EventListener/AiCrawlerListener.php (customize logic if needed).
   • Tests: tests/ for edge cases (e.g., whitelisted UAs).

Implementation Patterns

Core Workflow

1. Request Interception: The bundle hooks into Symfony’s kernel.request event via AiCrawlerListener. It checks the User-Agent header against a predefined list of AI crawlers (e.g., GPTBot, CCBot, AhrefsBot).

2. Blocking Logic:

   • Default: Returns 403 Forbidden for matched UAs.
   • Customization: Override the listener or extend the AiCrawlerChecker service (see Gotchas).

3. Integration Tips:

   • Whitelisting: Add exceptions in config/packages/deuchnord_no_ai.yaml:

     deuchnord_no_ai:
         whitelist:
             - 'AmazonBot'  # Allow Amazon crawlers
     

   • Logging: Extend the listener to log blocked requests:

     use Psr\Log\LoggerInterface;
     
     public function __construct(private LoggerInterface $logger) {}
     
     public function onKernelRequest(): void {
         if ($this->isAiCrawler($request)) {
             $this->logger->warning('Blocked AI crawler', ['ua' => $request->headers->get('User-Agent')]);
             throw new AccessDeniedHttpException();
         }
     }
     

4. Performance:

   • The check runs before Symfony’s kernel, minimizing overhead.
   • For high-traffic sites, cache the User-Agent regex patterns (e.g., via Symfony\Component\HttpFoundation\RequestStack).

Gotchas and Tips

Pitfalls

1. False Positives:

   • Some legitimate bots (e.g., AmazonBot) may be blocked if not whitelisted.
   • Fix: Audit your whitelist config and test with real crawlers.

2. Dynamic User-Agents:

   • AI crawlers may spoof UAs or rotate them.
   • Tip: Use the AiCrawlerChecker service to add custom patterns:

     services:
         deuchnord_no_ai.ai_crawler_checker:
             arguments:
                 $patterns: ['/CustomAIPattern/i']
     

3. Symfony Cache:

   • If using Symfony’s HTTP cache, ensure the 403 response isn’t cached by AI crawlers.
   • Solution: Add Cache-Control: no-store headers in the listener.

4. Testing:

   • Mock User-Agent headers in PHPUnit:

     $request = new Request([], [], ['HTTP_USER_AGENT' => 'GPTBot/1.1']);
     $this->assertTrue($aiCrawlerChecker->isAiCrawler($request));
     

Debugging

• Enable Debug Mode: Temporarily disable blocking in config/packages/deuchnord_no_ai.yaml:

  deuchnord_no_ai:
      enabled: false
  

  Useful for diagnosing false blocks.

• Check Headers: Log the raw User-Agent to verify detection:

  $this->logger->debug('User-Agent', ['ua' => $request->headers->get('User-Agent')]);
  

Extension Points

1. Custom Blocking Logic: Override the AiCrawlerListener and inject your own logic (e.g., redirect to a CAPTCHA):

   public function onKernelRequest(Request $request): void {
       if ($this->isAiCrawler($request)) {
           return new RedirectResponse('/captcha');
       }
   }
   

2. Add New Crawlers: Extend the AiCrawlerChecker to include new patterns:

   public function __construct(array $patterns = []) {
       $this->patterns = array_merge($this->defaultPatterns, $patterns);
   }
   

3. Event Dispatching: Trigger events when crawlers are blocked (e.g., for analytics):

   $this->eventDispatcher->dispatch(new AiCrawlerBlockedEvent($request));