Laravel Robots Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require mguinea/laravel-robots
   

   Publish the config file (optional, but recommended for customization):

   php artisan vendor:publish --provider="Mguinea\Robots\RobotsServiceProvider" --tag="config"
   

2. Basic Usage: Define rules in config/robots.php:

   'rules' => [
       'User-agent: *' => [
           'Disallow: /private',
           'Allow: /public',
       ],
       'User-agent: Googlebot' => [
           'Allow: /',
       ],
   ],
   

   Generate the robots.txt file via:

   php artisan robots:generate
   

   Or dynamically in a route/controller:

   use Mguinea\Robots\Facades\Robots;
   
   return Robots::generate();
   

3. First Use Case:

   • Block sensitive paths (e.g., /admin, /api) in production while allowing public access.
   • Override rules per environment (e.g., allow all crawlers in staging).

Implementation Patterns

Dynamic Rule Management

• Environment-Specific Rules: Use Laravel’s config caching or environment variables to toggle rules:

  'rules' => env('APP_ENV') === 'production'
      ? ['User-agent: *' => ['Disallow: /admin']]
      : [],
  

• Conditional Rules: Dynamically append rules based on user roles or other logic:

  $rules = Robots::getRules();
  $rules['User-agent: *'][] = 'Disallow: /user/' . auth()->id();
  Robots::setRules($rules)->generate();
  

Integration with Laravel Features

• Route-Based Rules: Sync robots.txt with route middleware (e.g., auth, guest):

  // In a service provider
  Robots::setRules(function () {
      return collect(Route::getRoutes())
          ->where('middleware', 'contains', 'auth')
          ->pluck('uri')
          ->map(fn ($uri) => "Disallow: $uri")
          ->toArray();
  });
  

• Caching: Cache generated robots.txt for performance:

  Cache::remember('robots.txt', now()->addHours(1), function () {
      return Robots::generate();
  });
  

• API Endpoint: Expose robots.txt as an API endpoint:

  Route::get('/api/robots.txt', function () {
      return response(Robots::generate(), 200, ['Content-Type' => 'text/plain']);
  });
  

Data Source Flexibility

• Database Backend: Use the optional migration to store rules in robots_rules table:

  php artisan migrate
  

  Then fetch rules dynamically:

  Robots::setDataSource(function () {
      return \DB::table('robots_rules')->get()->toArray();
  });
  

• External API: Fetch rules from a headless CMS or third-party service:

  Robots::setDataSource(function () {
      return json_decode(file_get_contents('https://api.example.com/robots-rules'), true);
  });
  

Gotchas and Tips

Pitfalls

• Caching Conflicts: If using Laravel’s route caching (php artisan route:cache), regenerate robots.txt after caching to avoid stale rules. Fix: Clear route cache or use php artisan robots:generate post-deployment.

• Case Sensitivity: robots.txt rules are case-sensitive for paths (e.g., /Admin ≠ /admin). Ensure consistency in your config.

• Wildcard Overrides: Explicit rules (e.g., User-agent: Googlebot) take precedence over wildcards (User-agent: *). Test edge cases:

  // This will allow Googlebot to access /private despite the wildcard Disallow
  'rules' => [
      'User-agent: *' => ['Disallow: /private'],
      'User-agent: Googlebot' => ['Allow: /private'],
  ],
  

Debugging

• Validate Syntax: Use Google’s robots.txt Tester to validate generated output. Tip: Add a debug endpoint:

  Route::get('/debug/robots', function () {
      return response(Robots::getRules(), 200, ['Content-Type' => 'application/json']);
  });
  

• Log Rule Changes: Log rule modifications to track unintended changes:

  \Log::info('Robots rules updated', ['rules' => Robots::getRules()]);
  

Extension Points

• Custom Directives: Extend the package to support non-standard directives (e.g., Sitemap):

  // In a service provider
  Robots::extend(function ($robots) {
      $robots->addDirective('Sitemap', 'https://example.com/sitemap.xml');
  });
  

• Middleware Integration: Dynamically block crawlers via middleware:

  public function handle($request, Closure $next) {
      if (Robots::isCrawler($request->userAgent())) {
          abort_if(Robots::isDisallowed($request->path()), 403);
      }
      return $next($request);
  }
  

• Multi-Tenant Rules: Scope rules by tenant (e.g., SaaS applications):

  Robots::setRules(function () {
      return Tenant::rules()->pluck('robots_rules', 'id')->toArray();
  });
  

Performance Tips

• Minify Output: Compress robots.txt by removing whitespace:

  return str_replace(["\r\n", "\n", "\r"], '', Robots::generate());
  

• Lazy Loading: Defer rule generation until first request:

  if (!Cache::has('robots.txt')) {
      Robots::generate();
  }