Laravel Robots Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require alibaba258/laravel-robots
   

   Publish the config file (if needed):

   php artisan vendor:publish --provider="Alibaba258\Robots\RobotsServiceProvider" --tag="config"
   

2. Basic Usage: Generate a robots.txt file in your public directory:

   use Alibaba258\Robots\Facades\Robots;
   
   Robots::generate();
   

   This creates a default robots.txt at /public/robots.txt.

3. First Use Case: Restrict a search engine (e.g., Google) from crawling a specific path:

   Robots::disallow('Googlebot', '/admin');
   

Implementation Patterns

Core Workflows

1. Dynamic Rule Management: Use middleware to set rules per route:

   public function handle($request, Closure $next) {
       Robots::disallow('*', route('admin.dashboard'));
       return $next($request);
   }
   

2. Conditional Generation: Generate robots.txt only in production:

   if (app()->environment('production')) {
       Robots::generate();
   }
   

3. Custom Sitemaps: Add sitemaps dynamically:

   Robots::addSitemap('https://example.com/sitemap.xml');
   

Integration Tips

• Cache Optimization: Cache the generated robots.txt for performance:

  Robots::generate()->cacheFor(3600); // Cache for 1 hour
  

• Route-Based Rules: Attach rules to specific routes in routes/web.php:

  Route::get('/private', function () {
      Robots::disallow('*', '/private');
      return view('private');
  });
  

• User-Agent Groups: Apply rules to multiple user agents:

  Robots::disallow(['Googlebot', 'Bingbot'], '/temp');
  

Gotchas and Tips

Pitfalls

1. File Permissions: Ensure the public/robots.txt file is writable by the web server. Use:

   chmod 644 public/robots.txt
   

2. Overwriting Rules: Rules are appended by default. Use clear() to reset before generating:

   Robots::clear()->disallow('*', '/old-path')->generate();
   

3. Case Sensitivity: User-agent names are case-sensitive (e.g., Googlebot ≠ googlebot).

Debugging

• Verify Output: Check the generated robots.txt manually or via:

  dd(Robots::getRules());
  

• Logging: Enable debug mode in config (debug=true) to log rule changes.

Extension Points

1. Custom Directives: Extend the package by adding custom directives (e.g., Host):

   Robots::addDirective('Host', 'example.com');
   

2. Event Listeners: Listen for robots.generated events to post-process the file:

   Robots::generated(function ($path) {
       // Add custom logic (e.g., log, notify)
   });
   

3. Configuration Overrides: Override defaults in config/robots.php:

   'default_user_agent' => 'Bingbot',
   'file_path' => 'custom/robots.txt',