Robots Txt File Laravel Package

Getting Started

Minimal Steps

1. Installation

   composer require webignition/robots-txt-file:^3.0
   

   Requires PHP 7.2+ (enforced in v3.0).

2. Basic Usage

   use Webignition\RobotsTxtFile\RobotsTxtFile;
   
   $robotsTxt = new RobotsTxtFile();
   $robotsTxt->addRule('*', ['Disallow' => '/private']);
   $content = $robotsTxt->render();
   

3. First Use Case Generate a robots.txt file for Laravel (v3.0+ compatible):

   // In a controller or service
   $robotsTxt = new RobotsTxtFile();
   $robotsTxt->addRule('*', ['Disallow' => '/admin']);
   $robotsTxt->addRule('Googlebot', ['Allow' => '/public']);
   
   return response($robotsTxt->render())
       ->header('Content-Type', 'text/plain');
   

Implementation Patterns

Core Workflows

1. Dynamic Rule Management (PHP 7.2+)

   // Add rules conditionally (e.g., based on environment)
   if (app()->environment('production')) {
       $robotsTxt->addRule('*', ['Disallow' => '/temp']);
   }
   

2. Integration with Laravel Routes

   // In routes/web.php (Laravel 7+)
   Route::get('/robots.txt', function () {
       $robotsTxt = new RobotsTxtFile();
       $robotsTxt->addRule('*', ['Disallow' => '/vendor']);
       return response($robotsTxt->render())->header('Content-Type', 'text/plain');
   });
   

3. Caching Rendered Output (Laravel 8+)

   // Cache the rendered robots.txt for 1 hour
   $cacheKey = 'robots.txt';
   $content = Cache::remember($cacheKey, now()->addHour(), function () {
       $robotsTxt = new RobotsTxtFile();
       $robotsTxt->addRule('*', ['Disallow' => '/cache-me']);
       return $robotsTxt->render();
   });
   

4. Environment-Specific Config (Laravel 7+) Use Laravel’s config to manage rules per environment:

   // config/robots.txt.php
   return [
       'production' => [
           ['user-agent' => '*', 'disallow' => '/private'],
           ['user-agent' => 'Googlebot', 'allow' => '/public'],
       ],
       'staging' => [
           ['user-agent' => '*', 'disallow' => '/staging-only'],
       ],
   ];
   

   // In a service (PHP 7.2+)
   $rules = config('robots.txt.' . app()->environment());
   foreach ($rules as $rule) {
       $robotsTxt->addRule($rule['user-agent'], [
           'Disallow' => $rule['disallow'] ?? null,
           'Allow' => $rule['allow'] ?? null,
       ]);
   }
   

5. Validation Before Rendering (PSR-12 Compliant)

   // Ensure no conflicting rules (e.g., Allow/Disallow for same path)
   $rules = $robotsTxt->getRules();
   foreach ($rules as $rule) {
       if (isset($rule['Allow']) && isset($rule['Disallow'])) {
           throw new \RuntimeException("Conflicting rules for {$rule['user-agent']}");
       }
   }
   

Gotchas and Tips

Pitfalls

1. PHP 7.2+ Requirement

   • Breaking Change: v3.0 drops PHP 7.1 support. Update your composer.json:

     "require": {
         "php": "^7.2",
         "webignition/robots-txt-file": "^3.0"
     }
     

2. Case Sensitivity in User-Agents The package treats Googlebot and googlebot as distinct. Normalize user-agent strings:

   $robotsTxt->addRule(strtolower($userAgent), ['Disallow' => '/path']);
   

3. Wildcard (*) Overrides Rules for * apply to all bots unless overridden. Test edge cases:

   $robotsTxt->addRule('*', ['Disallow' => '/all']);
   $robotsTxt->addRule('Googlebot', ['Allow' => '/all']); // Overrides for Googlebot
   

4. Empty Rules Adding an empty rule (e.g., addRule('Bot', [])) may not render as expected. Validate inputs:

   if (!empty($disallowPaths)) {
       $robotsTxt->addRule($userAgent, ['Disallow' => $disallowPaths]);
   }
   

5. PSR-12 Compliance

   • v3.0 enforces PSR-12 coding standards. If extending the package, ensure compliance with:
     • Method naming (camelCase).
     • Type hints (e.g., array → array $directives).
     • Docblocks for public methods.

Debugging

1. Inspect Rules Before Rendering

   $rules = $robotsTxt->getRules();
   dd($rules); // Debug the structure (PHP 7.2+)
   

2. Validate Output Use Google’s robots.txt Tester to verify rendered content.

3. Logging (Laravel 7+) Log rendered output for auditing:

   \Log::debug('Robots.txt rendered:', ['content' => $robotsTxt->render()]);
   

Extension Points

1. Custom Rule Validators (PHP 7.2+) Extend the package by adding validation logic:

   class CustomRobotsTxt extends RobotsTxtFile {
       public function addRule(string $userAgent, array $directives): void {
           if (isset($directives['Disallow']) && strpos($directives['Disallow'], '/') !== 0) {
               throw new \InvalidArgumentException("Disallow paths must start with '/'");
           }
           parent::addRule($userAgent, $directives);
       }
   }
   

2. Integration with Laravel’s Filesystem (Laravel 7+) Save the rendered file to public/robots.txt:

   use Illuminate\Support\Facades\Storage;
   
   Storage::disk('public')->put('robots.txt', $robotsTxt->render());
   

3. Dynamic Rule Loading (PHP 7.2+) Load rules from a database or API:

   $rules = DB::table('robots_txt_rules')->get();
   foreach ($rules as $rule) {
       $robotsTxt->addRule($rule->user_agent, [
           'Disallow' => $rule->disallow_paths ?? null,
           'Allow' => $rule->allow_paths ?? null,
       ]);
   }
   

4. Testing (Laravel 8+) Use Laravel’s HTTP tests to verify robots.txt:

   public function test_robots_txt()
   {
       $response = $this->get('/robots.txt');
       $response->assertStatus(200);
       $this->assertStringContainsString('Disallow: /private', $response->getContent());
   }
   

5. Static Analysis (phpstan)

   • v3.0 includes phpstan compatibility. Run static analysis:

     vendor/bin/phpstan analyse --level=5 src/