Robots Txt Laravel Package

Getting Started

Install the package via Composer: composer require spatie/robots-txt. Publish the config file with php artisan vendor:publish --provider="Spatie\RobotsTxt\RobotsTxtServiceProvider". The config (config/robots-txt.php) defines default rules and allows environment-specific overrides. By default, a /robots.txt route is auto-registered (GET /robots.txt) and returns a dynamic response. Start by checking config/app.php to ensure the service provider auto-discovers (no manual registration needed). For first use, update config/robots-txt.php to add your production sitemap and disallow paths like /admin. If upgrading to 2.5.5, validate the generated robots.txt after deployment because rule precedence behavior has changed.

Implementation Patterns

• Environment-aware control: Use RobotsTxt::configureUsing() in AppServiceProvider@boot to apply logic-based rules (e.g., if (app()->isLocal()) { $robots->disallow('/'); }).
• Programmatic rule building: In config or service providers, chain methods fluently:

  return RobotsTxt::create()
      ->userAgent('*', fn ($userAgent) => $userAgent
          ->disallow('/admin/')
          ->allow('/public/')
          ->sitemapUrl(env('APP_URL') . '/sitemap.xml')
      );
  

• Dynamic sitemaps: Inject environment variables or config keys for sitemap URLs to support multi-environment deployments without code changes.
• Precedence-aware rules: After 2.5.5, Allow and Disallow specificity use the same pattern matching, including wildcards (*) and end anchors ($). A longer matching wildcard rule can override a shorter one, and an equal-length Allow wins per RFC 9309 §2.2.2. Design rules intentionally and avoid relying on old ordering behavior.
• Regression testing: Add feature tests for GET /robots.txt and for critical allow/disallow decisions. Cover wildcard paths, $ anchors, and equal-length allow/disallow conflicts; these are the cases most likely to change crawl behavior after upgrading.
• Integration with Horizon/Queues: Temporarily disable indexing during maintenance or high-load periods by toggling a config flag and reusing the same RobotsTxt builder.
• Testing: Write feature tests asserting GET /robots.txt returns expected status and directives using Pest’s assertSee() or PHPUnit’s assertSnapshotEquals() against a known-good config snapshot.

Gotchas and Tips

• Rule precedence changed in 2.5.5: Allow rules containing wildcards (*) or end anchors ($) can now override less-specific Disallow rules. Longer matching wildcard Disallow rules also correctly override shorter Allow rules, and equal-length matches favor Allow. There are no public API or dependency changes, but crawl decisions can flip in either direction; re-check production rules and run regression tests before and after upgrading.
• Route collisions: If you have a /robots.txt file in public/, Laravel will serve it instead of the route—delete the static file to avoid confusion.
• Caching pitfalls: The package caches output per request unless you add custom caching; for high-traffic apps, wrap the config closure in Cache::remember() or use Laravel’s response caching middleware.
• Case sensitivity: User-agent token matching is defined by robots.txt semantics, but avoid inventing version-specific tokens such as Googlebot/2.1; use the product token (Googlebot) or * unless you have verified the bot’s emitted user-agent handling.
• Config fallbacks: Rules from the config file are applied before configureUsing() callbacks—combine both carefully to avoid unintentional overrides.
• Debugging: Run php artisan route:list | grep robots to confirm the route is registered. Use dd() inside configureUsing() to inspect runtime context (e.g., config('app.env')). After upgrading, dump the rendered robots.txt and test representative URLs against your intended allow/disallow rules.
• SEO impact: Missing or malformed robots.txt can block indexing—always validate responses in tools like Google Search Console’s robots.txt Tester, especially after a precedence-related release.