Symfony Bundle Laravel Package

Getting Started

• Install the package with Composer, then skim the service provider to see what gets bound and whether the robots.txt route is registered automatically or must be opted into.
• Publish the package config (php artisan vendor:publish) before touching anything else. The generated config is the source of truth for how bot rules are merged and rendered.
• Open RobotsFilter and read the AI_BOTS constant first. That list is the package's default registry of AI user-agents (now including ExaSearchBot as of v0.9.3) and is the thing most applications will interact with.
• First use case: serve a robots.txt that disallows AI crawlers while leaving normal search engines alone. Wire the filter into whatever route/controller the package documents, hit /robots.txt in the browser, and confirm the output before adding any custom rules.
• If your app is on a non-production environment, check how the package behaves by default (empty response, global disallow, or environment-specific rules) — do not assume production output until you've seen the dev output.

Implementation Patterns

• Treat AI_BOTS as a baseline, not a policy. Build your own list of allowed/disallowed agents in config, and use the package's registry as the default set you selectively override. This keeps you resilient when the package adds new agents in a patch release.
• Explicit site rules win. The v0.9.3 note makes this explicit for ExaSearchBot: if a site already names that user-agent, the site's rule is kept rather than replaced by the package default. Lean on this — declare the handful of agents you care about explicitly, and let the registry cover the long tail.
• Wrap the filter in a thin application service. Instead of calling RobotsFilter directly from a controller or Blade view, expose something like a RobotsTxt class or a route closure that composes: global disallows → environment rules → registry defaults → sitemap directive. This gives you one place to test and one place to change ordering.
• Per-environment generation. Disallow everything outside production, or serve a minimal allow-all file in local so local crawlers/scrapers aren't confused. Gate this on app()->environment() inside the composing service, not inside the vendor class.
• Feature-test the output. $this->get('/robots.txt')->assertSee('User-agent: ExaSearchBot') and similar assertions catch accidental regressions when config or the bot list changes. Assert on User-agent: blocks and Disallow: lines rather than exact full-file string matches.
• Custom rules via the filter's extension points. If the package exposes a way to register additional agents or directives, use that rather than hand-rolling string concatenation on top of the rendered output — otherwise a future format change will break you.
• Pair with sitemap generation. The same route/controller should emit the Sitemap: directive; keeping robots and sitemap concerns in one class makes it obvious when a sitemap path changes.

Gotchas and Tips

• This is pre-1.0. v0.9.3 is a patch, but 0.x semver permits breaking changes in minor bumps. Pin to a caret range you've tested and read the changelog on every update, especially for changes to AI_BOTS.
• The AI bot list changes silently on upgrade. Adding ExaSearchBot changes the generated file with no code change on your side. After any composer update, diff your /robots.txt output against the previous version — this is the most likely source of surprise.
• User-agent matching is exact-string, and casing matters. ExaSearchBot is not the same as exasearchbot in most implementations. Don't hand-edit the casing of names you copy out of AI_BOTS.
• Ordering within a User-agent group is significant, and rule precedence (Allow vs Disallow, longest-match) is defined by the robots.txt spec, not the package. If your rules seem ignored, check for a conflicting group for the same agent higher up in the file.
• Explicit blocks override defaults — verify this is what you want. Because a site naming ExaSearchBot keeps its own rule, an old hand-written Allow: / for that agent will survive the v0.9.3 registry addition. Grep your config/templates for agent names before assuming the package default is in effect.
• CDN and proxy caching. robots.txt is aggressively cached. After changing rules, purge your CDN and confirm with curl -I that you are not serving a stale copy — otherwise you'll debug the package for an hour when the problem is at the edge.
• robots.txt is not access control. Blocking AI crawlers here stops polite crawlers only. Do not treat a Disallow as protection for sensitive routes; keep real authorization in middleware.
• Watch for a route conflict. If you already define /robots.txt (static file in public/, nginx rule, or another package), the vendor route may never be hit. Remove the static file or check php artisan route:list to see which handler actually wins.
• Check the published config is actually loaded. A stale cached config (php artisan config:cache) is the usual culprit when the AI_BOTS additions don't appear locally but do in production, or vice versa.