Robots Txt Bundle Laravel Package

Getting Started

1. Install and pin deliberately. The package is Symfony2-era (last release 2015-11-04), so composer require evheniy/robots-txt-bundle will only resolve cleanly on a Symfony 2.x app with a compatible PHP (5.5–7.x) constraint. In a modern app, expect to install it with --ignore-platform-reqs at best, or to fork it. Check composer show evheniy/robots-txt-bundle before committing.
2. Register the bundle in app/AppKernel.php:

   new Evheniy\RobotsTxtBundle\RobotsTxtBundle(),
   

3. Look here first, in this order:
   • DependencyInjection/Configuration.php — the authoritative list of config keys (user agent, disallow/allow paths, sitemap, host, crawl-delay, etc.). This is where the package's actual API lives.
   • Resources/config/routing.yml (and services.yml) — tells you whether the bundle exposes a /robots.txt route or a service you call from your own controller.
   • README.md "Usage" — examples are sparse, so the DI config tree is your real documentation.
4. First use case: environment-aware crawl policy. Wire rules per environment so staging and dev are never indexed:

   # app/config/config.yml (default = locked down)
   robots_txt:
       user_agent: '*'
       disallow: ['/']
   
   # app/config/config_prod.yml
   robots_txt:
       user_agent: '*'
       disallow: ['/admin', '/login', '/search']
       sitemap: 'https://example.com/sitemap.xml'
   

   Clear the cache (app/console cache:clear --env=prod) and hit /robots.txt in a browser to confirm the output before touching production.

Implementation Patterns

• Config-per-environment is the primary workflow. Keep a restrictive default in config.yml and open up only in config_prod.yml. This removes an entire class of "staging got indexed" incidents without code changes.
• Serve through the bundle route rather than a static file if you want rules to follow config. If a physical web/robots.txt exists, the web server will usually win — delete it or you will spend an afternoon debugging "why isn't my config applied?"
• Multi-agent blocks. Where the config tree allows it, define separate groups so you can be permissive to major crawlers and stricter to others (e.g. block aggressive scrapers with their own user-agent section rather than a global Disallow).
• Keep sitemaps absolute. sitemap: https://example.com/sitemap.xml, never /sitemap.xml. Some crawlers tolerate relative URLs; many don't.
• Treat robots.txt as part of the deploy artifact / smoke test. A functional test that requests /robots.txt and asserts 200 plus a marker string (e.g. your sitemap URL) catches route regressions that unit tests miss:

  $client = static::createClient();
  $crawler = $client->request('GET', '/robots.txt');
  $this->assertSame(200, $client->getResponse()->getStatusCode());
  $this->assertContains('Sitemap: https://example.com/sitemap.xml', $client->getResponse()->getContent());
  

• Pair with your sitemap generator. If you already produce sitemaps (or generate them from a controller), reference them here so the two features stay consistent. Robots rules and sitemaps should be configured in the same place conceptually, even if different bundles.
• Cache awareness. Because output is config-derived, changes require a cache clear in prod. Add a note to your deploy checklist, or wrap the controller output in a short-lived HTTP cache (Cache-Control: max-age=3600) rather than rebuilding it per request.
• Escape hatch for one-off rules. If you need comments or exotic directives the config tree doesn't model, override the bundle's template or replace the route with a tiny controller returning a hand-authored string — don't force the package beyond its vocabulary.

Gotchas and Tips

• Abandonment risk is the headline. Five stars, zero dependents, 2015 release. Before adopting, ask whether fifteen lines of controller + a Twig template (or a config parameter rendered once at deploy) would do the same job with no dependency. If you adopt, consider vendoring or forking.
• Symfony2 only. It is not compatible with Symfony 3+, let alone 4/5/6. Do not assume Flex recipes, autowiring, or services.yaml conventions.
• PHP version constraints will fight modern Composer. Pin explicitly in composer.json and document the decision so a composer update doesn't silently drop the bundle.
• Config key names are the #1 source of "it does nothing." The README is thin — read Configuration.php for the exact keys. A typo in the tree may be rejected by Symfony's config validator (good) or, worse, silently ignored if the tree is loosely defined.
• Blank lines and trailing newlines matter. A robots.txt with a blank line in the middle of a user-agent group terminates that group for some crawlers. If you post-process the output, preserve a single trailing newline and no stray blank lines.
• Disallow: vs Disallow: /. An empty Disallow allows everything; Disallow: / blocks everything. This is the classic staging-leak mistake — assert your test environment output explicitly.
• Don't rely on robots.txt for security. Disallow is advisory. Admin paths must still be behind auth; robots rules only reduce crawl noise and accidental indexing.
• Static file shadowing. Nginx/Apache serve existing files before hitting PHP. Confirm with curl -I https://example.com/robots.txt that Content-Type is text/plain and the body matches your config (a stale web/robots.txt will show Last-Modified from deploy, not request time).
• Extension points. For customizations, prefer (in order): overriding the bundle's template, overriding the service/controller in your app config, then a compiler pass. Avoid editing vendor/ — a fresh composer install will erase it.
• Debugging checklist: cache:clear the correct env → verify route exists (app/console debug:router | grep robots) → curl -I and curl the URL → if output is stale, check for a physical file and any reverse-proxy/CDN cache in front of the app.
• Decommissioning is cheap. If you later replace this with a static file or your own controller, you only need to remove the kernel entry and the config blocks — there is no database or migration surface.