Robots Txt Bundle Laravel Package

Technical Evaluation

Architecture fit — poor, with a framework mismatch to resolve first. Despite being distributed via Composer as a PHP package, evheniy/robots-txt-bundle is a Symfony2 bundle, not a Laravel package. It has no Illuminate\Support\ServiceProvider, no Laravel auto-discovery entry, and no config publish tags. Its extension points are Symfony's Bundle, DependencyInjection/Extension, and Resources/config/services.yml conventions. There is no drop-in path into a Laravel application; any adoption is really a port, not an integration.

What it actually does. The value proposition is narrow: serve a configurable robots.txt from application config rather than a hand-maintained static file, so rules can vary by environment (e.g., Disallow: / in staging, open crawl in production). That is a legitimate requirement, but it is also roughly 50–100 lines of code in a modern Laravel app.

Integration feasibility — low as a dependency, high as a reference. Two possible reuse paths:

Technical risk — high, dominated by abandonment and platform age.

• Last release 2015-11-04, ~a decade stale. Symfony 2.x has been end-of-life for years, so the package's own test matrix and CI badges describe an environment nobody runs today.
• 5 stars, 0 dependents — effectively no user base, which means no incidental bug-finding and no community answers.
• PHP version constraints in composer.json are likely pinned to 5.x and will conflict with a modern Laravel composer.json (php: ^8.2, Symfony 6/7 components). Expect dependency resolution failures at install time, not just runtime surprises.
• Low intrinsic security exposure (robots.txt is public, read-only), but an unmaintained package in the dependency graph is still supply-chain and audit noise.

Key questions to close before committing engineering time:

1. Is the underlying requirement environment-specific rules, dynamic rule generation, or simply "stop hand-editing a text file"? Only the second justifies code at all.
2. Are the generator classes genuinely framework-agnostic, or do they import Symfony DI/Config? A 30-minute spike answers this.
3. Does it support Allow:, Crawl-delay, Sitemap:, and per-user-agent blocks — or only a flat disallow list? Feature gaps change build-vs-port math.
4. Is there any internal policy requiring provenance/maintenance checks on new dependencies? If yes, this fails on sight.
5. Do we need multi-domain or multi-tenant robots.txt (per-host rules)? The bundle is single-app scoped; that would force a rewrite anyway.

Verdict: do not adopt as a dependency. Treat it as a design reference and implement natively.

Integration Approach

Stack fit — no direct fit; reimplement inside Laravel's routing and config layers. The idiomatic Laravel equivalent is a routes/web.php entry returning a text/plain response built from config/robots.php, or a static public/robots.txt when rules never vary. This preserves the bundle's only real idea (config-driven rules) while using first-class framework primitives.

Migration path. There is no migration — there is a substitution:

1. Capture intent. Read the bundle's README and DependencyInjection/Configuration.php (or equivalent) to enumerate the exact config surface: user-agent blocks, disallow paths, sitemap lines, environment toggles.
2. Re-express as Laravel config. config/robots.php with environment-aware defaults; use .env overrides for staging no-index behavior so the rule is visible in deployment config, not buried in code.
3. Serve it. Either a route with response()->make($content, 200, ['Content-Type' => 'text/plain']), or a build step writing public/robots.txt. Prefer the static file plus a deploy-time generator if traffic is high — it removes PHP from the request path entirely.
4. Cache. Cache::rememberForever keyed on the config hash, invalidated on deploy. robots.txt is fetched by every crawler on every host; it should never hit the framework cold.

Compatibility. The only portable piece is likely the string-building logic, and only if it has no Symfony imports. Laravel already depends on symfony/http-foundation, so a Response-returning generator may drop in — but a Symfony bundle class will not boot, will not be discovered, and its service definitions will never load. Plan for zero shared runtime code.

Sequencing (suggested 1–2 day effort):

1. Requirement confirmation and rule inventory (0.5 day) — includes the framework-agnosticism spike on the package.
2. Write config/robots.php + generator class + route/static-file mechanism (0.5 day).
3. Tests: assert production rules, staging Disallow: /, correct Content-Type, no route shadowing of static assets (0.5 day).
4. Deploy verification: fetch /robots.txt per environment, confirm CDN/cache headers, confirm Search Console can read it (0.5 day).
5. Record the decision in an ADR noting the package was evaluated and rejected — important, because the same package will resurface in future "quick win" searches.

Operational Impact

Maintenance. Near-zero once implemented natively — it is a config file and a dozen lines of routing, owned by whoever owns SEO/SEO-adjacent routing. The alternative, carrying the bundle, creates a permanent exception in dependency-update workflows: every PHP or Laravel upgrade forces a manual audit of a package that will never publish a compatible release.

Support. None available. Five stars and zero dependents mean no Stack Overflow corpus, no issue-tracker triage, no upstream fixes. Any bug is our bug, and the original author's context is a decade gone.

Scaling. Genuinely trivial — robots.txt is a tiny, highly cached, read-only response. Serve it as a static file behind a CDN and it costs nothing. This is an argument against adding a dependency: the workload is too small to justify supply-chain surface area. If rules become per-tenant or per-host, the shape changes and a custom implementation was the right call from the start.

Failure modes to design against:

• Environment leakage — staging Disallow: / deployed to production, de-indexing the site. This is the highest-severity failure and is a config problem the bundle does not solve. Enforce with a post-deploy smoke test that asserts the production rule set.
• Route shadowing — a dynamic route conflicting with a static public/robots.txt, producing unexpected content or a 404. Laravel serves public/ first by design; be explicit about which mechanism owns the file.
• Wrong Content-Type — served as text/html, which some crawlers tolerate and some do not. Assert text/plain in tests.
• Malformed directives — unsupported Allow:/wildcard syntax or a missing blank line between user-agent groups silently changes crawl behavior with no error surfaced. Validate the generated output in a unit test against expected literal text.
• Stale cache — cached rules surviving a deploy that changed them. Key the cache on config content, not a static string.

Ramp-up. Very low for a Laravel engineer — under half a day to understand and extend. The real cost is not learning the package; it is the recurring cost of not adopting it: every future engineer who finds this package in a search spends time re-evaluating it. Documenting the rejection is the cheapest mitigation.

Bottom line for the plan: score 0.355 with a 2015 release date and 0 dependents is an accurate signal. The requirement is real and small; the package is neither. Allocate a day to build it natively in Laravel and close the item.