Robots Extension Laravel Package

Getting Started

This is a PyroCMS / Streams Platform addon, so the first step is confirming you actually run that stack. A Laravel app alone will not resolve the service provider, migrations, or admin views without the Streams Platform base.

Minimal path:

composer require anomaly/robots-extension
php artisan addon:install anomaly/robots-extension

Then look in this order:

1. src/RobotsExtensionServiceProvider.php — what it registers (routes, view namespaces, config, migrations).
2. resources/config/ — the settings the addon exposes and their defaults.
3. resources/views/ and the addon's migrations/ — the admin surface and any persisted directives.
4. routes.php (or the service provider's route block) — confirm whether it serves /robots.txt itself or expects your web server to do it.

First use case: let editors manage robots.txt content (or its key directives) from the control panel rather than editing a static file, and preview the result per environment. Verify by hitting /robots.txt and matching the admin output exactly.

Implementation Patterns

Environment-aware directives. The most valuable pattern with any robots addon is varying output by environment: Disallow: / on staging and local, a curated allow/disallow set in production. Configure this once in the addon settings (or a small config/robots.php override in config/ if the addon merges config), and keep the per-environment values in .env so staging never ships crawlable.

Composer-only, no route hijack. If your webserver already serves public/robots.txt, let it. Use the addon only to generate or edit that file, and treat the served file as the source of truth. Prefer this over an application route — it removes a request from the PHP stack and avoids cache/header surprises. Only use the route-based approach if you need dynamic per-host directives (multi-tenant, domain aliases).

Keep sitemap references explicit. Production robots.txt should include Sitemap: https://example.com/sitemap.xml. If you also run a sitemap addon, wire the two together (or hard-code the absolute URL) rather than relying on the robots addon to infer it — inference rarely works.

Admin-editable, deploy-guarded. Let the client edit directives in the CP for day-to-day tweaks, but include a deploy step that snapshots the current output (or seed the settings from config) so a bad edit is one rollback away. Add a test asserting the production output contains your required Sitemap: line and does not contain Disallow: /.

Middleware-adjacent pattern. If the addon does register a route, gate it behind throttle and a cache header (Cache-Control: public, max-age=3600) so bots and CDNs don't hammer PHP for a static string.

Gotchas and Tips

• Adoption is effectively zero (0 stars, 0 dependents). Treat every assumption above as needing verification against the actual source before you commit. Budget time to read the addon end-to-end — it is likely small enough to audit in under an hour, and you should.
• Pin the version. With no dependents, upstream regressions won't be caught by anyone else. Use an exact constraint ("anomaly/robots-extension": "1.x.y") and commit the lockfile.
• Static file vs. route collision. If public/robots.txt exists and the addon registers a route, the webserver usually wins and your CP edits silently never reach production. Decide which layer owns the file, then delete the other.
• Caching layers. robots.txt is aggressively cached by CDNs, proxies, and browsers and by search engines themselves (crawlers may not refetch for hours/days). After changes, purge CDN/OPcache caches and validate with curl -I plus Google Search Console's robots tester; do not assume a bad directive is live immediately, in either direction.
• Trailing slash and wildcard semantics. Disallow: /admin matches /admin and /adminfoo; write Disallow: /admin/ unless you want the prefix behavior. Wildcards (*) and $ are supported by major crawlers but not universally — keep rules simple.
• Accidental lockdown. The single most common incident is a staging Disallow: / leaking into production. Guard it with an explicit environment check rather than a copy-pasted settings record, and alert on it in CI.
• Admin permission surface. Confirm the addon registers proper permission keys; otherwise any CP user can rewrite crawl rules. If it doesn't, wrap the settings screen behind your own gate.
• Extension points. Whatever hooks exist (settings events, view overrides via resources/views/vendor/, or config merge), prefer those over editing vendor files — the addon is unmaintained enough that you may eventually fork it, and clean overrides make that trivial.
• Fork readiness. Given the repository signals, keep a fork (or a vendored copy) ready. A 30-line Laravel service provider that reads directives from config and writes public/robots.txt on deploy is a legitimate replacement if the addon stops working with your Streams version.