Robots Bundle Laravel Package

Getting Started

First, the honest orientation step: this package will not run in a Laravel app. fourlabs/robots-bundle is a Symfony bundle — a Bundle subclass, a DependencyInjection extension, a kernel.response event subscriber, and Doctrine annotations for metadata. Laravel has no bundles, no kernel.response, and no annotation router. composer require fourlabs/robots-bundle will not boot; there is nothing to configure.

So the "getting started" work is reading it as a reference spec, not installing it. Read exactly two things in the repo:

1. The response listener — that is the entire product. Everything else is plumbing.
2. The annotation class — that gives you the directive vocabulary it supports.

The portable contract is tiny: read a per-route marker, project it onto an X-Robots-Tag response header, with a global default. That is one middleware and one attribute in Laravel, roughly 100 lines:

#[Attribute(Attribute::TARGET_CLASS | Attribute::TARGET_METHOD)]
final class Robots
{
    public function __construct(public string $value) {}
}

final class ForceRobotsHeader
{
    public function handle(Request $request, Closure $next): Response
    {
        $response = $next($request);

        if ($value = $this->resolve($request)) {
            $response->headers->set('X-Robots-Tag', $value);
        }

        return $response;
    }

    private function resolve(Request $request): ?string
    {
        $route = $request->route();
        if (! $route) {
            return config('robots.default');
        }

        // 1. Route defaults beat everything — cheapest and most explicit at the call site.
        if ($value = ($route->defaults['robots'] ?? null)) {
            return $value;
        }

        // 2. PHP attribute on the controller method, falling back to the class.
        $uses = $route->getAction('uses');
        if (is_array($uses) && is_string($uses[0] ?? null)) {
            $method = new ReflectionMethod($uses[0], $uses[1] ?? '__invoke');

            $attr = ($method->getAttributes(Robots::class)[0] ?? null)
                 ?? ($method->getDeclaringClass()->getAttributes(Robots::class)[0] ?? null);

            if ($attr) {
                return $attr->newInstance()->value;
            }
        }

        return config('robots.default');
    }
}

Register it as global middleware in bootstrap/app.php (or Kernel.php on older versions):

->withMiddleware(function (Middleware $middleware) {
    $middleware->append(ForceRobotsHeader::class);
})

And a publishable config/robots.php:

return [
    'default' => env('APP_ENV') === 'production' ? null : 'noindex, nofollow',
];

Your first use case should be environment gating, not per-route work. Ship the middleware with only the global default — every non-production host emits noindex, nofollow. That single config line prevents the most common and most expensive incident in this space (staging, preview, or white-label tenant domains outranking production), it requires zero controller changes, and it gives you a live, tested code path before you ask anyone to annotate anything.

Then pick one real route — a filtered search page, a gated PDF, an internal search result — and prove the attribute path end-to-end with curl -sI.

Implementation Patterns

Define the precedence ladder once and write it in the README of your own middleware. The bundle's implicit order is method annotation → class annotation → framework default. Extend it to: route default → method attribute → class attribute → route-group default → config default. Ordering disputes are the number one source of "why is this page noindex" tickets; make the answer a table, not a debugging session.

Merge, never blind-set. The bundle's weakness is that headers->set() clobbers. Multiple sources can contribute here — your middleware, a controller setting it manually, an edge worker. Union the directives:

private function merge(Response $response, string $value): void
{
    $existing = array_filter(array_map('trim',
        explode(',', $response->headers->get('X-Robots-Tag', ''))
    ));

    $directives = collect($existing)
        ->merge(array_map('trim', explode(',', $value)))
        ->filter()
        ->unique()
        ->implode(', ');

    $response->headers->set('X-Robots-Tag', $directives);
}

Prefer rewriting that function to array_unique on the whole comma-joined string if you ever need to support user-agent:-scoped directives — those take the form googlebot: noindex, nofollow and must stay on their own header line.

Use the right tool per case. The day-to-day value is the split:

Support route defaults as a first-class citizen. ->defaults('robots', 'noindex, nofollow') survives php artisan route:cache, needs no reflection, and keeps the declaration in routes/web.php next to the route — which is exactly the reviewability argument for the whole pattern. Reach for the attribute only on controller-heavy resources where the route file is already noisy.

Cover the non-HTML cases deliberately. Register the middleware globally, not on the web group, or your API routes and file-download endpoints — the ones where the header does the work robots.txt cannot — will be the ones that miss it. If you must be selective, use a route-group middleware around api and the download routes rather than relying on the default web group.

Handle error responses. Middleware that returns $next($request) never sees the rendered exception page, because exceptions unwind past it to the kernel's handler. If 404s and 500s should be noindex (they usually should, at least in staging), set it in withExceptions():

->withExceptions(function (Exceptions $exceptions) {
    $exceptions->respond(function (Response $response) {
        if (config('robots.default')) {
            $response->headers->set('X-Robots-Tag', config('robots.default'));
        }
        return $response;
    });
})

Ship it dark first. Add a shadow mode that computes the intended value, logs a diff against what was actually sent, and emits X-Robots-Tag-Shadow rather than the real header. Run it through one crawl cycle, grep the logs, then flip enforcement. Indexability regressions are commercial incidents, and logs are cheap.

Inventory existing emission before you write anything. grep -rn "X-Robots-Tag" app/ and check the CDN/edge config. Existing ad-hoc $response->headers->set() calls are the real scope of the migration, and an edge rule that rewrites or strips the header will silently invalidate all your testing.

Test it like a security control, because it is one:

it('marks filtered listings noindex', function () {
    $this->get('/products?filter=sale')->assertHeader('X-Robots-Tag', 'noindex, nofollow');
});

it('does not mark the homepage noindex in production', function () {
    $this->app['env'] = 'production';
    $this->get('/')->assertHeaderMissing('X-Robots-Tag');
});

it('does not noindex 404s in production', function () { /* ... */ });

That negative test is the one that saves you. Make it a required check.

Gotchas and Tips

The package description is wrong, and it will mislead your planning. The README claims environment-aware robots.txt rules, multiple hosts, sitemap and crawl-delay configuration. The actual bundle sets one response header from an annotation. Do not scope work against the description.

Do not install it "just to try." Pulling an unmaintained Symfony bundle with an active doctrine/annotations dependency into a Laravel composer.json buys you a dependency your dependency-audit tooling will flag, a transitive pin on a deprecated library, and zero working code — Laravel has no kernel.response to subscribe to. Vendor the ~80-line listener with MIT attribution if you want provenance, and delete the rest.

The classic footgun: Disallow plus noindex. If robots.txt blocks a path, Googlebot never fetches it, so it never reads your X-Robots-Tag: noindex, and the URL can still appear in results (as a bare link, no snippet). These two mechanisms must not be applied to the same URLs. Audit for it explicitly — it is the single most common configuration error in this area.

Redirects carry the header on the redirect itself. A 301 to /login with noindex does nothing useful: Google follows the redirect and evaluates the destination, which needs its own policy. If unauthenticated users get bounced to a login page, the login page is what needs the noindex, not the gated route.

Multiple header lines vs one. X-Robots-Tag: noindex + X-Robots-Tag: nofollow on separate lines is valid and Google merges them, but only the comma-joined single line is universally understood by crawlers and log parsers. Exception: user-agent:-scoped directives (googlebot: noindex) must be their own line — a comma-joined googlebot: noindex, nofollow scopes the entire string to Googlebot, not just the first directive, which is rarely what you meant.

Conflicting directives resolve to the most restrictive. If your middleware emits noindex and a CDN rule injects index, noindex wins — which is safe for gating but means a stale CDN rule cannot rescue an over-eager app rule. Conversely, a CDN rule that strips your header is invisible from the app and is the most likely reason "the header isn't there in production."

Case sensitivity is a trap at the edge, not in Laravel. Symfony's and Laravel's HeaderBag are case-insensitive, so headers->get('X-Robots-Tag') finds x-robots-tag. Your Nginx proxy_hide_header / Fastly VCL / Cloudflare Transform Rule probably is not. Test against the real edge, not php artisan serve.

Middleware ordering matters for 304s. An X-Robots-Tag on a 304 Not Modified is not read by crawlers (the cached response is reused), and ETag/Cache-Control middleware that short-circuits can return before yours runs. Register the robots middleware outermost ($middleware->prepend(...) in the global stack) so it observes the final response object regardless of what produced it.

Route caching is fine; static memoization is not. Reflection on attributes works under route:cache because attributes live on the class definition. But if you memoize resolved headers into a static array keyed by route name "for performance," remember to key it on something that changes when the deploy does — or you will ship a policy change and watch the old header keep serving until a PHP worker recycles.

Closures and invokable controllers need explicit handling. $route->getAction('uses') returns a Closure for closure-routed endpoints — guard with is_array($uses) as above, and fall back to route defaults for those. For invokable controllers, $uses is a single-element array and the method is __invoke.

Watch the interaction with Vary and cached HTML. X-Robots-Tag is per-route, so it does not need Vary, but it is cached with the response. Changing an indexability policy means purging the CDN, not just deploying — otherwise you're left reasoning about which POP has which policy.

unavailable_after uses RFC 822, and timezones bite. Format it explicitly (gmdate('D, d M Y H:i:s \G\M\T', $ts)) rather than passing a Carbon instance through strval(), and never write a test that asserts against now() without freezing time.

Debugging checklist, in order: curl -sI https://host/path | grep -i robots from outside the network → check the CDN's header-rewrite rules → check storage/logs from your shadow-mode pass → php artisan route:list --columns=method,uri,action to confirm the middleware resolved the controller you expect. If the header is present locally and absent in production, it is the edge layer or a stale cache, not your code.

Extension point worth knowing: keep all directive strings in one constants class (Robots::NOINDEX, Robots::MAX_SNIPPET, …) rather than scattered literals. It is the one piece of the bundle's design worth copying, and it makes the "which routes are noindex" audit a single grep.

Do not put marketing-editable policy behind this. Everything here ships on deploy. If a stakeholder needs to toggle indexability without a release, that is a CMS or feature-flag requirement, and answering it with an attribute will produce a ticket every time.