Weave Code
Laravel Robots Middleware Laravel Package
spatie/laravel-robots-middleware
Laravel middleware to control search engine indexing via X-Robots-Tag/robots meta behavior. Extend the base middleware and implement shouldIndex() to allow or block indexing per request (e.g., disable for admin routes), without changing your views.
Enable or disable the indexing of your app
Frequently asked questions about Laravel Robots Middleware
- How do I block search engines from indexing specific Laravel routes (e.g., admin panel) using this package?
- Extend the middleware class and override `shouldIndex()` to return `false` for blocked routes. For example, return `false` if `$request->is('admin/*')`. Register your custom middleware in `Kernel.php` to replace the default. Dynamic rules like user roles or request data can also be applied here.
- Does this package work with Laravel 11.x, or is it limited to older versions?
- The package explicitly supports Laravel 8.0–11.x, with no breaking changes between versions. Check the `composer.json` constraints for your Laravel version. Spatie maintains backward compatibility, so updates are safe for existing projects.
- Can I use this middleware alongside other header-modifying packages (e.g., caching layers or security middleware)?
- Yes, but test for conflicts. Middleware runs in sequence, so later middleware (e.g., caching) may override headers. For CDNs like Cloudflare or Varnish, ensure `X-Robots-Tag` is whitelisted and cached headers are invalidated. Use `debug()` mode to log execution order.
- What’s the difference between HTTP headers (this package) and HTML `<meta>` tags for robots control?
- HTTP headers (via `X-Robots-Tag`) are more reliable for crawlers and work even if JavaScript renders content. `<meta>` tags require HTML parsing and may be ignored by some bots. This package uses headers, which are ideal for Laravel’s API-first or headless apps.
- How do I dynamically allow indexing for logged-in users but block guests?
- Extend the middleware and inject logic into `shouldIndex()`. For example, return `true` if `auth()->check()` and `false` otherwise. Combine with Laravel’s auth middleware or pass request data (e.g., `shouldIndex($request->user())`) for granular control.
- Will this package break my existing robots.txt file or SEO setup?
- No, this package only adds HTTP headers and doesn’t modify `robots.txt`. It’s additive—use both for full control. Audit your current setup with Google Search Console to ensure alignment. Headers take precedence over `robots.txt` for specific paths.
- How do I test this middleware in PHPUnit or Pest without hitting real crawlers?
- Mock the middleware in tests by overriding `shouldIndex()` to return predictable values. Use Laravel’s `actingAs()` for auth tests or `Route::toHttpResponse()` to simulate requests. Verify headers with `$response->headers->get('X-Robots-Tag')`.
- Can I use this for API endpoints (e.g., GraphQL or REST) where HTML `<meta>` tags aren’t applicable?
- Absolutely. HTTP headers work seamlessly for APIs. Block sensitive endpoints (e.g., `/api/private`) by returning `false` in `shouldIndex()`. Test with tools like Postman or cURL to confirm headers are applied to API responses.
- Are there performance implications for adding this middleware globally?
- Minimal. The package is optimized for low overhead, with no database or external calls. If performance is critical, register it only for specific routes (e.g., `Route::middleware(RobotsMiddleware::class)->group(...)`). Benchmark with Laravel’s `route:list` to verify.
- What if I need to exclude certain paths (e.g., `/public/*`) but allow all others by default?
- Use the `allowAllExcept()` method in your custom middleware. For example: `RobotsMiddleware::allowAllExcept(['/public/*', '/admin/*'])`. This inverts the logic, making exceptions explicit. Combine with route-based conditions for flexibility.
Weaver
How can I help you explore Laravel packages today?