Robots Txt Bundle Laravel Package

Getting Started

This is a Symfony bundle, not a Laravel package — and v2.0.1 doesn't change that. The release line supports PHP >= 7.1 and Symfony 2.x | 3.x | 4.x only: PHP 7.1 went EOL in 2019, Symfony 2/3 are long dead, and Symfony 4.4 left maintenance in November 2023. There's no Laravel service provider, no facade, no config publishing — just a Symfony DI extension and a YAML config tree. Do not composer require it into a Laravel app; you'll pull in symfony/dependency-injection, symfony/http-kernel, and symfony/routing at versions that collide with Laravel's container and autoloader.

Treat the repo as a reference implementation. The whole thing is roughly 150 lines of string concatenation wrapped in Symfony plumbing. Porting it into a small Laravel service is cheaper than fighting the dependency graph, and the port will outlive the package.

What 2.0.1 actually tells you: the version bump plus "100% code tests coverage" is the useful signal here. The test suite is now a reliable executable spec — read the assertions in Tests/ before the docs, and copy the expected-output fixtures into your own Pest/PHPUnit tests. That's the highest-value thing this release gives you.

Where to look first in the repo:

1. Tests/ — with 100% coverage claimed, the fixtures are the authoritative output format (ordering, blank lines, trailing newline, Disallow: vs omitted lines).
2. DependencyInjection/Configuration.php — the config tree. This is the entire "spec": host, per-user-agent disallow / allow lists, crawl_delay, sitemap.
3. Resources/config/routing.yml — maps /robots.txt to a controller action. Nothing clever.
4. The generator service — walks config and concatenates lines. Copy this logic verbatim.

First use case in Laravel: replace your static public/robots.txt with a generated, environment-aware response so staging never gets indexed and production always advertises your sitemap.

Minimal port:

// routes/web.php
Route::get('/robots.txt', RobotsTxtController::class)->name('robots');

// config/robots.php
return [
    'sitemap' => env('APP_URL').'/sitemap.xml',
    'groups' => [
        ['user_agent' => '*', 'disallow' => ['/admin', '/cart', '/search'], 'allow' => []],
    ],
];

// app/Http/Controllers/RobotsTxtController.php
public function __invoke(): Response
{
    if (! app()->environment('production')) {
        return response("User-agent: *\nDisallow: /\n", 200)
            ->header('Content-Type', 'text/plain; charset=UTF-8');
    }

    $lines = [];
    foreach (config('robots.groups') as $group) {
        $lines[] = 'User-agent: '.$group['user_agent'];
        foreach ($group['disallow'] as $path) {
            $lines[] = 'Disallow: '.$path;
        }
        foreach ($group['allow'] as $path) {
            $lines[] = 'Allow: '.$path;
        }
        $lines[] = '';
    }
    if ($sitemap = config('robots.sitemap')) {
        $lines[] = 'Sitemap: '.$sitemap;
    }

    return response(implode("\n", $lines)."\n", 200)
        ->header('Content-Type', 'text/plain; charset=UTF-8');
}

Immediately delete public/robots.txt — see Gotchas.

Implementation Patterns

1. Config-shaped generator, not hardcoded strings. Mirror the bundle's config array (groups → user agent → allow/disallow → sitemap) in config/robots.php. It keeps crawl policy reviewable in a diff and lets ops change it without touching code. Add a Generator class with generate(): string so it's testable and swappable:

final class RobotsTxtGenerator
{
    public function __construct(private array $config) {}

    public function generate(): string
    {
        return collect($this->config['groups'])->map(function ($g) {
            return collect(["User-agent: {$g['user_agent']}"])
                ->merge(collect($g['disallow'] ?? [])->map(fn ($p) => "Disallow: {$p}"))
                ->merge(collect($g['allow'] ?? [])->map(fn ($p) => "Allow: {$p}"))
                ->implode("\n");
        })->push($this->config['sitemap'] ? "Sitemap: {$this->config['sitemap']}" : null)
          ->filter()->implode("\n\n");
    }
}

2. Environment gating is the whole point. The bundle's real value is that dev/staging output differs from prod. Two strategies:

• Blunt (preferred): off-production, emit only User-agent: * + Disallow: /. Best when staging shares a domain family with prod.
• 404 approach: return 404 for /robots.txt off-production. Crawlers treat 404 as "no restrictions, crawl freely," so this is worse than disallow-all unless you also gate with basic auth. Prefer Disallow: /.

3. Multiple user-agent groups. Real-world use: block aggressive crawlers (AhrefsBot, SemrushBot) while leaving Googlebot untouched, or Allow: /api/public/ while Disallow: /api/ for *. Put * last; Google resolves by longest match, but Bing and others have historically been order-sensitive.

4. Cache the rendered string. robots.txt is hit constantly by bots. Cache::remember('robots.txt', now()->addHour(), fn () => app(RobotsTxtGenerator::class)->generate()). Invalidate on deploy or when sitemap config changes. Do not bake the host into the cached value if you serve multiple domains — cache per host, or omit Host: entirely.

5. Static-file variant for CDN/edge. If robots.txt is served by nginx/Cloudflare before PHP runs, the route never fires. In that setup, generate the file with an Artisan command and write it to public/:

// app/Console/Commands/BuildRobotsTxt.php
public function handle(RobotsTxtGenerator $gen): int
{
    File::put(public_path('robots.txt'), $gen->generate());
    return self::SUCCESS;
}

Wire it into your deploy pipeline. This is the pattern most Laravel teams actually want, and it sidesteps the static-shadowing gotcha entirely.

6. Multitenancy / multi-domain. Derive Sitemap: (and Host:, if you use it) from request()->getSchemeAndHttpHost() rather than config, and key the cache by host. The bundle used a static host config value — that breaks the moment you add a second domain.

7. Testing — port the bundle's fixtures. This is where 2.0.1 earns its keep. With 100% coverage, Tests/ encodes exactly how the generator treats empty arrays, ordering, and blank lines. Lift those expectations into your own suite:

it('disallows admin and advertises sitemap', function () {
    $this->get('/robots.txt')
        ->assertOk()
        ->assertHeader('Content-Type', 'text/plain; charset=UTF-8')
        ->assertSee("Disallow: /admin")
        ->assertSee("Sitemap: https://example.com/sitemap.xml");
});

it('blocks everything outside production', function () {
    $this->app['env'] = 'staging';
    $this->get('/robots.txt')->assertSee("Disallow: /\n");
});

8. Integration with sitemap generation. If you already use spatie/laravel-sitemap, read the sitemap URL from that package's config so the two never drift. Same for any X-Robots-Tag middleware — keep them consistent.

Gotchas and Tips

• A static public/robots.txt silently wins over your route. Both php artisan serve and nginx serve existing files before dispatching to Laravel. Delete the file, or pass /robots.txt through to PHP. This is the single most common "my dynamic robots.txt doesn't work" cause.
• Content-Type matters more than you'd think. Google is lenient, but Yandex and several SEO tools reject text/html. Always set text/plain. Laravel's response() defaults to text/html — override it explicitly.
• Disallow: with an empty value means "allow everything." Omitting the Disallow line entirely is different from Disallow: /. The bundle could emit either depending on config — check the 2.0.1 fixtures, then decide deliberately and test it.
• Sitemap: must be an absolute URL. Relative paths are silently ignored by every crawler. Same for Host:.
• Host: is Yandex-only. Google ignores it, Bing deprecated it. Including it is harmless; don't build logic around it. The bundle still supports it — you almost certainly don't need it.
• Crawl-delay is ignored by Google and advisory at best for Bing. Don't rely on it to throttle anything; rate-limit at nginx or Cloudflare instead. The bundle still exposes crawl_delay in config — porting it is fine, expecting it to work is not.
• Query strings: Disallow: /search blocks /search?q=foo too, because the rule matches the path including the query. Disallow: /search? is the explicit "block all parameterised URLs" form. Be precise about which you mean.
• CDN/edge caching. Cloudflare caches robots.txt aggressively. After a rules change, purge the URL and verify with curl -sI https://example.com/robots.txt | grep -i cf-cache-status. Bots can see stale rules for hours.
• Don't accidentally noindex the robots.txt response. If you have global middleware adding X-Robots-Tag: noindex, confirm it doesn't apply here (usually harmless, always confusing during an audit).
• The "supports PHP >= 7.1" constraint is open-ended but misleading. The declared ceiling is Symfony 4, and Symfony 2/3 components do not run on PHP 8 at all. Do not read 2.0.1 as PHP 8 support; nothing in the release notes suggests the DI extension, YAML config tree, or routing loader were modernised. No breaking changes for a Laravel consumer — because you aren't consuming it.
• Extension point that's actually useful: the generator is a standalone service. In Laravel, bind RobotsTxtGenerator to an interface in a service provider so a tenant-specific or feature-flagged implementation can replace it without touching the controller.
• Debugging checklist when a bot complains: curl -sI for status and Content-Type, curl -s for the exact body (check for a stray BOM or missing trailing newline), confirm no static file shadows the route, confirm the edge cache is cold, and confirm APP_ENV is what you think it is in that process (php artisan about).