Robotstxtparser Laravel Package

Getting Started

composer require vipnytt/robotstxtparser

No service provider, facade, or config file. It is a plain PHP library — construct it where you need it, or bind a wrapper in a service provider if you want it injected.

The first place to look is README.md for the two or three usage snippets, then src/ and tests/. The README is deliberately thin; the test suite is the real specification for wildcard handling, $ anchoring, and longest-match precedence.

Minimal check:

use vipnytt\RobotsTxtParser\RobotsTxtParser;

// URI form: the library fetches <base>/robots.txt itself
$parser = new RobotsTxtParser('https://example.com');
$parser->userAgent('MyLaravelBot');

$parser->isAllowed('/products/42');   // bool
$parser->getSitemaps();               // ['https://example.com/sitemap.xml']
$parser->getCrawlDelay();             // seconds, or null

Day-one use case: a queued scraper or link-checker that must not hammer paths it isn't allowed to touch. Set the user agent before the first isAllowed() call — the UA is the selector, not a parameter, and it drives which group of rules is applied.

If you want Laravel's HTTP client (proxies, retries, Http::fake() in tests) to own the fetch, fetch the body yourself and hand it to the parser/validator rather than letting the URI constructor make its own request. Confirm in src/ whether your pinned version exposes a raw-text constructor; if it doesn't, that's a one-file subclass or wrapper.

Implementation Patterns

Wrap it once, cache the body, not the object.

namespace App\Support;

use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Facades\Http;
use vipnytt\RobotsTxtParser\RobotsTxtParser;

class RobotsTxt
{
    public function for(string $baseUrl): RobotsTxtParser
    {
        $body = Cache::remember('robots:'.md5($baseUrl), now()->addHours(12), function () use ($baseUrl) {
            $res = Http::timeout(10)
                ->withHeaders(['User-Agent' => config('crawler.user_agent')])
                ->get(rtrim($baseUrl, '/').'/robots.txt');

            // 404 => allow everything; 5xx => conservative disallow-all (RFC 9309)
            if ($res->status() === 404) {
                return '';
            }

            if ($res->failed()) {
                throw new \RuntimeException("robots.txt unavailable: {$res->status()}");
            }

            return $res->body();
        });

        return (new RobotsTxtParser($baseUrl))->setContent($body); // raw-text entry point, if available
    }
}

Cache the raw text and re-parse per request. Parsing is cheap; serialising a parser that may hold an HTTP client is not worth the risk. Honour ETag/If-Modified-Since if you crawl the same host heavily, and keep TTLs short (12–24h matches Google's guidance).

Guard the queue, not the HTTP call. Check permission in the job that schedules fetches so you never enqueue work you'll have to discard, and apply crawl-delay with Laravel's rate limiter:

$delay = (int) ceil($parser->getCrawlDelay() ?? 1);

Redis::throttle("crawl:{$host}")
    ->allow(1)
    ->every($delay)
    ->then(fn () => FetchPageJob::dispatch($url), fn () => $this->release(10));

Use it for sitemap and host discovery. getSitemaps() gives you seed URLs for a crawler without hardcoding paths, and the Host directive (Yandex) tells you the preferred canonical host for mirror sites. Feed those into a sitemap index parser rather than regexing the file yourself.

Read the non-Google directives when mirroring a site. getCleanParam(), getNoindex(), getRequestRate(), and getVisitTime() are what justifies this package over a ten-line allow/deny script. If you're building anything that mirrors, indexes, or syndicates a site, map these into your own policy objects early — they're cheap to read and expensive to retrofit.

Test your own /robots.txt like any other route. This is the highest-value use most teams miss: a feature test that parses the response your app actually serves and asserts an allow/deny matrix.

$body = $this->get('/robots.txt')->assertOk()->getContent();
$parser = (new RobotsTxtParser(config('app.url')))->setContent($body);

$this->assertFalse($parser->userAgent('Googlebot')->isAllowed('/admin'));
$this->assertTrue($parser->userAgent('Googlebot')->isAllowed('/products'));

Audit drift. Store a hash of the fetched body plus the extracted getRules()/getSitemaps() per host. Robots.txt changes are a recurring cause of "the crawler silently stopped" — a weekly diff job pays for itself.

Gotchas and Tips

• Effectively unmaintained. Last release April 2021. Pin ^2.0, keep it behind your own RobotsTxt interface, and be ready to swap. Check composer.json for ext-curl/ext-mbstring requirements before it lands on a slim production image.
• The parser may fetch for you — which bypasses Http::fake(). If your tests fake HTTP and the parser still hits the network, you're using the URI path. Inject raw text in tests, or the suite becomes flaky and slow.
• 404 vs 5xx is your decision, not the library's. RFC 9309 says 4xx → allow all, 5xx → disallow all (or retry). The parser parses; it does not implement crawl policy. Encode this explicitly or you'll silently crawl a site that was down.
• Allow beats Disallow on equal-length matches; otherwise longest match wins. The library gets this right — do not layer your own regexes on top and "fix" it. Wildcards * and the $ end-anchor are supported; most homegrown implementations get $ wrong.
• Paths are case-sensitive, user-agent names are not. Set the UA once and consistently — a mismatch between your configured crawler UA and the UA you query with produces confidently wrong answers.
• Missing, empty, or BOM-prefixed files. Robots.txt must be UTF-8; a BOM or stray bytes on line one can drop the first group. Parse the body you actually received, not a trim()ed summary, and check the package's cleaner helper if you see a directive silently ignored.
• crawl-delay and request-rate are non-standard. Google ignores both. Honour them for the hosts that declare them; don't assume they exist (?? 1 above), and note crawl-delay can be a float.
• Sitemap URLs are not guaranteed absolute. Resolve them against the base URI before dispatching.
• "Extensible" without docs. Directives are implemented as handlers you can extend, but the extension API is undocumented — read src/ before subclassing. Composition (a wrapper service) is almost always the better bet than inheritance here.
• Small community (25 stars). Expect no Stack Overflow answers and few blog posts. Budget an afternoon with the test suite the first time an edge case surprises you; that time is still cheaper than reimplementing wildcard precedence and Yandex directives.
• When not to use it. If you only need Disallow:/Allow: for one or two known bots and never touch sitemaps or vendor directives, a small maintained reader (or fifty lines of your own with tests) is a lower-risk dependency. Reach for this package when you need the full directive set or per-vendor rule isolation.