Weave Code
Robots Txt Laravel Package
spatie/robots-txt
Spatie's robots-txt package checks whether a page may be indexed or followed using robots.txt rules, robots meta tags, and HTTP headers. It supports custom user agents, custom robots.txt locations, disallow reasons, and crawl delays.
Determine if a page may be crawled from robots.txt and robots meta tags
Frequently asked questions about Robots Txt
- How do I install spatie/robots-txt in a Laravel application?
- Install it with composer require spatie/robots-txt. Laravel's package auto-discovery registers the service provider, so no manual wiring is needed. You can optionally publish config/robots-txt.php to control the robots.txt source, user agent, and related defaults per environment. If you prefer explicit control, disable auto-discovery and register the provider in your app config.
- Which PHP and Laravel versions does spatie/robots-txt support?
- The package is framework-agnostic at its core and integrates with Laravel through a native service provider, config file, and route registration. Its CI pipeline is green on PHP 8.2 through 8.4 and additionally covers PHP 8.5, which is useful confirmation for teams planning an 8.5 upgrade. Because public APIs and dependencies are unchanged in recent releases, existing Laravel applications do not need code or config migrations to stay current.
- How do I check whether a URL may be indexed or followed?
- Create a Robots instance and call mayIndex() or mayFollowOn() with the absolute URL. For example, Robots::create()->mayIndex('https://example.com/admin') returns a boolean based on robots.txt rules, robots meta tags, and X-Robots-Tag headers combined. The same instance can be reused for multiple URLs. You can also pass a custom user agent string to Robots::create() to evaluate permissions as a specific crawler.
- Can I use a custom user agent or a custom robots.txt location?
- Yes. Pass the user agent to Robots::create('UserAgent007') and it will be used when matching groups in robots.txt. Use withTxt() to point at a different URL or a local file path, such as Robots::create()->withTxt(__DIR__.'/public/robots.txt'). The lower-level RobotsTxt class accepts a raw robots.txt string directly, which is handy for tests and edge-case checks.
- How do I test robots.txt rules without hitting a live website?
- Instantiate RobotsTxt with an inline robots.txt string or call withTxt() with a local file path so no HTTP request is made. This keeps your PHPUnit or Pest suite fast and deterministic, and lets you assert on allows(), whyDisallows(), and crawlDelay(). For meta tag and header behavior, fake the underlying HTTP responses instead of calling the real site. Adding a snapshot of your generated robots.txt to CI makes future behavior changes visible as a diff.
- How can I find out why a URL is disallowed?
- Use whyDisallows('/admin', 'google'), which returns the matching rules as objects. Each rule exposes the userAgent that produced the match, so you can tell whether the disallow came from a wildcard group, a specific bot group, or another directive. This is especially useful when debugging conflicting Allow and Disallow lines. It also helps confirm behaviour changes after an upgrade.
- How do I read the crawl-delay directive from robots.txt?
- Call crawlDelay() with the path and user agent, for example crawlDelay('/admin', '*'), which returns the value as a string such as '1.5'. The package exposes the raw directive value rather than converting it, so you decide how to apply throttling in your crawler or queue. If no crawl-delay matches, expect a null return and handle that case explicitly.
- Does the package check robots meta tags and X-Robots-Tag headers as well as robots.txt?
- Yes. Robots::create() evaluates robots.txt rules alongside robots meta tags (noindex, nofollow) and X-Robots-Tag HTTP headers, which is why mayIndex() and mayFollowOn() can differ for the same URL. Remember that a URL blocked by robots.txt cannot be fetched to read its meta tags, so robots.txt effectively wins for crawlers that obey it. The package reflects that reality rather than treating the sources as independent.
- Why did my generated robots.txt results change after upgrading to 2.5.5?
- Release 2.5.5 corrects wildcard (*) and end-anchor ($) matching so behaviour aligns with RFC 9309 section 2.2.2, including the rule that Allow wins on a tie. As a documented consequence, some previously denied URLs become allowed and some previously allowed URLs become denied. Applications whose configs contain only plain-path Allow and Disallow rules are functionally unaffected. Capture your robots.txt output before and after the upgrade and diff it per environment before promoting.
- Is it safe to deploy spatie/robots-txt 2.5.5 to production, and how do CDN caches affect it?
- It is a drop-in patch with no public API or dependency changes, but it is behaviour-changing for configs that use wildcards or anchors, so treat it as a crawl-policy release rather than a routine update. Snapshot the current /robots.txt, deploy to staging, confirm crawler-visible output with a crawler emulator, then promote in a low-traffic window with a rollback plan of pinning back to 2.5.4. Invalidate your CDN for /robots.txt after deployment, since search engines and edge caches may retain the old file for some time. Monitor crawl stats and origin request patterns for a few weeks afterwards.
Weaver
How can I help you explore Laravel packages today?