Weave Code
Robots Txt Laravel Package
spatie/robots-txt
Spatie's robots-txt package checks whether a page may be indexed or followed using robots.txt rules, robots meta tags, and HTTP headers. It supports custom user agents, custom robots.txt locations, disallow reasons, and crawl delays.
|
Package
|
Score
|
Description
|
Stars
|
Likes
|
Forks
|
Downloads
|
Issues
|
Score
|
Opportunity
|
License
|
Last Release
|
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| spatie/typed | 0.91 | Userland improved PHP type system with type inference and runtime checking: generics, union types, typed collections/lists, tuples, and structs. Proof-of-concept package from Spatie to add stronger type guarantees without language-level support. | 316 | 317 | 15 | 18 | 0 | 1.9 | 0.2 | MIT | 5 years ago | |
| spatie/url | 0.91 | Immutable URL parser/builder for PHP. Parse scheme, host, path and query, then fluently transform parts (withHost, withPath, withScheme) and manage query parameters. Supports allowed/sanitized schemes for safer URL handling. | 738 | 751 | 59 | 683K | 0 | 12.5 | 41.0 | MIT | 2 years ago | |
| spatie/crawler | 0.90 | A powerful PHP package for crawling websites, using Guzzle promises to fetch multiple URLs concurrently. It can render JavaScript via Chrome and Puppeteer, restrict crawls to internal links or depth, collect found URLs, and fake HTTP responses for testing crawl logic. | 2,829 | 2,843 | 368 | 1M | 0 | 52.5 | 41.5 | MIT | 2 months ago | |
| spatie/7to5 | 0.90 | Abandoned. Converts PHP 7.0 code to PHP 5, useful when developing on PHP 7 but deploying to PHP 5. Provides a CLI to convert entire directories and rewrites features like anonymous classes, type hints, return types, and null coalescing. | 177 | 175 | 32 | 47 | 0 | 2.6 | 1.0 | MIT | 8 years ago | |
| spatie/image | 0.90 | spatie/image provides an expressive, chainable PHP API for common image manipulation. Resize to fit a bounding box without altering aspect ratio, apply greyscale, adjust brightness and quality, rotate, sharpen, and save or overwrite images in just a few lines. Ideal for Laravel and PHP apps. | 1,367 | 1,389 | 143 | 4M | 0 | 40.9 | 56.6 | MIT | 2 days ago | |
| spatie/mixed-content-scanner | 0.90 | Scan a website for mixed content by crawling pages and flagging insecure http:// resources in common HTML tags (img, script, iframe, link, etc.). Use MixedContentScanner with a logger to report where mixed content is found or missing. | 105 | 106 | 18 | 59 | 0 | 10.1 | 2.7 | MIT | 3 years ago | |
| spatie/mixed-content-scanner-cli | 0.90 | CLI tool to detect mixed content on HTTPS sites. Crawls pages and reports HTML elements whose URLs use http:// (images, scripts, iframes, forms, etc.). Install via Composer globally and run: mixed-content-scanner scan . | 93 | 93 | 20 | 0 | 0 | 3.8 | — | MIT | 4 years ago | |
| lbonnet/on-page-seo-bundle | 0.88 | lbonnet/on-page-seo-bundle centralizes on-page SEO metadata for Symfony applications. Configure page titles, meta descriptions, canonical URLs, robots directives, and Open Graph/Twitter tags from a single service for consistent, maintainable SEO. | 0 | 0 | 0 | 0 | 0 | 8.8 | — | MIT | 1 month ago | |
| mguinea/laravel-robots | 0.87 | — | 15 | 16 | 6 | 2K | 0 | 0.7 | 22.0 | MIT | — | |
| anomaly/robots-extension | 0.86 | An extension for the Anomaly Streams platform that adds robots.txt management to your Laravel application, letting you control crawler directives and SEO indexing rules from the admin panel without editing files by hand. | 0 | 0 | 3 | 254 | 0 | 0.1 | 33.2 | MIT | — | |
| webignition/robots-txt-file | 0.86 | Parse and work with robots.txt files: convert raw content into a model, query directives per user-agent, check if a path is allowed, extract sitemap URLs, filter directives by type, and generate robots.txt strings programmatically. | 18 | 19 | 3 | 56K | 3 | 2.5 | 48.4 | MIT | 6 years ago | |
| miisieq/robots-txt-bundle | 0.86 | A Symfony bundle for generating and serving a dynamic robots.txt file, letting you define user-agent rules, disallow paths, and sitemap entries in configuration instead of a static file. Handy for multi-environment setups where crawl rules need to vary. | 3 | 3 | 1 | 0 | 0 | 0.0 | — | MIT | 8 years ago | |
| vipnytt/robotstxtparser | 0.86 | A PHP library for parsing robots.txt files, with support for user-agent groups, Allow/Disallow rules, wildcards, crawl-delay, and sitemap directives. Lightweight and framework-agnostic, it helps determine whether a crawler may access a given URL. | 25 | 26 | 6 | 12K | 1 | 0.4 | 36.9 | MIT | 5 years ago | |
| spatie/laravel-robots-middleware | 0.86 | Laravel middleware to control search engine indexing via X-Robots-Tag/robots meta behavior. Extend the base middleware and implement shouldIndex() to allow or block indexing per request (e.g., disable for admin routes), without changing your views. | 333 | 337 | 29 | 68K | 0 | 12.4 | 33.6 | MIT | 8 months ago | |
| spatie/laravel-markdown-response | 0.85 | Serve markdown versions of Laravel HTML pages to AI agents and bots. Detects text/markdown Accept headers, AI user agents, or .md URLs; converts via local PHP or Cloudflare Workers AI, caches results, and supports HTML preprocessing. | 84 | 83 | 9 | 60K | 0 | 22.2 | 46.0 | MIT | 2 weeks ago | |
| cdaguerre/robots-bundle | 0.85 | — | 0 | 0 | 1 | 0 | 0 | 2.1 | — | — | 9 years ago | |
| spatie/laravel-sluggable | 0.85 | Generate unique slugs for Eloquent models on create or update via a Sluggable attribute or HasSlug trait. Includes collision suffixes, translatable slugs, and self-healing URLs with 308 redirects. | 1,560 | 1,586 | 190 | 867K | 0 | 43.5 | 44.5 | MIT | 1 week ago | |
| fourlabs/robots-bundle | 0.85 | A lightweight PHP bundle for defining and serving robots.txt directives from configuration, making it easy to control crawler access to your application. Simple to integrate, but minimally adopted and sparsely documented, so evaluate fit carefully. | 1 | 2 | 0 | 3 | 0 | 0.0 | 5.5 | MIT | — | |
| alibaba258/laravel-robots | 0.85 | Manage robots.txt dynamically in Laravel. Build rules per environment (e.g., allow production with sitemap, disallow non-prod), generate plain-text output, and optionally persist configuration via migration with a swappable data source. | 0 | 0 | 0 | 0 | 0 | 0.6 | — | MIT | 3 years ago | |
| awcodes/botly | 0.84 | Botly is a lightweight Laravel package for detecting bots and crawlers, including AI scrapers, from incoming requests. It inspects user agents to identify automated traffic so you can block, log, or otherwise handle bots through simple middleware and configuration. | 6 | 6 | 1 | 621 | 0 | 18.8 | 31.4 | MIT | 1 month ago |
Weaver
How can I help you explore Laravel packages today?