Weave Code
Weaver
Helps Laravel developers discover, compare, and choose open-source packages. See popularity, security, maintainers, and scores at a glance to make better decisions.
Feedback
Share your thoughts, report bugs, or suggest improvements.
Subject
Message
Log in Register
Home
Packages
Robotstxtparser

Robotstxtparser Laravel Package

vipnytt/robotstxtparser

A PHP library for parsing robots.txt files, with support for user-agent groups, Allow/Disallow rules, wildcards, crawl-delay, and sitemap directives. Lightweight and framework-agnostic, it helps determine whether a crawler may access a given URL.

View on GitHub
Deep Wiki
Context7

Robots.txt parsing library, with full support for every directive and specification.

Weaver

How can I help you explore Laravel packages today?

Conversation history is not saved when not logged in.
Prompt
Add packages to context
No packages found.
shitoudev/permissionmanager
johannesclimacus/laravel-tui
hocvt/seotools
vitalybaev/sypexgeo
amphibee/themosis-ignition
gito-me/fast-excel
frankyso/laravel-ipaymu
geolize-studio/laravel-chile
boomerangui/chat
jinas/dhivatar
store-doc/laravel-storage-driver
plus-it/laravel-translation-manager
diephp/perhaps
diephp/telegraph
osmaviation/spreadsheet
microsoft/fluentui-system-icons
mmccook/php-json-canonicalization-scheme
middlewares/firewall
qubiqx/montapacking
nyoncode/laravel-package-console