vipnytt/robotstxtparser
A PHP library for parsing robots.txt files, with support for user-agent groups, Allow/Disallow rules, wildcards, crawl-delay, and sitemap directives. Lightweight and framework-agnostic, it helps determine whether a crawler may access a given URL.
|
Person
|
Packages
|
Contributions
|
Followers
|
|
|---|---|---|---|---|
|
Jan-Petter Gundersen
@JanPetterMG |
5 | 290 | 4 |
View on GitHub
|
|
Chun-Sheng, Li
@peter279k |
415 | 708 | 2,308 |
View on GitHub
|
|
Marc Ortolà
@marcortola |
9 | 17 | 16 |
View on GitHub
|
|
devCollaboratorPro
@devCollaboratorPro |
1 | 1 | 0 |
View on GitHub
|
How can I help you explore Laravel packages today?