webignition/robots-txt-file
Parse and work with robots.txt files: convert raw content into a model, query directives per user-agent, check if a path is allowed, extract sitemap URLs, filter directives by type, and generate robots.txt strings programmatically.
|
Package
|
Description
|
Stars
|
Likes
|
Forks
|
Downloads
|
Issues
|
Score
|
Opportunity
|
License
|
Last Release
|
|
|---|---|---|---|---|---|---|---|---|---|---|---|
| contao/contao | — | 448 | 449 | 178 | 263 | 336 | 27.6 | 28.7 | LGPL-3.0 | — | |
| contao/core-bundle | — | 123 | 130 | 53 | 22K | 0 | 2.2 | 31.0 | LGPL-3.0 | — | |
| terminal42/escargot | A Symfony HttpClient-based crawler framework for PHP that can crawl any HTTP resource, not just websites. Jobs are tracked by ID and backed by a queue, so long crawls can pause and resume. | 59 | 59 | 7 | 55K | 0 | 23.3 | 47.9 | — | 3 weeks ago |
How can I help you explore Laravel packages today?