awcodes/botly
Botly is a lightweight Laravel package for detecting bots and crawlers, including AI scrapers, from incoming requests. It inspects user agents to identify automated traffic so you can block, log, or otherwise handle bots through simple middleware and configuration.
Confirm what you're actually installing first. The package metadata in this listing describes a bot-detection framework (fluent definitions, webhook handling, conversation state), while the release notes and every artifact describe a Filament robots.txt manager. That inconsistency was never reconciled. Trust the release notes and the repository tagline, then verify the installed artifact's composer.json and src/ layout before you plan any work around it.
Prerequisites. Laravel plus Filament as your admin panel. Filament isn't a soft dependency — the editable rules table, the page, and the notifications are the product. Without it you're buying a rules-to-string renderer and writing your own UI.
Install path.
composer require awcodes/botly
php artisan vendor:publish --tag="botly-config"
php artisan vendor:publish --tag="botly-migrations"
php artisan migrate
Confirm the actual publish tags against the published docs (docs.aw.codes) or the package's service provider — the tags above follow the awcodes convention but tags drift between packages, and the docs live on an external hub rather than the README.
Register the panel surface. The package ships a Filament page; add it to your panel provider's discovery paths or plugin registration depending on how it's authored. Check the page class for a static getNavigationIcon() / slug override if you need it to sit under a specific navigation group — the v1.0.6 release notes reference a navigation heroicon fix, so that's a supported extension point.
Where to look first, in order:
config/botly.php — directive definitions, the AI-crawler block list, the custom-directives escape hatch, and whatever switch controls static-file-vs-database precedence. This file answers most of your integration questions before you read any PHP.First use case: reproduce, generate, diff. Don't start by authoring new rules. On a canary or staging environment, transcribe your current committed robots.txt into the rules table, generate the output, and diff it against the file in the repo. This is the single highest-value half hour you'll spend. v1.0.6 normalized emitted directive casing (Disallow, Allow, Crawl-delay, Clean-param), so your first generation will diff on casing even though crawler behavior is unchanged. Catching that in a controlled diff is fine; catching it because your snapshot tests went red in CI during a release is not.
Decide once whether generation is request-time or write-to-disk, and default to write-to-disk. Serving robots.txt by booting Laravel and rendering on every crawler hit is the wrong default — you pay framework boot cost for a file that changes weekly, and you introduce a failure mode where a database outage produces a 500 for the one endpoint crawlers treat specially. Preferred shape:
rules table (admin edits) → artisan generation command → public/robots.txt → CDN purge
Wire the generation command into the release script so every deploy emits a fresh file, and keep an artisan invocation handy for the "marketing changed a rule at 3pm" case so you don't need a full deploy to regenerate.
Keep the committed static file as a rollback artifact. Until a full release cycle has run clean with generated output, do not delete public/robots.txt from the repo (or whatever path wins). If generation breaks, you want a one-line revert that restores known-good crawler policy, not a scramble to reconstruct rules from memory.
AI crawler policy belongs in config, per environment. The package separates GPTBot-class blocks from path rules and drives them from config. That means it rides your existing config:cache lifecycle: staging stays permissive, production restricts, and nobody edits a database row to toggle a policy that legal asked about. Keep this in config and out of the editable UI — the UI is for path rules and crawl-delay, the config is for the policy question.
Treat custom directives as a documented escape hatch, not a feature. Custom directives from config are emitted verbatim, bypassing the v1.0.6 casing normalization that applies to first-class directives. If you use one, be deliberate: any downstream tool, snapshot test, or human auditor that is case-sensitive will see something different from the rest of the file. Document why the custom directive exists in the config comment.
Test the renderer, not the UI. The valuable regression test is a snapshot of generated output for a known rules set:
it('renders the expected robots.txt', function () {
// seed a known rules set
$rendered = /* generation call, see docs for the command/class */;
expect($rendered)->toMatchSnapshot();
});
This is what catches casing changes like the v1.0.6 normalization on the next dependency bump — which is exactly the kind of thing a low-adoption single-maintainer package ships without a migration note.
Governance lives outside the package. The persistent rules table gives you a change record and human-readable directive labels, but it isn't an audit trail with diffs and approvals. Add a review step for production rule edits in whatever process you already have, and name the approver. The package makes it easy to edit crawler policy; it does not make it safe to do so unreviewed.
Handle the multi-panel case explicitly if you have one. If you run separate admin panels, decide which panel hosts the page and whether each panel's users should see it. Tenancy isn't evidenced anywhere in these artifacts, so if you need per-domain or per-tenant rendering, evaluate the persistence model before you build anything — retrofitting tenancy into a rules table is a schema change, not a config change, and that's the actual deciding factor, not the UI.
Trust the release notes, not the description. The listing metadata describes a bot-detection framework; the code manages robots.txt. The metadata was never reconciled after a pivot or a template copy. If you're evaluating this against a requirement for "detect and block bots," you're looking at the wrong package — this one authorizes crawlers, it doesn't classify live traffic.
Casing changed in v1.0.6 and will diff your artifacts. disallow: /cart became Disallow: /cart. Crawlers match case-insensitively, so production behavior is identical — but your committed file, your snapshot tests, and anything cached at the edge will diff on first regeneration. Purge the CDN on regeneration or the old artifact lingers.
Custom directives are emitted verbatim. The escape hatch bypasses normalization deliberately. That's a latent inconsistency if anything downstream is case-sensitive.
Translations may surface as raw keys. v1.0.6 fixed a specific instance (delete/rename notifications on the static file page showed a translation key instead of the message). Check whether translations are published at install or shipped in the package, and spot-check the UI in your locale after publishing. If you see a dotted key in a toast, that's the class of bug — publish translations rather than patching around it.
Know what happens when the database is unreachable. If you serve dynamically and the DB is down, crawlers may get a 5xx. A 5xx on robots.txt is worse than a stale file — well-behaved crawlers back off or treat it as "no restrictions." Write to disk and serve statically so the filesystem, not the database, is the dependency at crawl time.
Decide and document static-file vs database precedence. When both a static robots.txt and a rules table exist, the package's behavior (override, merge, or defer) is the thing you'll be debugging at an inconvenient moment. Write it into your runbook while you're reading the code, not after an incident.
Pin the version and read the diffs before bumping. Single maintainer, six stars, zero dependents, external docs, no deprecation policy. That's fine for an SEO convenience and unacceptable for anything contractual. Most of the v1.0.6 diff is CI normalization onto shared workflows plus Rector/PHPStan path alignment — good hygiene and a poor adoption signal at the same time.
Write your own runbook; don't link the docs. Docs moved to docs.aw.codes and the README was trimmed to a landing page. Your runbook — not a bookmark — should hold the config keys, the generation command, the deploy hook, the precedence rule, the rollback procedure, and the CDN purge step. Onboarding shouldn't depend on a third party staying reachable and in step with the code.
Scope check before you sprint. This deletes the authoring slice of a crawler-management epic: rule CRUD, directive rendering, AI-bot toggles. It does not do log analysis, bot verification, crawl-budget reporting, crawl simulation, or sitemap generation. Over-scoping because a package "does robots.txt" is the failure mode.
Blast radius is human, not technical. A mistyped directive in a low-friction UI can hide the site from search. Whatever review step you'd apply to a one-line change in a committed file, apply here — the UI's convenience is precisely what removes the friction that used to catch mistakes.
How can I help you explore Laravel packages today?