awcodes/botly
Botly is a lightweight Laravel package for detecting bots and crawlers, including AI scrapers, from incoming requests. It inspects user agents to identify automated traffic so you can block, log, or otherwise handle bots through simple middleware and configuration.
Build vs. buy on an admin surface, not on a crawler engine. The scope is deliberately narrow: a publishable config file, one migration for a rules table, a Filament page for authoring, and a renderer that emits robots.txt. The buy decision is really "do we want to spend a sprint on settings UI, validation, persistence, and file-writing for a file that is usually under twenty lines?" For a team already on Filament the answer tilts to buy — the marginal cost is one Composer dependency and one table. For anyone else the math flips, because you'd be paying for templating and CRUD you could write yourself in an afternoon.
Who owns robots.txt. This is the decision the package actually unlocks, and it is a governance decision before it is a technical one. Moving crawler policy from "an engineer edits a committed file in a PR" to "SEO/marketing edits rules in a UI" trades review latency for blast radius. The package makes that trade easy to make accidentally, so make it explicitly: name the approver, decide whether production edits require a second pair of eyes, and decide whether the current static file survives as a rollback artifact.
AI crawler policy as a first-class, per-environment question. The package separates GPTBot-class blocks from path rules and drives them from config, which gives a non-engineering answer to a question legal, SEO, and leadership are all now asking — "do we allow AI training crawlers?" — with a default that rides your existing config:cache release lifecycle. Staging can stay permissive while production restricts, without a deploy.
Whether crawler policy is config or content. If you classify robots.txt as content, it belongs in a CMS with workflow and audit trail. If you classify it as infrastructure config, it belongs in a PR. This package asserts a third model — config-like blast radius with content-like editing ergonomics — and adopting it means endorsing that model. Worth one line in the decision record either way, because it will be cited the next time someone asks why marketing can edit a file that can deindex the site.
Auditability and compliance storytelling. The persistent rules table plus human-readable directive labels ("Disallow" rather than the stored lowercase key) gives you a change record and a readable answer to "when did we start blocking that crawler, and who asked for it?" It is thin as audit trails go — no evidenced diff history, no approval log — but strictly better than a git blame on a hand-edited file that no non-engineer ever touched.
Roadmap scoping: what a crawler-management epic actually contains. If an SEO/crawler epic exists, this package deletes the authoring slice — rule CRUD, directive rendering, AI-bot toggles. It does not touch log analysis, bot verification, crawl-budget reporting, crawl simulation, or sitemap generation. Scoping the epic correctly is the PM decision; over-scoping it because a package "does robots.txt" is the failure mode.
Dependency posture as a portfolio signal. Six stars, zero dependents, one maintainer, MIT, and a release whose diff is mostly CI normalization onto shared workflows plus Rector/PHPStan path alignment. That is good hygiene and a poor adoption signal simultaneously. Adopting it is a deliberate bet that a tiny dependency is cheaper than maintenance — reasonable for an SEO convenience, unreasonable for anything contractual.
Adopt when most of these hold:
robots.txt edit currently routes through a PR and a release train, the package pays for itself in latency alone.config:cache lifecycle fit cleanly into release practice with no new mechanism.docs.aw.codes), so onboarding depends on a third party staying reachable and in step with the code.Look elsewhere, or treat the package as a rules renderer only, when:
robots.txt management, while the packaged summary describes a bot-detection framework with middleware, allow/deny lists, and webhook handling. That metadata was never reconciled after a pivot or a template copy. Trust the release notes, not the summary — and confirm which artifact you are adopting before spending engineering time.robots.txt through the framework on every crawler hit is the wrong default. Deprioritize unless generation can be written to disk from a deploy hook or an artisan command.Due-diligence questions to close before committing: Which Laravel, PHP, and Filament majors are supported, and is Filament a hard dependency or optional integration? Is generation request-time, on save, scheduled, or artisan-driven? When both a static file and the rules table exist, which wins — override, merge, or defer? What serves robots.txt if the database is unreachable? Are translations published at install, or will operators see raw keys the way the delete/rename notification did before v1.0.6?
For executives. This is a small, cheap package that takes an engineering-only chore — editing robots.txt — and turns it into a reviewed admin action for marketing and SEO. The strategic value is speed and control over crawler policy, including the AI-crawler question every company is now being asked, without adding headcount or standing up a new subsystem. The costs are honest: single-maintainer open source, six stars, zero dependents, no support commitment. So we pin a version, keep the current static file as a rollback, and treat production rule changes like any other change with real blast radius. Recommendation: pilot on staging, reproduce our current file as rules, and diff the generated output before anything touches production. Expected effort is roughly half a day for a Laravel developer. The residual risk is human, not technical — a one-click UI over a file that can hide the site from search.
For engineering. A Laravel package that stores robots.txt rules in a database table, exposes them through a Filament page, and serves the rendered result — either written to disk or dynamically. No new runtime: no queue worker, no daemon, no external service, no schema beyond the package's own migration. v1.0.6 is a patch with no API boundary change. It normalizes emitted directive casing to the conventional Disallow / Allow / Crawl-delay / Clean-param (crawlers match case-insensitively, so production behavior is unchanged, but committed files, snapshot tests, and CDN artifacts will diff on first regeneration), fixes a raw translation key leaking into delete/rename notifications, and labels the persistent rules table correctly. The rest is CI normalization onto shared fleet workflows pinned to a commit SHA plus static-analysis path alignment — healthy, and also confirmation this is one person's project. Docs moved to docs.aw.codes and the README was trimmed to a landing page, so our runbook — not a link — should hold config keys, the generation command, and the deploy hook. Integration path: install on a canary environment, publish config/migrations/translations, reproduce the existing file as rules, diff the output, wire generation into the release script, then retire the hand-maintained file. Keep the committed static file as rollback until a full cycle runs clean, and make sure whatever sits at the edge is purged on regeneration.
For SEO / marketing stakeholders. You will be able to add or remove crawler rules yourself — including AI crawler blocks — and see the result without a ticket or a sprint. Two commitments from our side: production rule changes go through a short review, because a mistyped directive can hide the site from search, and generated files behind the CDN need a cache purge before changes take effect, so expect a short delay rather than an instant flip.
How can I help you explore Laravel packages today?