Symfony Bundle Laravel Package

Product Decisions This Supports

• AI-content policy as a first-class product surface. RobotsFilter::AI_BOTS turns "do we let AI crawlers ingest our content?" into a versioned, code-reviewed constant rather than an ad-hoc robots.txt edit. That means pricing/packaging decisions (e.g., paywalled content vs. AI indexing, licensed crawler access) can be expressed in the same repo as the rest of the product.
• Build vs. buy on crawler policy infrastructure. Rather than maintaining an in-house user-agent list, the package tracks an upstream registry (gt-backend). The PM decision becomes "which agents do we care about" instead of "who owns the regex."
• Roadmap input on a per-agent allow/deny matrix. v0.9.3's addition of ExaSearchBot is the clearest signal of the recurring decision this package forces: each new AI search/crawl agent requires a policy call (block, allow, or allow-with-conditions). Expect this cadence to continue; budget review time, not engineering time.
• Defensible defaults with per-site override. Because a site that already names an agent in its own robots.txt keeps its rules, you can ship a conservative default in the package while letting individual properties (docs site, marketing site, customer-facing app) diverge. That supports differentiated policies across a portfolio without forking the codebase.
• Compliance and partner-commitment hygiene. Newly enumerated agents map to real third parties. Adding one is a decision with contractual and reputational weight—useful evidence when legal or partnerships ask "did we intend to allow this crawler?"

When to Consider This Package

• Adopt when you operate multiple Laravel properties with non-identical AI-crawler policies, or when "which AI bots can read us" is a decision someone outside engineering cares about and should be reviewable in a diff.
• Adopt when you want upstream-maintained agent coverage and are comfortable with the release cadence this implies—each new agent lands as a version bump, so pinning matters.
• Adopt when your current robots.txt is hand-edited, untested, and drifting from the intent of marketing, legal, or content licensing.
• Look elsewhere when you need full RFC 9309 conformance testing, crawl-budget analytics, or bot mitigation at the edge (WAF/CDN layer). This is a policy declaration mechanism, not enforcement—a misbehaving crawler ignores robots.txt regardless.
• Look elsewhere when you have a single small site; the abstraction cost likely exceeds one hand-written file.
• Be cautious when you rely on the default AI_BOTS set as a security control. Treat it as intent, not protection, and pair it with rate limiting or signed-agent verification if blocking actually matters.

How to Pitch It (Stakeholders)

• To executives: This is a small, low-risk dependency that lets us state and maintain our position on AI crawlers—who may index our content and who may not—in one reviewed place. This release adds Exa's search crawler to the known-agent registry; no action is required unless we want a different policy for that specific crawler. No breaking changes, no migration, no customer-visible behavior unless we opt in.
• To engineering: Additive only—ExaSearchBot joins RobotsFilter::AI_BOTS, and any site that already specifies rules for it is unaffected. Low upgrade cost; the real cost is ongoing review of new agent entries as they arrive. Recommend pinning the version and treating upgrades as a routine policy checkpoint rather than a routine dependency bump.
• To legal/partnerships: The package surfaces crawler-name changes as explicit, auditable diffs. Use each bump as the trigger to confirm whether a given agent is covered by existing terms or content-licensing agreements before allowing it.