Laravel Audit Laravel Package

Technical Evaluation

Architecture fit. laravel-audit is a Laravel artisan-command package, not a library you call from application code. It installs into a Laravel app, publishes config, and drives an external agent CLI (Claude Code or OpenCode) through php artisan audit, audit:tool, and audit:orchestrate. State lives on disk under storage/audit/<run>/tool/ rather than in the database, so the package imposes no schema, no migrations, and no runtime dependency on the host application. That is a clean fit for the dominant TPM use case: a point-in-time or scheduled quality/security audit of an existing Laravel codebase, run from a developer machine or CI, whose output is markdown reports committed or attached to a PR. It is a poor fit if what you actually need is an in-request static-analysis library.

Key architectural properties worth noting for planning:

• Two execution modes. Interactive (php artisan audit, offers to wait out rate limits and to pick a model) versus headless (audit:orchestrate, one agent session with audit:tool as its only tool). The headless mode is the CI-shaped one.
• Stage model. Tasks are partitioned into analyse, standards, security, with start --stage, --pending, --failed. This gives a TPM a natural knob for cost/scope tiering: security-only on every PR, full three-stage run nightly.
• Durable run state. Resume (audit --resume[=<run>]), per-task outcome persistence, and pause.json with a known reset time mean the workflow survives rate limits, budget caps, and interrupted sessions without losing completed work. This is the single most operationally important design decision in the release.
• Guardrails are config-first. agent.max_attempts (2), tool.max_calls (400), agent.run_budget_usd, agent.orchestrator.max_budget_usd (7200 s timeout, 300 turns), agent.verify_max_turns (6). Cost and blast radius are bounded by configuration rather than by operator discipline.
• Laravel Boost coupling. Standards come from Boost-written rule blocks in AGENTS.md/CLAUDE.md plus .ai/rules and .ai/guidelines, inlined under ## Coding standards (capped at standards_max_chars, 6000). If the org does not use Boost, agent.standards = none or an explicit file list keeps the feature usable, but the default path assumes Boost adoption.

Integration feasibility. Low ceremony: composer require → audit:install (registers the laravel-audit skill and guideline with Boost, interactively via boost:update --discover or non-interactively with --yes + boost.json entry) → configure provider credentials → run. The new model check (a tool-less turn before the preflight, for audit, --resume, and audit:orchestrate) is a genuine onboarding win: it fails fast with a clear message instead of burning a run on an unavailable model, and it offers a model picker in interactive consoles (agent.check_model to disable).

Blocking prerequisites to confirm before committing:

1. An agent CLI binary present and version-pinned on every machine/runner (claude aliases or opencode models).
2. Provider credentials and egress to the model API from wherever the audit runs — this is source-code egress, and is a security-review item, not just a networking item.
3. Laravel/PHP version support and whether the audit runs against a checkout of the repo (recommended) or a live environment.
4. storage/ writability and a retention policy for storage/audit/<run>/.

Technical risk. This is a v0.2.0 first tagged release that ships MVP2 and MVP3 together, and the release notes themselves document churn within the release:

• Removed in MVP2, re-added in MVP3: the headless orchestrator (--orchestrate, orchestrator => agent, agent.orchestrator.*). Anyone who adopted the MVP2-era shape and then took MVP3 sees the reverse migration. Pre-1.0 semver makes this permissible, but it means: pin the exact version, vendor-lock the config, and treat config keys as unstable.
• Breaking renames: agent.mode / agent.concurrency / --mode → agent.max_concurrency / --concurrency; agent.verify.always_categories removed in favour of agent.verify.from (default medium). Removed keys are reported as no longer read — good failure signalling, but a silent behaviour change if you skip the warning.
• Codex driver removed because it "works through a shell and cannot keep agents away from .env." Read this as the package's own admission that sandboxing is driver-dependent. Any plan must include filesystem isolation (container, worktree, no .env, no secrets in the checkout) rather than relying on the harness.
• Nondeterminism. Findings vary by model, by project.context text, and by turn budget. The notes give a concrete cautionary tale: Haiku treated a prompt example (single-user app) as fact and downgraded four findings. Mitigations shipped (don't guess the user model; name the assumption severity depends on; verify every medium+ finding unless confirmed) reduce but do not eliminate this. Budget a calibration period before findings are trusted as a gate.
• Cost variance. Bounded per run by agent.run_budget_usd and per run by tool.max_calls, but the ceiling is still hundreds of agent calls; multi-audit --audits= runs multiply it.
• Internals volatility. AgentRunner was split into loop / JobQueue / FollowUp, with an arch test capping files at 250 lines. Healthy engineering signal, but it also signals active refactoring — extensions and patches will conflict across minor versions.

Key questions.

• What is the supported Laravel/PHP/OS matrix, and is Windows (or WSL-only) in scope?
• Is there a CI recipe (exit code 4 handling, artifact upload of storage/audit/<run>/, caching of the agent CLI)? Exit code 4 on pause is well-designed but needs an explicit pipeline branch.
• What is the measured wall-clock and dollar cost of a full three-stage run on a mid-size app at agent.max_concurrency: 1?
• What is the data-handling posture: which files and how much of them leave the machine, is there a redaction/ignore list, and does anything prevent .env or credentials from being read by the agent?
• How stable is the audit:tool JSON contract (next, finished_since, failed_since, spend_usd, pending) for teams that want to build their own orchestrator on top?
• Licensing and support model for a pre-1.0 package, and who owns upgrades.
• Does the ## Checks section give enough determinism to use "findings count by severity" as a trend metric across runs?

Integration Approach

Stack fit. Best fit: Laravel 10/11-era monolith or modular app, PHP 8.2+, Git-based workflow, GitHub Actions or GitLab CI, and either an existing Boost installation (unlocks the standards path with zero extra work) or a willingness to point agent.standards at .ai/rules / explicit files. Weak fit: shops without a Claude Code/OpenCode entitlement, air-gapped environments, polyglot repos where only the PHP slice is auditable, or teams expecting deterministic gate-able output on day one.

Migration path. There is nothing to migrate from — this is a greenfield adoption, and no prior assessment exists. Treat it as a net-new capability with a pilot, not a replacement for PHPStan/PHP-CS-Fixer/Semgrep. The honest framing for stakeholders: this complements static analysis by producing judgement-based findings (security, dead code, database design, conventions) with severity, confidence, and a verify pass — it does not replace a linter, and audit:orchestrate's repair pass explicitly fixes linter warnings alongside its own findings, which only makes sense if the linter is also present.

Compatibility. Three surfaces to pin:

1. Package version — pin 0.2.x exactly; do not float on ^0.2 until config keys settle.
2. Agent CLI version — the model check helps at runtime, but CLI output formats drive the pause/limit detection; pin and upgrade deliberately.
3. Model aliases — model names are the most externally volatile dependency. Record the chosen model in version-controlled config (AUDIT_ORCHESTRATOR_MODEL, agent.model), not in a shell profile, so runs are reproducible and comparable.

Config hygiene: keep config/audit.php in the repo, set agent.max_concurrency explicitly, set agent.run_budget_usd and agent.orchestrator.max_budget_usd to a number the finance owner has agreed, and set project.context deliberately — the release notes show that a wrong or example-contaminated context actively distorts severity ratings.

Sequencing.

1. Week 0 — preconditions. Confirm credentials, egress approval, agent CLI in the base image/dev setup, storage/ writability, and a pruning policy for storage/audit/.
2. Week 1 — sandboxed pilot. One repo, one branch, container or throwaway worktree with no .env and no secrets. composer require the pinned version, run audit:install --yes, run the model check, then a single audit with --stage=security and a conservative agent.run_budget_usd. Review the report manually against known issues to calibrate signal quality.
3. Week 2 — standards path. Wire Boost (or agent.standards file list), set standards_max_chars, confirm the ## Coding standards block actually contains your conventions, and check the ## Checks section reads sensibly.
4. Week 3 — full manual runs. php artisan audit across all three stages, exercising resume and the pause path deliberately (set a low budget to force a pause; verify exit code 4 and the resume command). This is the cheapest way to learn the failure modes.
5. Week 4 — headless trial. Introduce audit:orchestrate in a non-blocking CI job, upload storage/audit/<run>/ as an artifact, and capture orchestrator.md.
6. Week 5+ — cadence. Nightly full run; optional security-stage-only run on PRs once false-positive rate is understood. Keep it advisory; promote to blocking only for a narrow, hand-curated subset (e.g. confirmed security findings above a threshold), if ever.

Do not attempt --audits= multi-audit, custom max_turns tuning, or per-task skill maps in the pilot — those are second-order optimisations whose value depends on knowing your baseline.

Operational Impact

Maintenance. Two independent upgrade streams (package, agent CLI) plus a third (model aliases), each capable of changing output. Budget roughly a half-day per package minor release for a config diff and a re-baseline run, and expect that number to hold for the foreseeable pre-1.0 life of the project. The 250-line-per-file arch test and the AgentRunner decomposition are good signals about internal discipline, but they also mean internal churn; avoid forking or monkey-patching. Removed config keys are reported at runtime rather than silently ignored, which materially reduces the classic "we upgraded and nothing happened" failure — make sure those warnings are visible in logs, not buried.

Support. Expect three recurring support requests: (a) "the run paused" — exit code 4, pause.json, wait for the reset or resume; (b) "the findings are wrong/noisy" — almost always project.context or model choice, not the harness; (c) "it couldn't reach the model" — credentials, egress, or an alias the CLI no longer knows. Document these three runbooks before broad rollout, plus a named owner for the cost budget. Vendor support for a v0.2.0 package should be assumed to be "issues and source" rather than an SLA.

Scaling. agent.max_concurrency defaults to 1, and the harness caps a run at tool.max_calls (400) with agent.max_attempts (2) per task. Vertical scale (a bigger repo) therefore costs wall-clock linearly and dollars roughly linearly. Plan for parallel runs across repos rather than parallel tasks within a run, and stagger nightly schedules so concurrent runs don't collide on provider rate limits and convert into pauses. Storage grows per run (tasks/, reports, orchestrator.md, run.md) — a 30-day retention with artifact upload to CI storage is the sane default. Multi-repo rollout is straightforward because state is per-run and per-repo, with no shared service.

Failure modes.

• Usage/rate limit → run pauses, exit 4, unfinished tasks stay pending, resume command printed. Well-handled; the risk is a CI job that treats exit 4 as a failure and pages someone. Encode exit 4 as "deferred," not "broken."
• Budget exhausted (agent.run_budget_usd) → pauses before the offending task. Same handling; expect this on large repos with a low cap, which is the correct trade.
• Tool-call ceiling (400) → after the cap only status, wait, finish work. Effectively a truncated audit; the ## Checks section should show checks as not audited / skipped, so read that section before trusting a report.
• Model unavailable / wrong alias → stopped by the preflight model check before spend. Best-case failure mode; keep agent.check_model on.
• Orchestrator session ends without finish → reports are written anyway by the command. Degraded but not lost; orchestrator.md should be reviewed to see why the session gave up.
• Secret exposure → the reason the Codex driver was dropped. Treat .env isolation as an infrastructure responsibility: no secrets in the audited checkout, run in a container, review .gitignore and any ignore config the harness honours. This is the one failure mode with a non-financial blast radius, and it should be signed off before the pilot, not after.
• Severity distortion from stale context — the Haiku incident (example misread as fact, four findings downgraded) is the template. Guard with a maintained project.context and a spot-check of medium/likely findings each cycle.

Ramp-up. A single developer reaches a first useful report in under a day: install, audit:install, model check, --stage=security with a small budget. Meaningful fluency — reading the ## Checks table, interpreting severity/confidence ordering, knowing when to resume versus re-run, tuning project.context, and judging whether a finding is real — is a two-to-three-week curve for one owner. Team-wide adoption needs three artifacts: the three runbooks above, a one-page "how to read an audit report" (numbered by severity → confidence → task; clean / N findings (IDs) / not audited / skipped), and a written policy that the audit is advisory until a calibrated baseline exists. The interactive model picker, the interactive "wait for reset and continue," and the readable-report work in MVP3 all shorten this curve noticeably — the release is clearly being hardened for exactly this onboarding path, which is the strongest argument for piloting now rather than waiting for 1.0.