Pest Plugin Evals Laravel Package

Product Decisions This Supports

• Release gating for AI features. Evals are just Pest tests in tests/Evals, so "a prompt or model change ships only if the eval suite still passes" becomes a blocking CI check rather than a review-time opinion. That is a roadmap commitment you can defend to stakeholders, and it costs engineering time instead of a vendor contract.
• Model and vendor migration calls. v5.1.0 changes the default judge to gpt-6-luna and requires laravel/ai ^1.0, both with cost, latency, and reproducibility consequences. The plugin gives you a repeatable way to compare incumbent versus candidate models on identical inputs before committing — provided you pin the judge model explicitly instead of inheriting the default.
• Build vs. buy for the eval harness. If the team already writes Pest tests on Laravel, the harness is nearly free: no new platform, no new DSL, no separate dashboard contract, and assertions read like the tests engineers already author. Buying a dedicated eval platform then becomes a decision about dataset management, trend reporting, and drift alerting — not about whether quality is measurable at all.
• Where to spend on assertion rigor. The expectation ladder is effectively a cost/governance framework you can set policy on: deterministic checks (toContain, toBe, toBeJson, toHaveToolCalls, toFollowTrajectory) are free; embeddings (toBeSimilar) cost a little; judge calls (toBeCorrect, toSatisfy, toBeRelevant, toBeSafe) cost the most. Deciding which behaviours earn a judge call — and which must be proven deterministically — is a product decision about where quality risk actually lives.
• Classification and routing quality. Structured evaluations plus judgeUsingLaravelAiClassifier() move this past "score a freeform answer 0–1" to "verify the classifier picked the right intent or category," which directly supports triage, moderation, and support routing. The trade-off is legibility: classification reports the chosen level and its probability, not its reasoning, so a failure tells you what was picked but not why.
• Reliability as a stated property. ->repeat(3) through ->repeat(5) asserts every sample passes; ->not-> asserts no sample matches. A PM can now put a number on flakiness tolerance and hold the product to it, rather than arguing about whether one green run meant anything.
• Cost governance for judge calls. Evals skip unless explicitly enabled (--evals or PEST_EVALS=1), including custom JudgeDriver classes. You can approve eval spend per CI run without slowing the everyday developer loop, and set a per-run budget expectation with finance.
• Threshold policy on coarse boundaries. Judge, relevance, and safety scores snap to five levels (0, 0.25, 0.5, 0.75, 1.0), with a default threshold of 0.7. Gates should sit on those boundaries. Coarse levels make thresholds stable and explainable, but small regressions are invisible by construction — if the roadmap depends on detecting fine-grained drift, plan around that gap rather than file it as a bug. Note LaravelAiClassifier weights levels by probability and can return values between levels (e.g. 0.95), so thresholds tuned under LaravelAiJudge do not transfer automatically.

When to Consider This Package

Consider it when:

• The product is Laravel/PHP and the suite is already Pest — adoption cost is low and assertions look like existing tests.
• You have AI-backed behaviour whose quality is a release risk (summarisation, extraction, classification, customer-facing generation) and no reliable way to catch regressions today.
• You need a merge gate, not a dashboard: pass/fail plus failure detail is exactly what a required check requires.
• You want classification correctness verified by a model-assisted judge rather than brittle string matching, and you accept per-run judge cost for it.
• You need tool-use and trajectory assertions — toHaveToolCalls and toFollowTrajectory with strict or loose ordering cover agent behaviour plain output matching cannot.
• You are on, or can move to, laravel/ai ^1.0 and pestphp/pest ^5.2.1. This is a hard prerequisite for v5.1.0, not a preference.

Look elsewhere when:

• You are on laravel/ai 0.x, Pest 4.x, or PHPUnit without Pest. There is no reason to adopt Pest in order to acquire evals; v5.1.0 is a forced upgrade with API rewrites.
• You need fine-grained scoring, statistical confidence intervals, multi-turn agent simulation, or large human-labelled golden datasets. This is an assertion library for release gating, not an eval platform.
• You need executive reporting — dashboards, trend lines, week-over-week drift alerts. Pair it with something, or buy.
• The stack is Python or TypeScript.
• You are not prepared to own a volatile dependency. Five GitHub stars, zero dependents, and six BREAKING lines in one v5.1.0 minor bump are the honest signals: early-stage tooling, fast-moving API, concentrated contribution. The value is real, but treat maintenance as a recurring line item and keep the integration thin.

Adoption cautions specific to this release:

• Six breaking changes in a minor version. Pin exact versions, read every changelog, budget upgrade time per bump.
• Custom JudgeDriver implementations must be rewritten (generate() → judge(Evaluation): Verdict); any in-house driver becomes a maintenance liability tied to this package's API surface.
• Judge::using() and JudgeResponse are removed in favour of Judge::evaluate(). Grep for both before upgrading — silent behaviour change inside a test suite is the worst failure mode available.
• Custom drivers run only under --evals. If CI invokes plain pest, those assertions skip and the build looks green while judging nothing. The fix is one flag in the canonical CI config; the failure mode is invisible and will not announce itself.
• An invalid judge level now throws instead of scoring 0.0. Better hygiene, but it converts latent flakiness into a red build on first run. Name an owner who can distinguish "judge drifted" from "product regressed."
• Pin the judge model explicitly in every environment via PEST_EVALS_LARAVEL_SCORING_*, _EMBEDDING_*, and _CLASSIFICATION_* (or tests/Pest.php). Baselines recorded before an upgrade are not comparable.
• Guard tests/Evals with a beforeEach that skips when provider credentials are absent, so an intentional --evals run stays informative rather than failing for the wrong reason.

How to Pitch It (Stakeholders)

To executives: We can treat AI output quality like any other release criterion — tested, automated, and blocking. This is an open-source Pest plugin, so the cost is engineering time rather than a vendor contract, and it puts AI features behind the same merge gate we already trust for the rest of the product. Three decisions need your sign-off. First, a modest per-CI-run judge-model bill; evals are opt-in, so we control when that spend happens. Second, an upgrade cycle — the current version requires up-to-date Laravel AI and Pest and rewrites our custom judge code, which I estimate at one to two sprints including threshold reconciliation and re-baselining. Third, an honest risk note: this package is early-stage, has almost no external adoption, and shipped six breaking changes in one minor release. My recommendation is to adopt behind a non-blocking CI job first, pin the exact version, and only then promote it to a required check. The strategic payoff is concrete — we can gate routing and moderation features on a measured classification judge instead of on reviewer judgment.

To engineering: Pest-native evals — write a test, prompt the agent, assert on a verdict. expect(Agent::class)->prompt(...) runs the agent, and class names resolve through the container so constructor injection works. v5.1.0 moves to laravel/ai ^1.0 and Pest ^5.2.1 and replaces the driver contract with judge(Evaluation): Verdict; Judge::using() and JudgeResponse are gone in favour of Judge::evaluate(). Custom drivers now run only under --evals, which usefully keeps judge cost out of the default loop — but make sure that flag lives in the canonical CI config, not someone's local alias, or evals will silently skip. Start deterministic and escalate to model-assisted scoring only when a property cannot be expressed otherwise; scorers return 0.0–1.0 with a default threshold of 0.7, so raise it for safety-critical behaviour and lower it for acceptable partial answers. Use ->repeat() when reliability, not a single answer, is the property under test. Under LaravelAiJudge scores snap to five levels, so write thresholds on those boundaries and document the mapping in the harness README instead of leaving it tribal. A judgeUsing() closure is treated as a stub — legitimate for no-spend wiring smoke coverage in CI, but it proves nothing about model quality.

To the team owning quality: the one thing to settle before adoption is what "good enough" means at each level boundary, and who triages a judge failure versus a product failure. The tooling is cheap; the disagreement about thresholds is the actual cost. Two practical notes: because classification judging returns a chosen level and probability but no reasoning, keep LaravelAiJudge available for debugging the cases the classifier flags. And treat recorded baselines as versioned artifacts — every judge or model change requires re-baselining, and a stale baseline will read as a regression that never happened.