Pest Plugin Evals Laravel Package

Technical Evaluation

Architecture fit. pestphp/pest-plugin-evals v5.1.0 is test infrastructure, not a runtime dependency. It converts AI quality from review-time opinion into a blocking or non-blocking CI check written in the Pest syntax the team already uses. For a Laravel + Pest shop with AI behavior in production — summarisation, extraction, classification, support routing, agent tool use — the fit is strong because there is no new platform, no separate DSL, and no dashboard contract to negotiate. Provider and model selection flow through laravel/ai ^1.0, so a provider swap is configuration rather than a harness rewrite. The fit collapses for PHPUnit-native or non-PHP stacks; do not adopt Pest to acquire evals.

Integration feasibility. Installable in an afternoon; adoptable in 1–2 sprints for any team with a non-trivial existing eval suite. v5.1.0 is a broad breaking release inside a minor version and the blast radius is real:

• JudgeDriver::generate() → judge(Evaluation): Verdict rewrites every custom driver.
• judgeUsing() closures change signature and must return float or Verdict.
• Judge::using() and JudgeResponse are removed; grep both before upgrading.
• Scores snap to five levels (0, 0.25, 0.5, 0.75, 1.0) — the only change that alters behavior, not syntax.
• Custom JudgeDriver classes run only under --evals.
• An invalid judge level now throws instead of scoring 0.0.
• Default judge model changed to gpt-6-luna; pin explicitly.

Hard prerequisites are laravel/ai ^1.0 and pestphp/pest ^5.2.1. Most effort is not code: it is re-deriving threshold policy at level boundaries and re-baselining recorded scores.

Technical risk, ranked, with owners. Each needs a named owner before we gate a release on it.

1. Silent false-green — highest. Custom drivers and evals only run under --evals or PEST_EVALS=1. If canonical CI invokes plain pest, assertions skip and the build stays green while judging nothing. One flag fixes it; the failure mode is invisible. Owner: CI/platform, not individual developers.
2. Threshold semantics shift. Snapping to 0.25 steps makes gates stable and explainable but masks small regressions by construction. Note that LaravelAiClassifier weights levels by probability and can emit intermediate scores (e.g., 0.95), so thresholds tuned under LaravelAiJudge do not transfer. Owner: quality lead.
3. Dependency volatility. Five GitHub stars, zero dependents, six BREAKING lines in one minor bump, concentrated contribution. Pin exact versions, read every changelog, keep integration thin. Owner: engineering manager.
4. Judge-cost and reproducibility drift. Un-pinned models and provider config make baselines non-comparable across upgrades. Owner: whoever owns tests/Pest.php and the PEST_EVALS_LARAVEL_SCORING_*, _EMBEDDING_*, _CLASSIFICATION_* env vars.
5. Triage ambiguity. An invalid-level throw converts latent flakiness into a red build. Someone must distinguish "judge drifted" from "product regressed." Owner: on-call rotation.

Key questions to resolve before commitment. Are we on laravel/ai ^1.0 and Pest ^5.2.1 today, or is that a forced upgrade with API rewrites? Which behaviors must be proven deterministically, and which earn a judge call? What is the per-CI-run judge budget, and who approves it? Do we need fine-grained drift detection, in which case this is the wrong tool? Who owns threshold policy and failure triage? Is a non-blocking job acceptable for the first two release cycles?

Integration Approach

Stack fit. Adopt only where Laravel + Pest + laravel/ai ^1.0 are already the path. The expectation ladder is a cost/governance framework a TPM can set policy on: deterministic checks (toContain, toMatch, toBe, toBeJson, toHaveToolCalls, toFollowTrajectory) cost nothing; embeddings (toBeSimilar) cost a little; judge calls (toBeCorrect, toSatisfy, toBeRelevant, toBeSafe) cost the most. Start deterministic and escalate only when the property cannot be expressed otherwise. Scorers return 0.0–1.0 with a default threshold of 0.7; raise for safety-critical behavior, lower for acceptable partial answers.

Migration path. Sequence in four stages. First, stand up a non-blocking CI job pinned to an exact version, with tests/Evals and a beforeEach that skips when provider credentials are absent so an intentional --evals run stays informative. Second, port existing evals to the v5.1.0 contract — rewrite custom drivers to judge(Evaluation): Verdict, replace Judge::using() and JudgeResponse with Judge::evaluate(), and update judgeUsing() closure signatures. Third, re-baseline: re-derive thresholds on the 0.25 boundaries, document the mapping in the harness README instead of leaving it tribal, and re-check any threshold carried over from LaravelAiJudge if switching to LaravelAiClassifier. Fourth, promote to a required check only after two clean release cycles and a documented triage runbook.

Compatibility. Pin exact versions of this plugin, Pest, and laravel/ai, and pin the judge model explicitly in every environment — the default moved in this release and un-pinned suites are not reproducible. Guard against the --evals skip trap by putting the flag in canonical CI config, never in a local alias. If CI must run credential-free, a judgeUsing() closure is treated as a stub and is a legitimate no-spend smoke test for wiring — but it proves nothing about model quality.

Sequencing decisions for the TPM. Approve eval spend per CI run rather than per developer loop; use evals for merge gating, not dashboards; pair with a reporting tool if executive trend lines are required. Use ->repeat(3) through ->repeat(5) when reliability rather than a single answer is the property under test; ->not-> asserts no sample matches. Treat model and vendor migration decisions as eval-backed comparisons on identical inputs, not as judgment calls.

Operational Impact

Maintenance. Budget a recurring line item: this is early-stage tooling with a fast-moving API and concentrated contribution. Six breaking changes in one minor version means pinning, changelog review, and upgrade time per bump are non-negotiable. Any in-house JudgeDriver is now a maintenance liability tied to this package's API surface — keep custom drivers few, thin, and covered by the smoke path. Owner: engineering manager with a standing upgrade slot each cycle.

Support. Two support surfaces, not one. First, the harness: thresholds, datasets, env configuration, and the README mapping. Second, the triage path: who distinguishes judge drift from product regression, and what the escalation is when an invalid level throws. Because classification judging reports the chosen level and its probability but not its reasoning, keep LaravelAiJudge available to debug cases the classifier flags. A failure tells you what was picked, not why — staff accordingly.

Scaling. Evals scale with judge-call volume and dataset size, not with application traffic. Per-run cost is controllable because evals are opt-in; set a per-run budget expectation with finance and review it when repeat counts or judge-heavy assertions grow. The coarser limit is analytical: five-level snapping masks fine-grained drift by construction, and this is an assertion library, not an eval platform. If the roadmap depends on statistical confidence intervals, multi-turn agent simulation at scale, or large human-labelled golden datasets, pair it with something or buy.

Failure modes to instrument. Silent false-green from a missing --evals flag — mitigate in CI config and assert the job actually judged something. Threshold drift after a judge or model change — mitigate by re-baselining and documenting the mapping. Credential-absent failures on intentional runs — mitigate with a skip guard. Judge-model default changes — mitigate by pinning. Flaky single-sample verdicts — mitigate with ->repeat() and by writing assertions deterministic-first. Red builds on first upgrade from invalid-level throws — mitigate by owning the alert before the upgrade lands.

Ramp-up. Engineering: low — a Pest test that prompts an agent and asserts a verdict; v5.1.0 changes are mechanical once the contract is known. Organizational: higher. The real cost is the disagreement about what "good enough" means at each level boundary and who triages a judge failure versus a product failure. Budget one to two sprints including threshold reconciliation, and sequence adoption as non-blocking first, required check second. Keep the integration thin so a future replacement — dedicated eval platform for datasets, trend reporting, and drift alerting — is a swap of the reporting layer, not a rewrite of the quality gate.