Pest Plugin Evals Laravel Package

Getting Started

Confirm the prerequisites before writing anything. This plugin is not standalone: v5.1.0 requires laravel/ai ^1.0 and pestphp/pest ^5.2.1. If you are on laravel/ai 0.x or Pest 4, stop — there is no partial-adoption path, and you should not adopt Pest purely to get evals.

Install and pin.

composer require --dev pestphp/pest-plugin-evals --with-all-dependencies

Then hard-pin the version in composer.json. This package shipped six BREAKING changes in a single minor bump; a floating constraint is an outage waiting for a Tuesday.

Understand the gate before you run anything. Evals do not execute under a plain pest invocation. You must opt in with --evals or PEST_EVALS=1. Also note that custom JudgeDriver classes only run under --evals, so this flag is not optional decoration — it is the difference between "judged nothing" and "green build you can trust."

Where to look first, in order:

1. tests/Evals/ — the conventional home. Keep evals next to but separate from unit tests.
2. tests/Pest.php — where you configure judge drivers, models, and the credentials guard.
3. Your agent class — expect() resolves class names through the container, so constructor injection works exactly as it does in production.

First use case: one behaviour, one prompt, one cheap assertion. Start with something a string can prove, then escalate only when it cannot.

// tests/Evals/AccountAgentTest.php
it('answers cancellation questions from the supplied context', function () {
    expect(AccountAgent::class)
        ->prompt('What is the cancellation window?')
        ->toContain('30 days');
});

Run it explicitly:

./vendor/bin/pest tests/Evals --evals

Only once that is green should you reach for toBeCorrect(), toBeSimilar(), or a classifier — each of those costs money per run.

Implementation Patterns

One describe() per agent, one it() per behaviour. This keeps failure output legible and lets you run a single behaviour while iterating. Name the it() after the property under test, not the prompt text.

Follow the cost ladder, ascending. Deterministic assertions (toContain, toMatch, toBe, toBeJson, toHaveToolCalls, toFollowTrajectory) cost nothing. Embeddings (toBeSimilar) cost a little. Judge calls (toBeCorrect, toSatisfy, toBeRelevant, toBeSafe) cost the most. Write the cheapest expectation that actually proves the behaviour, and reserve judge calls for properties that are genuinely semantic — same meaning, different wording.

Use repeat() when reliability is the property. ->repeat(3) through ->repeat(5) asserts that every sample passes. This is how you put a number on flakiness tolerance instead of arguing about a single green run.

expect(AccountAgent::class)
    ->prompt('What is the cancellation window?')
    ->repeat(3)
    ->toBeCorrect(expected: 'Customers can cancel within 30 days.');

->not-> inverts that to "no sample may match" — useful for prompt-injection and unsafe-advice cases.

Treat prompt() as stateless. Each call starts a fresh agent run; there is no conversation carry-over. Multi-turn flows become separate it() blocks, or a single prompt that embeds the history yourself. Attachments go in as the second argument to prompt().

Assert on tool behaviour separately from prose. toHaveToolCalls([...]) uses subset matching for optional arguments — you specify only the arguments you care about. toFollowTrajectory([...], strictOrder: bool) covers required ordering, with loose mode for parallel-safe agents. Both are free; use them before you pay a judge to tell you the answer looked plausible.

Reach for classification judging when the label matters more than the prose. judgeUsingLaravelAiClassifier() plus structured evaluations turns routing, triage, and moderation into verifiable categories. Configure it through the PEST_EVALS_LARAVEL_CLASSIFICATION_* env vars, alongside PEST_EVALS_LARAVEL_SCORING_* and PEST_EVALS_LARAVEL_EMBEDDING_* for the other paths. Do this in tests/Pest.php rather than scattering putenv() through the suite.

Write thresholds on the level boundaries. LaravelAiJudge, relevance, and safety scores snap to 0, 0.25, 0.5, 0.75, and 1.0. The default threshold is 0.7, which effectively means "must be 0.75 or above." Raise it for safety-critical behaviour; lower it where a partial answer is acceptable. Document the mapping in a harness README — it should not be tribal knowledge.

Guard the suite on credentials. Add a beforeEach in tests/Pest.php that skips when provider keys are absent, so an intentional --evals run on a laptop without keys produces "skipped" rather than a misleading red.

Wire the flag into canonical CI, then make it blocking. Put --evals in the committed CI config — not a local alias — as a dedicated job so judge spend is visible and budgetable per run. Run it non-blocking for a sprint, then promote to a required check.

Use Pest datasets for input sweeps. A dataset of prompts against one it() gives you breadth cheaply, as long as every entry uses deterministic expectations or you accept the multiplied judge cost.

Gotchas and Tips

The silent false-green is the headline risk. Evals skip unless --evals is passed, and custom drivers are gated the same way. If CI calls plain pest, the build is green while judging nothing, and nothing announces it. The fix is one flag; the failure mode is invisible. Give CI config ownership to a specific person.

Pin the judge model in every environment. The default became gpt-6-luna in v5.1.0. Un-pinned suites are not reproducible across upgrades, and any baseline recorded before the change is not comparable. Baselines must be re-derived after every judge or model change.

Judge::using() and JudgeResponse are gone. Grep for both before upgrading. They were replaced by Judge::evaluate(). A silent behaviour change in a test suite is the worst failure mode available — verify the call sites compile and that the assertions still mean what you think.

Custom drivers have a new contract. JudgeDriver::generate() became judge(Evaluation $evaluation): Verdict, and judgeUsing() closures now receive an Evaluation and return a float or a Verdict. If you maintain an in-house driver, treat it as a recurring maintenance line item tied to this package's API surface — keep the integration thin and the driver small.

judgeUsing() is a stub, not a confidence signal. A closure driver spends nothing and is a legitimate way to smoke-test the wiring in CI. It proves nothing about model quality. Use it to validate your harness, never to validate your product.

Invalid levels now throw. Previously a malformed judge level scored 0.0. This is better hygiene, but it converts latent flakiness into a red build on first run. Someone must be able to distinguish "judge drifted" from "product regressed" — a cheap tell is to re-run the failing case against the fixed baseline prompt.

Thresholds do not transfer between judges. LaravelAiClassifier weights all levels by probability and can legitimately produce an in-between score like 0.95. A threshold tuned under LaravelAiJudge will behave differently after a switch. Re-check, do not assume.

Classification gives you a label and a probability, not a reason. Keep LaravelAiJudge configured in a debugging profile so that when the classifier flags a failure you can ask a freeform judge why. Otherwise failure triage stalls at "it picked category B."

Coarse levels mask small drift by construction. The 0.25-step snapping is deliberate and makes thresholds stable and explainable — but a genuine 10% quality regression can be invisible. If fine-grained drift detection is on your roadmap, this package alone will not give it to you.

repeat() and ->not-> compose in ways that surprise people. Every expectation after repeat() must hold for every sample; with ->not->, no sample may match. A negation nested inside a repeated block is not "sometimes fails" — it is "never matches across all N runs," which is a much stricter bar.

Substring assertions are cheap and brittle. toContain('30 days') passes for "not 30 days." Prefer toMatch() with anchored patterns or toBe() on structured output parsed with toBeJson() first. When you find yourself writing a regex to prove a meaning, that is the signal to escalate to toBeSimilar() — not to write a longer regex.

Keep the blast radius small. Five stars, zero dependents, concentrated contribution, and six breaking lines in one minor version are the honest signals. Isolate evals in tests/Evals, wrap repeated setup in helpers defined in tests/Pest.php, pin exact versions, and read every changelog before bumping. The leverage is real — AI quality becomes a merge gate in syntax the team already writes — but only if upgrading stays a contained task rather than a refactor.