Weave Code
Pest Plugin Evals Laravel Package
pestphp/pest-plugin-evals
A Pest PHP plugin for writing and running evals directly inside your test suite. Define evaluation cases, assert on outputs with familiar Pest syntax, and benchmark prompts or AI-powered features alongside your regular tests, with results shown in your normal test output.
A PestPHP plugin for evaluating LLM agents with LLM-as-judge, semantic similarity, and deterministic scorers
Frequently asked questions about Pest Plugin Evals
- What is pestphp/pest-plugin-evals and how does it work with Pest?
- It is a Pest PHP plugin that lets you write LLM evals as ordinary Pest tests: define evaluation cases, run prompts against a model, and assert on response quality using fluent assertions and custom scorers. Results appear in your normal test output, so prompt and AI feature quality becomes part of your existing suite instead of a separate platform. It is test infrastructure only, not a runtime dependency, and it requires laravel/ai ^1.0 and pestphp/pest ^5.2.1.
- How do I install pest-plugin-evals in a Laravel app?
- Install it with Composer alongside its peer requirements, laravel/ai ^1.0 and pestphp/pest ^5.2.1, then add eval files to your test suite. Run them with the --evals flag or by setting PEST_EVALS=1. Pin an exact version, because the project is young and has shipped breaking changes inside a minor release.
- Which Pest and Laravel AI versions does pest-plugin-evals v5.1.0 require?
- Version 5.1.0 requires pestphp/pest ^5.2.1 and laravel/ai ^1.0, and provider and model selection flows through laravel/ai. Confirm your current Pest version before adopting, because upgrading from an earlier major means rewriting test APIs. Check the package composer.json for the supported PHP range.
- Why are my evals not running in CI?
- Custom JudgeDriver classes and evals only execute when Pest is invoked with --evals or when PEST_EVALS=1 is set. If canonical CI runs plain pest, the eval assertions are skipped and the build stays green while judging nothing, which is a silent false-green. Fix the flag once in the CI pipeline, owned by platform rather than individual developers, and assert it is present.
- How do I write a dataset-driven eval in Pest?
- You write evals as Pest tests using dataset-driven cases for prompts and expected outputs, then assert on the model response with the plugin's fluent assertions. Deterministic checks such as toContain, toMatch, toBe, toBeJson, toHaveToolCalls, and toFollowTrajectory are the cheapest. Start deterministic and escalate to a judge call only for properties you cannot express any other way.
- What assertions and scorers can I use on LLM responses?
- The plugin provides deterministic assertions, embedding-based similarity via toBeSimilar, and judge-backed assertions such as toBeCorrect, toSatisfy, toBeRelevant, and toBeSafe. You can also register custom scorers for domain-specific criteria, with scorers returning a score between 0.0 and 1.0. Custom JudgeDriver classes only run under --evals, so keep that in mind when debugging.
- How do I set a passing threshold for an eval?
- Scorers return a value from 0.0 to 1.0 and the default threshold is 0.7, which you can override per assertion or scorer. Scores snap to five levels (0, 0.25, 0.5, 0.75, 1.0), so thresholds effectively sit at level boundaries. Re-derive threshold policy and re-baseline recorded scores after any judge or model upgrade.
- What breaking changes are in pest-plugin-evals v5.1.0?
- v5.1.0 is a broad breaking release: JudgeDriver::generate() becomes judge(Evaluation): Verdict, judgeUsing() closures must return a float or a Verdict, and Judge::using() plus JudgeResponse are removed. An invalid judge level now throws instead of scoring 0.0. Grep for the removed APIs before upgrading and expect to rewrite every custom driver.
- How do I control the judge model and eval costs per CI run?
- The default judge model changed to gpt-6-luna, so pin the model explicitly and keep provider configuration under version control. Embedding, classification, and scoring behavior is tuned through the PEST_EVALS_LARAVEL_SCORING_*, _EMBEDDING_*, and _CLASSIFICATION_* environment variables. Set a per-CI-run judge budget and prefer cheap deterministic assertions first, since un-pinned models make baselines non-comparable across upgrades.
- Should I use pest-plugin-evals, or is there a better alternative?
- Adopt it only if Laravel, Pest, and laravel/ai ^1.0 are already your stack; do not switch to Pest just to get evals, and PHPUnit-native or non-PHP teams should use another harness. It is also the wrong tool if you need fine-grained drift detection, because scores snap to 0.25 steps. Weigh the risk of a young package with concentrated contribution by pinning exact versions and keeping the integration thin.
Weaver
How can I help you explore Laravel packages today?