Psalm Tester Laravel Package

Getting Started

Install it as a dev dependency and make sure Psalm is resolvable (0.4 requires vimeo/psalm ^6.10 || ^7.0.0-beta16 and composer-runtime-api ^2):

composer require --dev alies-dev/psalm-tester vimeo/psalm

There are two entry points, and the choice determines how much code you write:

• PsalmPhptTestCase (0.4+) — recommended. You subclass it, implement phptDirectory(), and it handles fixture discovery, --SKIPIF-- evaluation, one batched Psalm run, and --filter narrowing of which fixtures get analyzed. Roughly 15 lines of test code for a whole fixture suite.
• PsalmTester + PsalmTest::fromPhptFile() — the lower-level API. Use this when you need per-test control, custom PHPUnit assertions, or to wire results into your own reporting.

First use case: pin down one inferred type. Create tests/fixtures/array_values.phpt:

--FILE--
<?php

/** @psalm-trace $_list */
$_list = array_values(['a' => 1, 'b' => 2]);

--EXPECT--
Trace on line 9: $_list: non-empty-list<1|2>

Then a single test class:

final class TypeRegressionTest extends PsalmPhptTestCase
{
    protected static function phptDirectory(): string
    {
        return __DIR__ . '/fixtures';
    }

    protected static function psalmArguments(): array
    {
        // Point at YOUR project config, not the package default.
        return ['--no-progress', '--no-diff', '--config=' . dirname(__DIR__) . '/psalm.xml'];
    }
}

Run vendor/bin/phpunit --filter array_values. The fastest way to author a fixture is to write the --FILE-- section first, run Psalm on that snippet manually with the same arguments, and paste the real output into --EXPECT--; guessing at Psalm's exact formatting is a waste of time.

Where to look first: src/psalm.xml (the package's default config — confirm it isn't silently what you're using), the README section on --ARGS-- and runBatch(), and UPGRADING.md if you're coming from 0.3. Also read PsalmTester::runBatch() / runGroup() directly; the README defers to source for the batching details, and you'll want to understand the grouping key before you have 200 fixtures.

Implementation Patterns

The authoring loop is write → run → capture → freeze. For every type behavior you care about, write a minimal self-contained snippet with /** @psalm-trace $var */, run it once, copy the actual trace into the expectation, and commit. This is the single highest-leverage habit: fixtures are cheap, and each one is a permanent assertion that survives Psalm upgrades.

Prefer EXPECTF for anything with a line number. The moment you add a line above the trace, --EXPECT-- breaks. Trace on line %d: $_list: non-empty-list<%s> is stable against edits and against Psalm changing its generic rendering slightly. Reserve exact --EXPECT-- for cases where the precise type string is the contract you're asserting.

Use --XFAIL-- for known upstream bugs and pending refactors. This is the pattern that pays off most in day-to-day maintenance: it reports as incomplete while broken, and flips to a failure the moment Psalm fixes the bug. That gives you an automatic expiry signal instead of a skipped test that rots silently. Whenever you find yourself writing "we'll fix this later" in a fixture, use --XFAIL-- instead of markTestSkipped().

Batch aggressively if your plugin has expensive boot cost. If you use psalm-plugin-laravel (which boots a full application per Psalm process), the per-file default is unusable. Either runBatch() (0.3+) or PsalmPhptTestCase (which batches for you) collapses this into one Psalm invocation per argument set. The reference case — 758 fixtures — went from ~28.6s to ~10s wall, and ~4.6s under --filter.

Guard environment-specific fixtures with --SKIPIF--. Version gates and extension checks belong in the fixture, not in PHP conditional logic in the test class:

--SKIPIF--
<?php if (PHP_VERSION_ID < 80200) { echo 'skip requires PHP 8.2+'; }
--FILE--
...

getSkipReason() returns the reason with the skip token stripped, or null. The script runs in a separate PHP process, so exit()/die() inside it is safe.

Reach for --CONFLICTS-- when fixtures share state. Fixtures that register the same symbol, or that would interfere, get their own Psalm run; fixtures sharing a key never overlap; all runs alone. This is what lets you raise withConcurrency() without introducing flakiness.

Drive CI and dashboards off the structured API, not parsed text. 0.4's run() / runOne() returning a Result with an Outcome, formatted output, and structured Issues means gating and trend-collection are integration work. If you want a "type coverage over time" graph or a PR comment listing changed inference, this is the hook — don't scrape console output.

Bound every run in CI. withTimeout(), withConcurrency(), withWorkingDirectory(), withEnv() together make analysis deterministic. Timeout kills Psalm and its child processes, so a hung worker errors the affected tests rather than stalling the pipeline.

Set up a directory convention early. tests/fixtures/*.phpt plus a single PsalmPhptTestCase subclass keeps the PHPUnit config trivial and makes --filter names predictable. Name fixtures after the behavior (array_values_preserves_keys.phpt), not after the issue number.

Gotchas and Tips

0.4.0 is a breaking release — budget an upgrade pass. Specifically: --ARGS-- now appends to the configured arguments rather than replacing them (a --config inside it still replaces the configured config); arguments no longer go through a shell and withArguments() takes one argument per parameter; --EXPECT_EXTERNAL-- / --EXPECTF_EXTERNAL-- are rejected outright (inline the expectation); the human-facing progress output was removed, leaving only a single STDERR start line (psalm-tester: N phpt files (S skipped), G Psalm runs); and repeated sections now throw Duplicate section --X-- instead of silently overwriting. --CLEAN--, --ENV--, and --INI-- throw explicit "not supported" messages.

The default config is the package's own src/psalm.xml. This is the most likely reason a suite "passes but tests nothing." Always pass --config= pointing at your project's psalm.xml, and verify with a fixture you know should fail.

Batching shares one global symbol table. All fixtures in an argument group are analyzed in a single Psalm run, so any duplicate class or function name produces DuplicateClass / DuplicateFunction. Two mitigations: namespace every fixture's declarations uniquely (e.g. Fixture\ArrayValues\...), or split into --CONFLICTS-- groups. Prefer unique namespaces — it preserves batching speed.

Expectation drift from line numbers and whitespace is the #1 maintenance tax. --EXPECT-- compares against Psalm's formatted output, which includes the line number from the fixture body. Any edit above the trace invalidates the fixture. Default to EXPECTF and only tighten to exact matching when the string form is genuinely the contract.

A crashed Psalm run, unexpected exit status, non-issue-list output, or issues reported in other files now error the affected tests. That's a deliberate correctness improvement, but it means a fixture whose snippet accidentally pulls in a broken project file will fail the fixture rather than pass quietly. Keep --FILE-- bodies self-contained, and make sure your psalm.xml doesn't drag unrelated files into the analyzed set.

Each run gets --no-cache and its own cache directory. Cold every time, deterministic every time. Don't try to point multiple runs at a shared cache directory to speed things up; you'll trade determinism for a marginal gain.

withTimeout() on Windows kills only the Psalm process, not its children. On Windows CI agents, leave generous margins or avoid the timeout path entirely, and treat orphaned processes as a known risk.

Maturity caveat: 1 star, 0 dependents, single maintainer. That's not disqualifying for a dev-only test harness — the blast radius is your test suite — but pin the version in composer.json, read PsalmTester::runBatch() and runGroup() so you can debug without upstream help, and be prepared to vendor or patch. The release cadence is active (0.4.0 is a substantial improvement), which is a point in its favor.

Debug by reproducing outside the harness. When a fixture fails confusingly, extract the --FILE-- body to a real file and run Psalm on it with the exact same argument string. Result exposes the formatted output and structured Issues, so assert on those rather than on the failure message PHPUnit prints above the diff.

Don't over-invest before you have a reason. If all you need is three type assertions in one file, a single smoke fixture plus @psalm-trace is enough. The batch infrastructure, --CONFLICTS--, and the structured Result API earn their keep once you're maintaining a plugin, a stub set, or a framework integration where type inference is the product surface.