Phpunit Randomizer Laravel Package

Getting Started

1. Install as a dev dependency so it never ships to production:

   composer require --dev fiunchinho/phpunit-randomizer
   

2. Look first at three places in vendor/fiunchinho/phpunit-randomizer/:

   • README.md — the authoritative usage for the version you actually pulled (last release: 2018-05-09, so docs matter more than blog posts).
   • composer.json → the bin section — tells you whether the package ships a phpunit-randomizer binary or expects a bootstrap/extension hook.
   • composer.json → require / require-dev — confirms which PHPUnit major it was built against (this era targets PHPUnit 5–7). If your project runs PHPUnit 10/11, expect wiring to fail before you even start.

3. Run it against your existing config rather than a new one:

   # adjust to whatever the bin section exposes
   vendor/bin/phpunit-randomizer -c phpunit.xml
   

   If no binary exists, the README will tell you to point a bootstrap or a custom test suite at the package (e.g. register a randomizing TestSuite/listener in phpunit.xml or a tests/bootstrap.php).

4. First use case: establish a green baseline with the normal vendor/bin/phpunit run, then run the randomizer 5–10 times on a suite that already passes deterministically. Any new failure is a hidden inter-test dependency — exactly the class of bug this package exists to surface. If the suite is already flaky, fix determinism first; randomization on top of an unstable suite just makes triage impossible.

Implementation Patterns

Baseline → randomize → replay. Never randomize a suite you haven't first seen pass in file order. The workflow is: (1) deterministic run green, (2) random run, (3) on failure, capture the seed printed by the runner and re-run with that seed pinned so the failure is reproducible, (4) fix isolation, (5) repeat.

Seed pinning as the debugging contract. The seed is the single most important artifact of a random run. Log it in local output and surface it in CI (e.g. echo the seed in a before_script step). A random failure you can't replay is a random failure you can't fix. If the package exposes a seed flag/environment variable, wire it into a Composer script:

"scripts": {
  "test": "phpunit",
  "test:random": "phpunit-randomizer -c phpunit.xml",
  "test:seed": "phpunit-randomizer -c phpunit.xml --order-seed=$SEED"
}

(Flag names vary by version — confirm against the README.)

CI strategy: don't randomize every build. Keep the primary pipeline fast and deterministic, and run randomization on a schedule or a matrix of seeds:

# nightly or a separate job
- run: for s in 1 2 3 4 5; do vendor/bin/phpunit-randomizer --order-seed=$s || exit 1; done

This gives deterministic feedback on PRs and stochastic coverage nightly.

Isolation-audit loop. When a random run fails, capture the seed, re-run pinned with --stop-on-failure and -v, then bisect with --filter over the failing test plus its likely polluter. The failure report plus seed usually points at the pair quickly.

Pair it with isolation-friendly test infrastructure. Random order only pays off if each test owns its state: RefreshDatabase / per-test transactions, factory-built data, no reliance on rows or files left by earlier tests, no ordering assumptions in fixtures.

Parallel runners. If you use paratest or PHPUnit's own process isolation, confirm the randomizer's ordering actually applies — some runners re-partition tests after the suite is built. Randomization and parallelization are orthogonal goals, but the interaction is easy to get wrong.

Local pre-push habit. Run the randomizer once or twice before pushing on branches that touch shared fixtures, base test cases, or static/singleton state. It's cheap insurance for exactly the tests most likely to be coupled.

Extension point. The package is a thin suite/loader wrapper; if you need a different shuffle policy (e.g. always shuffle methods but keep classes grouped), the natural place is a subclass of its suite class or a small wrapper around your bootstrap. Keep this in your own repo, not a fork.

Gotchas and Tips

• Randomness is stochastic, not exhaustive. A single random pass can miss a dependency. Run multiple seeds; treat "no failures in N runs" as evidence, not proof.
• Seed reproducibility depends on the RNG. If the shuffle uses PHP's shuffle()/mt_rand with mt_srand(), the same seed should replay on the same PHP major — but pin PHP versions between local and CI, and don't assume cross-major reproducibility. Verify by replaying a known seed on both.
• @depends and setUpBeforeClass still impose order. Those are legitimate ordering constraints, not bugs. Don't "fix" them; exclude or accept them.
• Alphabetical naming hacks break. Suites using 01_, 02_ prefixes or filename ordering to sequence tests will fail under randomization — that's the point, but expect noise on the first runs.
• Static and global state are the usual culprits. Singletons, container bindings, config/env mutations, time, superglobals, and static caches all leak across tests. Randomization exposes them; fix with teardown resets, not with ordering.
• Random order does not shuffle data providers (or individual assertions). If your provider rows depend on execution state, this package won't catch it.
• Output gets harder to read. Verbose random runs interleave differently; rely on the seed plus test names for triage rather than diffing logs line-for-line.
• PHPUnit version decay is the biggest risk. PHPUnit gained native --order-by=random and --random-order-seed around 7.2+ and matured it in later majors; for a 2018 package, check whether native random ordering already covers your need before adding an unmaintained dependency. If you're on a modern PHPUnit, prefer the built-in feature and drop the package.
• XML schema drift breaks custom wiring. Custom test suite classes and listener/extension hooks registered in phpunit.xml are the most version-sensitive part; if the runner throws on startup, check the schema and the target PHPUnit major first.
• Don't evaluate performance under random order. Cache warm-up and shared fixtures make timings meaningless across orderings; benchmark separately.
• Budget CI time. Nightly seed sweeps multiply suite runtime by the number of seeds; cap it (3–5 seeds is usually enough) and keep the deterministic job as the merge gate.
• Bootstrap/autoload ordering still matters. If the randomizer is loaded via bootstrap, ensure it's required after Composer's autoloader but before PHPUnit builds the suite, or the hook silently won't apply and you'll get deterministic runs that look random.