Mutation Testing

Note: Before you can start using mutation testing, you need to have a successfully running test suite.

Source code: Pest Plugin Mutate

To start using Pest's Mutation plugin, you need to require the plugin via Composer.

composer require pestphp/pest-plugin-mutate --dev

Run mutation testing

Run from CLI

You can run the mutation testing from the CLI providing the --mutate option.

vendor/bin/pest --mutate

When you are working with a larger codebase, checkout the performance section first, otherwise you will not have satisfying experience.

By default, it uses the default configuration (if available) or you can use a different configuration by providing its name.

vendor/bin/pest --mutate="arithmetic only"

You can set or overwrite all options available.

vendor/bin/pest --mutate --path=src

Additionally, you can make use of most of the options available in Pest.
For example this would only run mutation tests for code covered by tests in the "unit" group.

vendor/bin/pest --mutate --covered-only --group=unit

mutate()

Another powerful technique is to call the mutate() function directly in your test file. This automatically start mutation testing ones you run vendor/bin/pest. Additionally it limits the test run, to the tests in this file.

This function is intended to be used in your daily development workflow to establish a mutation testing practice right when you are implementing or modifying a feature.

By default, it inherits the default configuration. You can change this by providing an alternative configuration name.

In conjunction with the next release of Pest, it will be possible to append the mutate() function direct to an individual test case or a describe block.

mutate();

test('sum', function () {
  $result = sum(1, 2);

  expect($result)->toBe(3);
})

Executing the ./vendor/bin/pest command will now automatically run mutation testing. It is not necessary to provide the --mutate option.

You can append options after calling mutate().

->mutate()
  ->path('src/functions.php')

test('sum', function () {
  $result = sum(1, 2);

  expect($result)->toBe(3);
});

Configuration

You can globally configure mutation testing in you Pest.php file.

mutate()
    ->paths('src');

For all the available options see Options section.

Alternative configurations

You can create multiple mutation testing configurations.

use Pest\Mutate\Mutators;

mutate('arithmetic only') // 'default' if not provided
    ->paths('src')
    ->mutators(Mutators::SET_ARITHMETIC);

And you can inherit from another configuration.

WIP: Configuration inheritance is not implemented yet!

use Pest\Mutate\Mutators;

mutate('arithmetic only')
    ->extends('default')
    ->mutators(Mutators::SET_ARITHMETIC);

Options

The following options are available.

• path()
• ignore()
• class()
• mutator()
• except()
• coveredOnly()
• uncommittedOnly()
• changedOnly()
• stopOnEscaped()
• stopOnNotCovered()
• bail()
• retry()
• min()
• ignoreMinScoreOnZeroMutations()
• --id
• --no-cache
• --clear-cache

---

path()

CLI: --path

Limit the directories or files to mutate by providing one or more paths to a directory or file to test.

If no paths are provided, it defaults to the source directories configured in your phpunit.xml file.

mutate()
    ->path('src');

ignore()

CLI: --ignore

Ignore one or more directory or file paths.

mutate()
    ->ignore('src/Contracts');

class()

CLI: --class

Limit the mutations to one or more classes by providing one or more class names.

mutate()
    ->class(MyClass::class, OtherClass::class);

mutator()

CLI: --mutator

Choose the mutators you want to use. Choose from various sets or provide individual mutators. If not set, Mutators::SET_DEFAULT is used.

A list of all available mutators can be found in the Mutator Reference.

use Pest\Mutate\Mutators;

mutate()
    ->mutator(Mutators::SET_ARITHMETIC);
// or
mutate()
    ->mutator(Mutators::ARITHMETIC_PLUS_TO_MINUS, Mutators::ARITHMETIC_MINUS_TO_PLUS);

On the CLI you can provide a comma separated list of mutator names.

vendor/bin/pest --mutate --mutator=ArithmeticPlusToMinus,ArithmeticMinusToPlus

except()

CLI: --except

Exclude specific mutators from being used. Especially useful if you want to use a set of mutators but want to exclude some of them.

use Pest\Mutate\Mutators;

mutate()
    ->mutators(Mutators::SET_ARITHMETIC);
    ->except(Mutators::ARITHMETIC_PLUS_TO_MINUS);

coveredOnly()

CLI: --covered-only

Limit mutations to code that is covered by tests. This is especially helpful if you are running only a subset of your test suite. See Only run parts of your test suite.

mutate()
    ->coveredOnly();

uncommittedOnly()

CLI: --uncommitted-only

Limit mutations to code that has uncommitted changes.

mutate()
    ->uncommittedOnly();

changedOnly()

CLI: --changed-only

Limit mutations to code that has changed relative to a common ancestor of the given branch (defaults to main).

mutate()
    ->changedOnly(); // or ->changedOnly('add-xyz');

stopOnEscaped()

CLI: --stop-on-escaped

Stop execution upon first escaped mutant.

mutate()
    ->stopOnEscaped();

stopOnNotCovered()

CLI: --stop-on-not-covered

Stop execution upon first not covered mutant.

mutate()
    ->stopOnNotCovered();

bail()

CLI: --bail

Stop execution upon first not covered or escaped mutant.

mutate()
    ->bail();

retry()

CLI: --retry

If a mutation previously escaped, you typically want to run them first. In such cases, you can use the --retry option.

The --retry flag reorders your mutations by prioritizing the previously escaped mutations. If there were no past escaped mutations, the suite runs as usual.

Additionally, it will stop execution upon first escaped mutant.

mutate()
    ->retry();

min()

CLI: --min

Enforce a minimum mutation score threshold. For more information see Minimum Score Threshold Enforcement.

mutate()
    ->min(100);

You can pass an optional second parameter to ignore the minimum score threshold if zero mutations are generated. In this case Pest will exit with code 0.

mutate()
    ->min(100, failOnZeroMutations: false);

ignoreMinScoreOnZeroMutations()

CLI: --ignore-min-score-on-zero-mutations

Ignores the minimum score threshold if zero mutations are generated. In this case Pest will exit with code 0.

mutate()
    ->ignoreMinScoreOnZeroMutations();

--id

Run only the mutation with the given ID. You can find the ID of a mutation in the console output of a previous run.

vendor/bin/pest --mutate --id=fa6913f68aa87747

--no-cache

Disables the cache (This option is only available on the cli).

vendor/bin/pest --mutate --no-cache

--clear-cache

Clears the cache (This option is only available on the cli).

vendor/bin/pest --mutate --clear-cache

Performance

Mutation testing is potentially very time-consuming and resource intensive because of the sheer amount of possible mutations and tests to run them against.

Therefore, Pest Mutation Testing is optimized to limit the amount of mutations and tests to run against as much as possible. To achieve this, it uses the following strategies:

• Limit the amount of possible mutations by having a carefully chosen set of mutators
• Run only tests that covers the mutated code
• It tries to reuse cached mutations
• Run mutations in a reasonable order
• Provide options to stop on first escaped or not covered mutation

But there is much more you can do to improve performance. Especially if you have a larger code base and/or you are using mutations testing while developing locally.

Use a code coverage driver

If you have a code coverage driver available, Pest will use it to only run tests that cover the mutated code.

Supports XDebug 3.0+ or PCOV.

Reduce the number of files to mutate

Reduce the number of mutations by only mutating a subset of your code base.

vendor/bin/pest --mutate --path=src/path/file.php

Reduce the number of classes to mutate

Reduce the number of mutations by only mutating a subset of your classes.

vendor/bin/pest --mutate --class="App\\MyClass,App\\OtherClass"

Only run parts of your test suite

Reduce the number of mutations and tests to execute by only running a subset of your test suite.

vendor/bin/pest --mutate --filter=SumTest

For more filter options see Filtering.

Only create mutations for covered files / lines

Reduce the number of mutations by only mutating code that is covered by tests. This is especially helpful if you are running only a subset of your test suite. See Only run parts of your test suite.

vendor/bin/pest --mutate --covered-only

Attention: Code not covered by tests will not be mutated. Ensure your test suite covers all code you want to mutate.

Run tests in parallel

Run tests against multiple mutations in parallel. This can significantly reduce the time it takes to run mutation tests.

Against a single mutation the tests are not run in parallel, regardless of the parallel option.

vendor/bin/pest --mutate --parallel

Reduce the number of mutators

Reduce the number of mutations by choosing a smaller set of mutators.

vendor/bin/pest --mutate --mutator=ArithmeticPlusToMinus

Profiling

You can profile the performance of the mutations by using the --profile option. It outputs a list of the ten slowest mutations.

vendor/bin/pest --mutate --profile

Ignoring Mutations

Ignore for a single line

Sometimes, you may want to prevent a line from being mutated. To do so, you may use the [@pest-mutate-ignore](https://github.com/pest-mutate-ignore) annotation:

if($age >= 18) // [@pest-mutate-ignore](https://github.com/pest-mutate-ignore)
    // ...
];

If you want to ignore only a specific mutator, you can add a comma separated list of mutator names:

if($age >= 18) // [@pest-mutate-ignore](https://github.com/pest-mutate-ignore): GreaterOrEqualToGreater
    // ...
];

Ignore for multiple lines

To ignore mutations on large parts of the code you can add the annotation to a class, method or statement to ignore all mutations within the elements scope.

To ignore only specific mutators, you can add a comma separated list of mutator names: [@pest-mutate-ignore](https://github.com/pest-mutate-ignore): GreaterOrEqualToGreater,IfNegated

Class level

/**
 * [@pest-mutate-ignore](https://github.com/pest-mutate-ignore)
 */
class Test {
    // ...
}

Method or function level

/**
 * [@pest-mutate-ignore](https://github.com/pest-mutate-ignore)
 */
public function test() {
    // ...
}

Statement level

/** [@pest-mutate-ignore](https://github.com/pest-mutate-ignore) */
for($i = 0; $i < 10; $i++) {
    // ...
}

Minimum Score Threshold Enforcement

Just like code coverage, mutation coverage can also be enforced. You can use the --mutate and --min options to define the minimum threshold value for the mutation score. If the specified threshold is not met, Pest will report a failure.

./vendor/bin/pest --mutate --min=100

If zero mutations are generated, the score is considered to be 0 and Pest will report a failure. You can use the --ignore-min-score-on-zero-mutations option to ignore the minimum score threshold if zero mutations are generated. In this case Pest will exit with code 0.

./vendor/bin/pest --mutate --min=100 --ignore-min-score-on-zero-mutations

Custom Mutators

WIP: Custom mutators are not implemented yet!

You may want to create your own custom mutators. You can do so by creating a class that implements the Mutator interface.
This example will remove use statements.

namespace App\Mutators;

use Pest\Mutate\Contracts\Mutator;
use PhpParser\Node;
use PhpParser\NodeTraverser;

class RemoveUseStatement implements Mutator
{
    public static function can(Node $node): bool
    {
         return $node instanceof Node\Stmt\Use_;
    }

    public static function mutate(Node $node): int
    {
        return NodeTraverser::REMOVE_NODE;
    }
}

Afterward you can use your mutator.

use App\Mutators\RemoveUseStatement;

mutate()
    ->mutators(RemoveUseStatement::class);

In the CLI you must provide the full class name.

vendor/bin/pest --mutate --mutators="App\\Mutators\\RemoveUseStatement"