The idea: plant a bug and see if the tests notice
Green tests do not prove much on their own. Maybe they would also be green if the code were
wrong. Mutation testing checks this directly. A tool makes many copies of the
code, each with one small change, such as - turned into +. Each copy is
a mutant. Then it runs the whole test suite against every mutant.
- If at least one test fails, the mutant is killed. The tests noticed the change.
- If every test passes, the mutant survived. A real bug of that kind would get past the tests too.
The code here is fare(from, to), a bus fare: 2 per stop, but never more than 15, which
is reached from 8 stops on. The suite starts with three tests. All three pass, and together they
run every line. Demo 1: one mutant, killed.
int fare(int from, int to) {
1 int d = to - from;
2 if (d < 0)
3 d = -d;
4 if (d >= 8)
5 return 15;
6 return 2 * d;
}
Mutation operators
A mutation operator is a rule for one kind of small change. The changes copy typical slips a programmer makes. The six mutants on the page:
| Mutant | Change | Operator | With T1, T2, T3 |
|---|---|---|---|
| M1 | to - from → to + from | arithmetic | ✓ killed by T1 |
| M2 | d < 0 → d <= 0 | boundary | equivalent |
| M3 | d = -d → d = d | negation | ✓ killed by T2 |
| M4 | d >= 8 → d > 8 | boundary | ✗ survived |
| M5 | return 15 → return 0 | return value | ✓ killed by T3 |
| M6 | 2 * d → 2 + d | arithmetic | ✗ survived |
Each mutant has exactly one change. The idea behind it is the coupling effect: tests that catch every simple slip tend to catch bigger bugs made of several slips too.
Killed or survived: reach, infect, propagate
A test kills a mutant only if four things happen in the same run:
- Reach: the test runs the changed line.
- Infect: the changed code gives a different value there.
- Propagate: the difference reaches the result.
- Check: an assertion compares the result with the expected value.
Demo 2: a survivor at 100% line coverage runs M4. Every test reaches line 4. T1 and T2
reach it with d = 2, where d >= 8 and
d > 8 are both false. T3 has d = 20, where both are true. No test
has d = 8, the one value where they differ. M6 survives by coincidence: T1 and T2
both have d = 2, and 2 + 2 happens to equal 2 × 2.
Mutation score versus line coverage
The mutation score is the share of mutants the suite kills:
mutation score = killed / (mutants − equivalent mutants)
For T1, T2, T3 it is 3 / (6 − 1) = 60%, while line coverage is 6/6 = 100% (Demo 3: whole suite, score 60%). Coverage says which lines ran. The score says whether the tests would have noticed if those lines were wrong.
The clearest case is a test without an assertion. Demo 7: no assertions: 100% lines, 0% keeps the same three calls but checks nothing. Every line still runs, and not one mutant is killed.
Equivalent mutants
M2 changes d < 0 to d <= 0. The two only disagree when
d = 0. Then the mutant runs line 3, d = -d, and minus zero is zero. The
result is the same for every input, so no test can ever kill M2. It is an
equivalent mutant. Demo 4: an equivalent mutant.
Deciding whether a mutant is equivalent cannot be done by a program in general, so a person has to look at each survivor. Equivalent mutants are left out of the score; counting M2 would make the best possible score 5/6 = 83%.
Turning a survivor into a test
A survivor points at the test that is missing:
| Survivor | Why it lived | New test | Mutant gives |
|---|---|---|---|
M4 d > 8 | no test at the boundary d = 8 | T4: fare(0, 8), expect 15 | 16 ✗ killed |
M6 2 + d | every short trip had d = 2 | T5: fare(1, 4), expect 6 | 5 ✗ killed |
Demo 5: add the boundary test, 80% adds T4 and the score goes to 4/5. Demo 6: kill the last survivor, 100% adds T5 and reaches 5/5. T4 is just the test that boundary value analysis would ask for; see Boundary Value Testing for how to pick such inputs.
Tools and cost
PIT (PITest) does this for Java, Stryker for JavaScript, C# and Scala, and mutmut for Python. PIT calls the operators here Math, Conditionals Boundary, Invert Negatives and Primitive Returns. A mutant that crashes or runs too long also counts as killed.
The cost is the problem: every mutant runs the suite again. A thousand mutants and a one-minute suite would take 17 hours. Tools cut this down. They run only the tests that cover the changed line, stop at the first failing test (as the page does), and run in parallel.
See also Control-Flow Coverage for statement, branch and path coverage, the numbers a mutation score goes beyond, and Test Doubles for what a unit test asserts and how it isolates the code under test.
See also Condition Coverage and MC/DC for choosing tests so that each condition of a decision is shown to matter on its own.
What the page leaves out
Higher-order mutants (several changes at once), the full operator sets (for example Offutt's operators for Java), mutants that crash or loop and are killed by a timeout, weak versus strong mutation, test selection and incremental analysis in real tools, mutation of the tests themselves, and condition coverage or MC/DC for compound conditions are left out.