Mutation testing
Mutation testing deliberately introduces small changes — mutants — into the code under test, then runs the suite against each one. A mutant that makes a test fail is killed; one that leaves the suite green survived, meaning no test actually checks that behaviour.
It is the direct answer to the weakness of code coverage: coverage counts execution, and a test that calls a function and asserts nothing covers it completely.
What a mutant looks like
| Operator | Before | After |
|---|---|---|
| Conditional boundary | a < b |
a <= b |
| Negate conditional | a == b |
a != b |
| Arithmetic | a + b |
a - b |
| Return value | return x |
return null |
| Remove call | log(x); |
(removed) |
| Boolean literal | true |
false |
If flipping < to <= in a billing calculation leaves every test
passing, the boundary is untested — and boundaries are where bugs live.
Mutation score
mutation score = killed / (total − equivalent)
Expect it to be well below your coverage percentage. A suite at 90% line coverage scoring 55% is ordinary, and the 35-point gap is the real finding: those tests execute the code without checking it.
Equivalent mutants are changes that do not alter observable behaviour, so nothing can kill them — a mutated statement whose result is discarded, for instance. Detecting them is undecidable in general, which is why 100% is not a target and why tools report a separate "survived" versus "no coverage" breakdown.
The tools
Stryker (JavaScript, TypeScript, C#, Scala), PIT / pitest (JVM),
mutmut and Cosmic Ray (Python), mutant (Ruby), go-mutesting, and
cargo-mutants (Rust)
(Stryker reports ·
PIT reports).
Why it is slow, and how to use it anyway
The suite runs once per mutant, and a medium codebase generates thousands. Mitigations that make it practical:
- Coverage-guided selection — only run the tests that cover the mutated line.
- Incremental mode — mutate only files changed in the branch.
- Parallelism, and a nightly full run rather than per-commit.
The realistic adoption pattern is incremental on pull requests, full on a schedule (scheduled HTML reports).
The report is the deliverable
A mutation score is a number with no action attached. The HTML report is the actionable part: source browsable line by line, each surviving mutant shown in place with the exact change it made and which tests ran against it.
That is a conversation, not a metric — "this survivor is equivalent", "this one is a real gap, I'll add the boundary case". Publishing the report to a permanent URL lets reviewers put those judgements next to each survivor, and keeps them readable next run (comment on HTML).
Try it
Comma is free — unlimited reports, unlimited commenters, unlimited revision history.