Mutation Testing
Mutation testing injects small faults into code and checks whether tests catch them, exposing weak assertions that coverage metrics hide. The mutation score is a strong measure of true test effectiveness, valuable when locking down behavior before migration.
Best Practice: Mutation Testing
Mutation testing measures the quality of a test suite, not just the quantity of code it executes. A mutation tool makes many tiny changes to the source, such as flipping a comparison operator or removing a statement, producing "mutants." It then runs the tests against each mutant. If the tests fail, the mutant is "killed"; if they pass, the mutant "survives," revealing a gap where tests execute code but do not actually assert its behavior. The mutation score (percentage of mutants killed) is a far stronger signal of test effectiveness than line coverage, which can be high while assertions are weak.
Step-by-Step Implementation Guidance
- Choose a mutation tool that fits your language and build.
- Run it on a focused module first, since mutation testing is computationally heavy.
- Review surviving mutants and decide whether each reveals a missing assertion or an equivalent (harmless) mutant.
- Strengthen tests to kill meaningful survivors.
- Configure incremental analysis so only changed code is mutated in CI.
- Set a mutation score threshold for critical modules.
- Track the score over time and revisit equivalent mutants periodically.
Common Mistakes Teams Make When Ignoring This Practice
- Trusting high line coverage while assertions are too weak to catch real bugs.
- Running full mutation analysis on the whole codebase every build, making CI slow.
- Treating every surviving mutant as a defect, ignoring equivalent mutants.
- Setting an unrealistic 100 percent score target.
- Failing to focus the practice on the highest-risk modules.
Tools and Techniques That Support This Practice
- PIT (Pitest) for the JVM.
- Stryker for JavaScript, TypeScript, C#, and Scala.
- mutmut and Cosmic Ray for Python.
- Incremental/changed-files mutation modes for CI.
- Reporting dashboards to track mutation score trends.
How This Practice Applies to Different Migration Types
- Cloud Migration: Verify that tests guarding portable logic actually assert behavior before relocating it.
- Database Migration: Use mutation testing on data-mapping code to ensure tests catch subtle conversion errors.
- SaaS Migration: Confirm adapter tests would catch behavioral differences from the new provider.
- Codebase Migration: Strengthen the safety net before a rewrite so regressions cannot slip past weak tests.
Checklist
- A mutation tool is configured for the language and build.
- Analysis starts on focused, high-risk modules.
- Surviving mutants are reviewed for missing assertions.
- Equivalent mutants are identified and excluded.
- Tests are strengthened to kill meaningful survivors.
- Incremental mutation runs on changed code in CI.
- Mutation score trends are tracked over time.