Awesome Mutation testing

Mutation testing is the practice of making better code by introducing bugs. As of now this repository is used to keep track of the current mutation testing tools developed and related resources. Its contents and purpose may however evolve depending on the community.

Discord Invite

Contents

• Existing libraries
• Abandoned libraries
• Visualization Tool
• Papers
• Blogs/Posts/Videos
• Conferences
• Friendly resources
• Tutorials

Existing libraries

• Alloy
  • MuAlloy
    • MuAlloy papers
• C/C++
  • llvm-mutate
  • Frama-C plugin
  • mull-project/mull
  • mutate_cpp
  • MuCPP
    • Assessment of Class Mutation Operators for C++ with the MuCPP Mutation System
  • accmut
    • Faster mutation analysis via equivalence modulo states
  • MUSIC
    • MUSIC: Mutation Analysis Tool with High Configurability and Extensibility
  • Dextool Mutate
  • SRCIROR - A mutation tool for source and IR
    • SRCIROR: A Toolset for Mutation Testing of C Source Code and LLVM Intermediate Representation
  • MART
• C#
  • stryker-mutator/stryker-net
  • ComparetheMarket/fettle
  • Testura.Mutation
  • Faultify
• Clojure
  • mutant
• Crystal
  • crytic
• Elixir
  • JordiPolo/mutation
• Erlang
  • parsifal-47/muterl
• Go
  • zimmski/go-mutesting
  • Ooze
  • go-gremlins
• Haskell
  • mucheck
  • rudymatela/fitspec
• Java/JVM
  • hcoles/pitest * PIT : A Practical Mutation Testing Tool for Java (Demo) * An Experimental Evaluation of PIT’s Mutation Operators
    • Introduction to Mutation Testing
    • Faster Mutation Testing (in Java)
      • Voxxed Athens 2017 :: Mutation Testing to the rescue of your Tests • Nicolas Fränkel
  • aliparsai/LittleDarwin
    • LittleDarwin: a Feature-Rich and Extensible Mutation Testing Framework for Large and Complex Java Systems
  • metamutator
  • Major
    • Publications related to the Major mutation framework
• JavaScript
  • stryker-mutator/stryker
• Kotlin
  • mutant-kraken
• PHP
  • infection/infection
  • PEST PHP
• Python
  • sixty-north/cosmic-ray
  • boxed/mutmut
• Ruby
  • mbj/mutant
    • Kill all the mutants - a deep dive into mutation testing and how the Mutant gem works
  • backus/mutest
• Rust
  • sourcefrog/cargo-mutants
  • llogiq/mutagen
• Scala
  • sugakandrey/scalamu
  • stryker4s
• Smalltalk
  • pavel-krivanek/mutalk
• Simulink
  • MUT4SLX - An open-source, model-based mutation testing tool for Simulink and Stateflow (MATLAB) models. Applies principles of shift-left testing.
    • Reproduction Package
    • MUT4SLX: Fast mutant generation for simulink
    • MUT4SLX: Extensions for Mutation Testing of Stateflow Models
• Swift
  • muter-mutation-testing/muter
• OCaml
  • mutaml
• Other
  • agroce/universalmutator
  • squaresLab/boggart
  • codeintegrity-ai/mutahunter
  • JoranHonig/vertigo

Visualization Tool

• sqrlab/VisMAn

Publications

• Jeongju Sohn, Ezekiel Soremekun, Michail Papadakis (2025) Latent Mutants: A large-scale study on the Interplay between mutation testing and software evolution
• Dolores Miao, Ignacio Laguna, Giorgis Georgakoudis, Konstantinos Parasyris, Cindy Rubio-González (2024) An automated OpenMP mutation testing framework for performance optimization
• Pedro Delgado-Pérez, Ana B. Sánchez, Sergio Segura, Inmaculada Medina-Bulo (2022) Mutation testing in the wild: findings from GitHub
• Goran Petrovic, Marko Ivankovic, Gordon Fraser, René Just (2021) Practical Mutation Testing at Scale: A view from Google
• Goran Petrovic, Marko Ivankovic, Gordon Fraser, René Just (2021) Does mutation testing improve testing practices?
• Giovani Guizzo, Federica Sarro, Jens Krinke, Silvia Regina Vergilio (2021) Sentinel: A Hyper-Heuristic for the Generation of Mutant Reduction Strategies
• Y. Ivanovaa, A. Khritankova (2020) RegularMutator: A Mutation Testing Tool for Solidity Smart Contracts
• Alessandro Viola Pizzoletoa, Fabiano Cutigi Ferraria, Jeff Offuttb, Leo Fernandesc, Marcio Ribeirod (2019) A Systematic Literature Review of Techniques and Metrics to Reduce the Cost of Mutation Testing
• August Shi, Jonathan Bell, Darko Marinov (2019) Mitigating the Effects of Flaky Tests on Mutation Testing
• Esther Guerra, Jesús Sánchez Cuadrado, Juan de Lara (2019) Towards effective mutation testing for ATL
• Lin Deng, Jeff Offutt (2018) Reducing the Cost of Android Mutation Testing
• Goran Petrovic, Marko Ivankovic (2018) State of Mutation Testing at Google
• Goran Petrovic, Marko Ivankovic, Bob Kurtz, Paul Ammann, René Just (2018) An Industrial Application of Mutation Testing/ Lessons, Challenges, and Research Directions
• Sten Vercammen, Serge Demeyer, Mohammad Ghafari, Markus Borg (2018) Goal-Oriented Mutation Testing with Focal Methods
• Qianqian Zhu, Annibale Panichella, Andy Zaidman (2018) A Systematic Literature Review of How Mutation Testing Supports Quality Assurance Processes
• Lingchao Chen, Lingming Zhang (2018) Speeding up Mutation Testing via Regression Test Selection: An Extensive Study
• Mike Papadakisa, Marinos Kintisa, Jie Zhangc, Yue Jiab, Yves Le Traona, Mark Harmanb (2017) Mutation Testing Advances: An Analysis and Survey
• Pablo C. Cañizares, Alberto Núñez, Juan de Lara (2017) OUTRIDER: Optimizing the mUtation Testing pRocess In Distributed EnviRonments
• Qianqian Zhu, Annibale Panichella, Andy Zaidman (2017) Speeding-Up Mutation Testing via Data Compression and State Infection
• Jie Zhang, Ziyi Wang, Lingming Zhang, Dan Hao, Lei Zang, Shiyang Cheng, Lu Zhang (2016) Predictive Mutation Testing
• Marinos Kintis (2016) Effective Methods to Tackle the Equivalent Mutant Problem when Testing Software with Mutation
• René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, Gordon Fraser (2014) Are Mutants a Valid Substitute for Real Faults in Software Testing?
• Lorena Gutiérrez-Madroñal, J. Domínguez-Jiménez, I. Medina-Bulo (2014) Mutation Testing: Guideline and Mutation Operator Classification
• L. Madeyski, W. Orzeszyna, R. Torkar, M. Józala (2014) Overcoming the Equivalent Mutant Problem/ A Systematic Literature Review and a Comparative Experiment of Second Order Mutation
• Lingming Zhang, Darko Marinov, Sarfraz Khurshid (2013) Faster Mutation Testing Inspired by Test Prioritization and Reduction
• Bouchaib Falah, Bouriat Salwa (2013) Effectiveness of Mutation Testing Techniques: Reducing Mutation Cost
• Quang Vu Nguyen, Lech Madeyski (2013) Problems of Mutation Testing and Higher Order Mutation Testing
• Lingming Zhang, Milos Gligoric, Darko Marinov, Sarfraz Khurshid (2013) Operator-Based and Random Mutant Selection: Better Together
• Kevin Jalbert, Jeremy S. Bradbury (2012) Predicting Mutation Scores using Source Code and Test Suite Metrics
• David Schuler (2011) Assessing Test Quality
• John A. Clarka, Haitao Danb, Robert M. Hieronsb (2011) Semantic Mutation Testing
• Yue Jia, Mark Harman (2010) An Analysis and Survey of the Development of Mutation Testing
• Gordon Fraser (2010) Mutation Testing (university course slides)
• Pedro Delgado-Pérez, Ana B. Sánchez, Sergio Segura, Inmaculada Medina-Bulo (2010) Performance Mutation Testing
• Macario Polo, Mario Piattini (2009) Mutation testing: practical aspects and cost analysis
• Jeff Offutt, Paul Ammann, Lisa (Ling) Liu (2006) Mutation Testing implements Grammar-Based Testing
• Robert Geist, Jeff Offutt (1992) Estimation and Enhancement of Real-Time Software Reliability through Mutation Analysis

Blogs/Posts

• The Power of Mutation Testing (2024) • The Green Report
• Enhancing Test Effectiveness with Mutation Testing (2024) • João Coelho
• Mutation Testing Google Blog Entry (2021) • Goran Petrovic (Google)
• GOTO 2019 • Making Mutants Work for You • Henry Coles
• Test Automation Research for Industry 2019: Mutation Testing, Opportunities and Pitfalls • Ali Parsai
• PHPDeveloperDay 2018: Mutation Testing • Théo Fidry
• RailsConf 2017: How to Write Better Code Using Mutation Testing • John Backus
• FOSDEM 2017: Mutation Testing: Leaving the Stone Age • Alex Denisov
• A note on Mutation Operators (2017) • Markus Schirp
• Jfokus 2016: From jUnit to Mutation-Testing • Sven Ruppert
• GOTO 2015: Mutation Testing in Python • Austin Bingham

Videos

• Rider Webinar 2023: How To Test C# Unit Tests With Mutation Testing • Stefan Pölz

Conferences

• International Workshop on Mutation Analysis
  • MUTATION 2018, The 13th International Workshop on Mutation Analysis

Friendly resources

• atodorov/mutation-testing-in-patterns
  • Practical examples of software and testing patterns related to mutation testing Mutation Testing in Patterns

Tutorials

• Introduction to Mutation Testing with PIT (Java)