Making your tests fail - Introduction to Mutation Testing

MUTATION TESTING WITHSTANDING THE EPIDEMIC @felix_codes felixwu.dev

http://bit.ly/flxwu-mutation-testing @felix_codes felixwu.dev

@felix_codes

@felix_codes WHY TESTS?

@felix_codes WHY TESTS? “Quality is never an accident; it is
always the result of intelligent effort.” John Ruskin

@felix_codes FUNCTIONALITY WHY TESTS?

@felix_codes CODE HOTSPOTS WHY TESTS?

@felix_codes STAYING AT THE TOP WHY TESTS?

@felix_codes WHY TESTS? Engineers are responsible for finding errors in
your product, not your user.

@felix_codes WHY TESTS? Designers are responsible for finding errors in

@felix_codes WHY TESTS? PMs are responsible for finding errors in

@felix_codes WHY TESTS? You are responsible for finding errors in

@felix_codes WRITING GOOD TESTS WHY TESTS?

@felix_codes TEST QUALITY WRITING GOOD TESTS WHY TESTS?

@felix_codes MEASURING TEST QUALITY WHY TESTS?

@felix_codes MEASURING TEST QUALITY 63% 1. WHY TESTS?

@felix_codes MEASURING TEST QUALITY 63% 1. 2. WHY TESTS?

@felix_codes MEASURING TEST QUALITY 63% 1. 2. Integration Unit End2End
3. WHY TESTS?

@felix_codes CODE COVERAGE WHY TESTS?

@felix_codes CODE COVERAGE 72% WHY TESTS?

@felix_codes COVERAGE REPORTS 72% CODE COVERAGE

@felix_codes CODE COVERAGE TOOLS Corbetura JCov JaCoCo Istanbul Blanket.js Jest
CODE COVERAGE

@felix_codes COVERAGE vs. MAINTAINABILITY Maintainability Code coverage IDEAL CODE COVERAGE

@felix_codes Maintainability Code coverage DANGER ZONE CODE COVERAGE COVERAGE vs.
MAINTAINABILITY

@felix_codes Maintainability Code coverage REALITY CODE COVERAGE COVERAGE vs. MAINTAINABILITY

@felix_codes Quality Quantity DANGER ZONE CODE COVERAGE QUANTITY vs. QUALITY

@felix_codes Quality Quantity CODE COVERAGE QUANTITY vs. QUALITY 85%

@felix_codes Quality Quantity CODE COVERAGE QUANTITY vs. QUALITY 85% 100%

@felix_codes Quality Quantity CODE COVERAGE QUANTITY vs. QUALITY 85% 100%
you get what you measure!

@felix_codes CODE COVERAGE Code coverage is not the best indicator
for test strength and thoroughness

@felix_codes CODE COVERAGE Code coverage is not the best indicator
for test strength and thoroughness ?

@felix_codes CODE COVERAGE We are in need of an effective
measurement of our tests quality

@felix_codes MUTATION TESTING

@felix_codes WHAT WHY HOW

@felix_codes WHAT “Making your test cases fail”

@felix_codes WHAT IS MUTATION TESTING

@felix_codes WHAT IS MUTATION TESTING STEPS OF MUTATION TESTING 1.

@felix_codes WHAT IS MUTATION TESTING STEPS OF MUTATION TESTING

@felix_codes WHAT IS MUTATION TESTING We can assess the quality
of the test cases which should be robust enough to fail mutant code.

@felix_codes WHAT IS MUTATION TESTING MUTATION EXAMPLE

@felix_codes MUTATION TESTING EXAMPLE GVODUJPODBO7JFX%PDVNFOU VTFS \ SFUVSOVTFSQFSNJTTJPO)JFSBSDIZ ^

@felix_codes QVCMJDCPPMFBODBO7JFX%PDVNFOU 6TFSVTFS \ SFUVSOVTFSQFSNJTTJPO)JFSBSDIZ ^ MUTATION TESTING EXAMPLE

@felix_codes QVCMJDCPPMFBODBO7JFX%PDVNFOU 6TFSVTFS \ SFUVSOVTFSQFSNJTTJPO)JFSBSDIZ ^ !5FTU QVCMJDWPJEUFTU7JFX%PDVNFOU \ BTTFSU&RVBMT
GBMTF DBO7JFX%PDVNFOU OFX6TFS ^ MUTATION TESTING EXAMPLE

@felix_codes QVCMJDCPPMFBODBO7JFX%PDVNFOU 6TFSVTFS \ SFUVSOVTFSQFSNJTTJPO)JFSBSDIZ ^ 2 3 1 4
SFUVSOUSVF SFUVSOGBMTF SFUVSOVTFSQFSNJTTJPO)JFSBSDIZ SFUVSOVTFSQFSNJTTJPO)JFSBSDIZ MUTATION TESTING EXAMPLE

@felix_codes QVCMJDCPPMFBODBO7JFX%PDVNFOU 6TFSVTFS \ ^ 1 SFUVSOUSVF !5FTU QVCMJDWPJEUFTU7JFX%PDVNFOU \
BTTFSU&RVBMT GBMTF DBO7JFX%PDVNFOU OFX6TFS ^ MUTATION TESTING EXAMPLE

@felix_codes QVCMJDCPPMFBODBO7JFX%PDVNFOU 6TFSVTFS \ ^ 2 SFUVSOGBMTF !5FTU QVCMJDWPJEUFTU7JFX%PDVNFOU \

@felix_codes QVCMJDCPPMFBODBO7JFX%PDVNFOU 6TFSVTFS \ ^ 3 SFUVSOVTFSQFSNJTTJPO)JFSBSDIZ !5FTU QVCMJDWPJEUFTU7JFX%PDVNFOU \

@felix_codes QVCMJDCPPMFBODBO7JFX%PDVNFOU 6TFSVTFS \ ^ 4 SFUVSOVTFSQFSNJTTJPO)JFSBSDIZ !5FTU QVCMJDWPJEUFTU7JFX%PDVNFOU \

@felix_codes 2 3 SFUVSOUSVF SFUVSOGBMTF SFUVSOVTFSQFSNJTTJPO)JFSBSDIZ SFUVSOVTFSQFSNJTTJPO)JFSBSDIZ !5FTU QVCMJDWPJEUFTU7JFX%PDVNFOU \

@felix_codes 2 3 SFUVSOGBMTF SFUVSO VTFSQFSNJTTJPO)JFSBSDIZ !5FTU QVCMJDWPJEUFTU7JFX%PDVNFOU \ BTTFSU&RVBMT
GBMTF DBO7JFX%PDVNFOU OFX6TFS ^ QVCMJDCPPMFBODBO7JFX%PDVNFOU 6TFSVTFS \ SFUVSOVTFSQFSNJTTJPO)JFSBSDIZ ^ MUTATION TESTING EXAMPLE

GBMTF DBO7JFX%PDVNFOU OFX6TFS ^ QVCMJDCPPMFBODBO7JFX%PDVNFOU 6TFSVTFS \ SFUVSOVTFSQFSNJTTJPO)JFSBSDIZ ^ BTTFSU&RVBMT USVF DBO7JFX%PDVNFOU OFX6TFS MUTATION TESTING EXAMPLE

GBMTF DBO7JFX%PDVNFOU OFX6TFS ^ QVCMJDCPPMFBODBO7JFX%PDVNFOU 6TFSVTFS \ SFUVSOVTFSQFSNJTTJPO)JFSBSDIZ ^ BTTFSU&RVBMT USVF DBO7JFX%PDVNFOU OFX6TFS BTTFSU&RVBMT USVF DBO7JFX%PDVNFOU OFX6TFS MUTATION TESTING EXAMPLE

@felix_codes 2 3 SFUVSOGBMTF SFUVSO VTFSQFSNJTTJPO)JFSBSDIZ QVCMJDCPPMFBODBO7JFX%PDVNFOU 6TFSVTFS \ SFUVSOVTFSQFSNJTTJPO)JFSBSDIZ
^ !5FTU QVCMJDWPJEUFTU7JFX%PDVNFOU \ BTTFSU&RVBMT GBMTF DBO7JFX%PDVNFOU OFX6TFS ^ BTTFSU&RVBMT USVF DBO7JFX%PDVNFOU OFX6TFS BTTFSU&RVBMT USVF DBO7JFX%PDVNFOU OFX6TFS MUTATION TESTING EXAMPLE

@felix_codes !5FTU QVCMJDWPJEUFTU7JFX%PDVNFOU \ BTTFSU&RVBMT GBMTF DBO7JFX%PDVNFOU OFX6TFS
^ BTTFSU&RVBMT USVF DBO7JFX%PDVNFOU OFX6TFS BTTFSU&RVBMT USVF DBO7JFX%PDVNFOU OFX6TFS MUTATION TESTING EXAMPLE

@felix_codes MUTANTS “Humans are mutants, everything's a mutant - things
that evolve.” Oscar Isaac

@felix_codes

@felix_codes Mutants simulate human programming errors

@felix_codes MUTANTS TYPES OF MUTANTS

@felix_codes MUTANTS TYPES OF MUTANTS Value mutation

@felix_codes MUTANTS TYPES OF MUTANTS Value mutation Operator mutation

@felix_codes MUTANTS TYPES OF MUTANTS Value mutation Operator mutation Statement
mutation

mutation int a = 12 int a = 15

mutation a++; a++; a++; int a = 12 int a = 15

mutation

@felix_codes MUTANTS TYPES OF MUTANTS Arithmetic Relational Conditional a +
b a - b while(a < b) while(true) if(a || b) if (a && b) Operator mutation

@felix_codes MUTANTS

@felix_codes Mutants simulate human programming errors MUTANTS

@felix_codes Mutants simulate errors of competent programmers “ ” MUTANTS

@felix_codes Mutants simulate errors of competent programmers if (age >
18) // check if adult “ ” MUTANTS

@felix_codes Mutants simulate minor errors of competent programmers. Major flaws
always arise from minor errors because of the coupling effect. “ ” MUTANTS

@felix_codes MUTANTS

@felix_codes Mutation Testing score number of mutants killed total mutants
amount 0% MUTANTS

@felix_codes HOW

@felix_codes WHY

@felix_codes WHY MUTATION TESTING? PITFALLS

@felix_codes WHY MUTATION TESTING? PITFALLS “Equivalent mutants” =

@felix_codes WHY MUTATION TESTING? PITFALLS JOUNBY JOU<>WBMVFT \ JOUS J
S GPS JJWBMVFTMFOHUIJ \ JG WBMVFT<J>WBMVFT<S> SJ ^ SFUVSOWBMVFT<S> ^

@felix_codes WHY MUTATION TESTING? PITFALLS JOUNBY JOU<>WBMVFT \ JOUS J
S GPS JJWBMVFTMFOHUIJ \ JG WBMVFT<J>WBMVFT<S> SJ ^ SFUVSOWBMVFT<S> ^ JOUNBY JOU<>WBMVFT \ JOUS J S GPS JJWBMVFTMFOHUIJ \ JG WBMVFT<J>WBMVFT<S> SJ ^ SFUVSOWBMVFT<S> ^

@felix_codes WHY MUTATION TESTING? PITFALLS

@felix_codes WHY MUTATION TESTING? PITFALLS “Equivalent mutants” =

@felix_codes WHY MUTATION TESTING? PEOPLE ARE REALIZING

@felix_codes WHY MUTATION TESTING? BENEFITS Maintainability Code coverage DANGER ZONE

@felix_codes WHY MUTATION TESTING? BENEFITS Maintainability Code coverage IDEAL

@felix_codes WHY MUTATION TESTING? CONTINIOUS INTEGRATION

@felix_codes WHAT IS MUTATION TESTING Aaltonen, Kalle & Ihantola, Petri
& Seppälä, Otto. (2015); Mutation Testing vs. Code Coverage in automated assessment

WITHSTANDING THE EPIDEMIC Felix Wu - @felix_codes MUTATION TESTING

Felix Wu - @felix_codes PITest(Java) Javalanche(Java) Stryker(JS) http://bit.ly/flxwu-mutation-testing

Making your tests fail - Introduction to Mutation Testing

Making your tests fail - Introduction to Mutation Testing

More Decks by Felix Wu

Other Decks in Programming

Featured

Transcript