Fixing Broken Robots - Android Mutation Testing

Transcript

1. Fixing broken robots - Android Mutation Testing DroidConSG 2019 1/45
2. About me Matthew Vern Twitter Github Mercari, Inc Software Engineer (Android) @panini_ja panpanini 2/45
3. Client Engineer Solving problems for our customers Shipping features Improve existing functionality My job 3/45
4. A non-shipped feature doesn't provide benefit Ship features as quick as possible My job 4/45
5. A shipped, broken feature doesn't provide benefit Ship quality features as quick as possible My job 5/45
6. Maintaining quality 6/45
7. QA Maintaining quality 6/45
8. QA Code Review Maintaining quality 6/45
9. QA Code Review Tests Maintaining quality 6/45
10. How do we know our tests are providing quality Maintaining quality 7/45
11. How do we know our tests are providing quality Use coverage to make sure that our tests are calling production code Maintaining quality 7/45
12. How do we know our tests are providing quality Use coverage to make sure that our tests are calling production code changes introduced will not break existing code Maintaining quality 7/45
13. How do we know our tests are providing quality Use coverage to make sure that our tests are calling production code changes introduced will not break existing code new code does what it says on the tin Maintaining quality 7/45
14. Who watches the watchmen? 8/45
15. How do we know that our tests are quality? Maintaining quality 9/45
16. What are tests 10/45
17. asserting that our assumptions about a piece of code are correct binary assertions of code correctness What are tests 11/45
18. Lets fail some tests 12/45
19. Unit tests assert code behaviour change code behaviour tests fail ???? profit Lets fail some tests 13/45
20. Mutation testing 14/45
21. proposed by Richard Lipton in 1971 computationally expensive, not a viable testing solution until recently Mutation testing 15/45
22. 1. Create a mutant 2. Run test suite 3. Confirm if mutant was detected or not 4. Repeat Mutation testing steps 16/45
23. A mutant is a biological entity which has undergone a change in its genetic structure. What is a mutant? 17/45
24. A mutant is a code block which has undergone a change in its structure. What is a mutant? 18/45
25. class SessionController( private val sessions: MutableList<Session> ) : EpoxyController() { fun setSessions(sessions: List<Session>) {} override fun buildModels() {} fun generateModels(sessions: List<Session>): List<SessionModel> {} } Creating mutations 19/45
26. Creating mutations fun setSessions(sessions: List<Session>) { this.sessions.clear() this.sessions.addAll(sessions) requestModelBuild() } 20/45
27. Creating mutations override fun buildModels() { generateModels(sessions) .forEach { it.addTo(this) } } 21/45
28. Creating mutations fun generateModels(sessions: List<Session>): List<SessionModel> { return sessions .map { session SessionModel_() .title(session.title) .imageUrl( if (session.speaker.profileImage "") { session.speaker.profileImage } else { null } ) } } 22/45
29. Creating mutations 23/45
30. Competent Programmer Hypothesis Creating mutations 23/45
31. Competent Programmer Hypothesis Coupling Effect Creating mutations 23/45
32. Replaces relational operators with boundary counterpart Original Original Mutated Mutated < <= <= < > >= >= > Conditionals boundary 24/45
33. Conditionals boundary original if (currentTime < startTime) { do something } mutated if (currentTime startTime) { do something } 25/45
34. Negates conditional checks Original Original Mutated Mutated == != != == <= > > <= Negate Conditionals 26/45
35. Negate Conditionals original fun buildModels() { SessionModel_() .title(session.title) .imageUrl( if (session.speaker.profileImage "") { session.speaker.profileImage } else { null } ) } mutated fun buildModels() { SessionModel_() .title(session.title) .imageUrl( if (session.speaker.profileImage "") { session.speaker.profileImage } else { null } ) } 27/45
36. removes void method calls Remove void calls 28/45
37. Remove void calls original fun setSessions(sessions: List<Session>) { this.sessions.clear() this.sessions.addAll(sessions) requestModelBuild() } mutated fun setSessions(sessions: List<Session>) { this.sessions.clear() this.sessions.addAll(sessions) } 29/45
38. So what? 30/45
39. Code coverage, but better Why mutation testing 31/45
40. 1. Introduce a fault into production code 2. Use code coverage to determine which tests to run 3. Run tests 4. Confirm if fault was detected or not 5. Repeat Mutation testing The better way 32/45
41. That's a lot of work you expect us to do there bud 33/45
42. Pitest 34/45
43. mutation testing system mutants stored in memory outputs pretty reports Gradle plugin Pitest pitest.org 35/45
44. apply plugin: pitest generates pitest<Variant> tasks Gradle plugin szpak/gradle-pitest-plugin 36/45
45. 37/45
46. written by Karol Wrótniak, forked from szpak/gradle-pitest-plugin works with Android projects has some Android specific helpers (eg: generating mockable Android jar) Android Gradle plugin koral--/gradle-pitest-plugin 38/45
47. plugins { id("pl.droidsonroids.pitest") } pitest { excludeMockableAndroidJar = false targetClasses = setOf("jp.co.panpanini.mypackage.*") outputFormats = setOf("XML", "HTML") } Android Gradle plugin 39/45
48. id("pl.droidsonroids.pitest") plugins { } pitest { excludeMockableAndroidJar = false targetClasses = setOf("jp.co.panpanini.mypackage.*") outputFormats = setOf("XML", "HTML") } Android Gradle plugin 39/45
49. pitest { excludeMockableAndroidJar = false targetClasses = setOf("jp.co.panpanini.mypackage.*") outputFormats = setOf("XML", "HTML") } plugins { id("pl.droidsonroids.pitest") } Android Gradle plugin 39/45
50. excludeMockableAndroidJar = false plugins { id("pl.droidsonroids.pitest") } pitest { targetClasses = setOf("jp.co.panpanini.mypackage.*") outputFormats = setOf("XML", "HTML") } Android Gradle plugin 39/45
51. targetClasses = setOf("jp.co.panpanini.mypackage.*") plugins { id("pl.droidsonroids.pitest") } pitest { excludeMockableAndroidJar = false outputFormats = setOf("XML", "HTML") } Android Gradle plugin 39/45
52. outputFormats = setOf("XML", "HTML") plugins { id("pl.droidsonroids.pitest") } pitest { excludeMockableAndroidJar = false targetClasses = setOf("jp.co.panpanini.mypackage.*") } Android Gradle plugin 39/45
53. Demo 40/45
54. Pitest tips & tricks 41/45
55. MutationInterceptor Removes mutants for Kotlin generated code Pitest kotlin pitest/pitest-kotlin 42/45
56. Run PITest on Unit tests only 43/45
57. Run PITest on CI 44/45
58. Mutation Testing panpanini/mutation_testing Github: panpanini Twitter: panini_ja 45/45