Re-thinking Regression Testing

Talk I gave at Mountain West Ruby Conf 2014 about mutation testing

Transcript

1. Re-thinking regression testing Mario González Macedo @maramono ortask.com
2. http://www.softwaretestpro.com/Item/5020/Minor-Risk-Patches/Cartoon
3. regression code coverage test redundancy ortask.com
4. ortask.com
5. regression code coverage test redundancy ortask.com
6. mutation testing
7. …vs code coverage
8. ortask.com
9. ortask.com
10. ortask.com
11. ortask.com
12. http://www.softwaretestpro.com/Item/5020/Minor-Risk-Patches/Cartoon
13. ortask.com
14. ortask.com
15. mutation score
16. (0, 1) worst mutation-adequate
17. ok… but what?
18. a type of fault injection
19. no extra code
20. change the code
21. mutate existing source code changes to tests: none
24. score… how?
25. mutants total killed mutants score =
26. zombies total killed zombies score =
27. ninjas total killed ninjas score =
28. how many mutations?
29. one mutation?
30. two mutations?
31. 100 mutations?
32. k = mutation qty
33. assumption
34. assumption (a valid hypothesis)
35. competent programmer hypothesis
36. competent programmer hypothesis Some software bugs are caused by syntactically small errors
37. competent programmer hypothesis k = mutation qty
38. competent programmer hypothesis k = 1
39. k
40. if k == 1 then first-order
41. if k == 2 then second-order
42. k-order mutant k mutations
43. k-order for simplicity easy to communicate
44. if k == 1 then low-order or simply first-order
45. if k > 1 then high-order
46. high-order… but why?
47. competent programmer hypothesis
48. competent programmer hypothesis coupling effect
49. coupling effect Other software bugs are caused by interactions of small errors
50. coupling effect 1 < k < 6
51. why it works
52. for regression analysis one developer on fixes and features competent programmer hypothesis
53. for regression analysis many contributors on fixes and features competent programmer hypothesis coupling effect
54. so… why better?
55. test redundancy mutation score correlation: code coverage & test redundancy ortask.com
56. ortask.com
57. mutation score test redundancy mutation score correlation: code coverage & ortask.com
58. code coverage correlation: likelihood of bugs & test redundancy ortask.com
59. http://www.sustainablebabysteps.com/inflating-tires.html
60. http://willyoubemyyokoono.blogspot.com/2012_09_01_archive.html
61. correlation: mutation score & test redundancy code coverage ortask.com
62. how to apply it in practice
63. heckle mutant muJava Jester NinjaTurtles
64. heckle: obsolete as of Ruby 1.9 mutant: did not use (rc) muJava Jester NinjaTurtles
65. https://github.com/ortask/mutator research-strength fast cross-language
66. painless mutation analysis a few guidelines
67. guideline 1: passing tests http://vector-magz.com/logo/quality-control-logo-item-4/
68. guideline 2: mutants 50 low-order experiment with high-order
69. guideline 3: cycle mutate score improve test
70. guideline 4: coupling low coupling high coupling detection
71. Thank you! Mario González Macedo @maramono ortask.com
72. Other mutation testing sources http://slid.es/markusschirp/mutation-testing https://github.com/mbj/mutant/tree/master/l ib/mutant/mutator/node https://github.com/seattlerb/heckle http://vimeo.com/10863662
73. Other sources http://vimeo.com/1683910