Who will test your tests?

Presentation for VodQA, Pune, November 2014

Transcript

1. WHO WILL TEST YOUR TESTS? Shirish Padalkar VodQA, Pune. November 2014 1
2. ABOUT ME 2
3. ABOUT ME 3
4. ABOUT ME 4
5. WE NEED TO TALK! 5
6. SOME QUESTIONS FIRST 6 https://flic.kr/p/2vgyWN
7. DOES YOUR PROJECT HAVE AN AUTOMATION TEST SUITE? 7
8. HOW OFTEN YOU NEED TO CHANGE / REFACTOR YOUR TEST SUITE / FRAMEWORK? 8
9. DO YOU FEEL COMFORTABLE UPDATING YOUR TESTS? 9
10. HOW IS THE CODE QUALITY OF YOUR TEST SUITE? 10
11. DO YOU TRUST YOUR TESTS AND TEST SUITE? 11
12. LET’S TALK! 12
13. WHY DO WE WRITE TESTS? 13 https://flic.kr/p/9uZhDS
14. WHY WE WRITE TESTS? 14 TESTS PROVE THAT THE CODE ACTUALLY WORKS
15. WHY WE WRITE TESTS? 15 TESTS REDUCE BUGS
16. WHY WE WRITE TESTS? 16 TESTS REDUCE THE COST OF CHANGE
17. WHY WE WRITE TESTS? 17 TESTS ALLOW REFACTORING
18. WHY WE WRITE TESTS? 18 TESTING HELPS IN FASTER DEVELOPMENT
19. WHY WE WRITE TESTS? 19 TESTS REDUCE FEAR
21. 21 NOW WHEN YOU HEAR…
22. Oh, it failed? Don’t worry. Just re-trigger it! ;) 22
23. Oh, it failed? Strange; it passed on my machine! 23
24. Oh, it failed? I know. It fails at 12PM! 24
25. 25 http://upload.wikimedia.org/wikipedia/commons/3/3b/Paris_Tuileries_Garden_Facepalm_statue.jpg
26. WHY DO WE WRITE TESTS? 26 Now, again,
27. WHY WE WRITE TESTS? 27 Tests prove that the code actually works Tests prove that the code sometimes works
28. WHY WE WRITE TESTS? 28 Tests reduce bugs Tests itself might have bugs
29. WHY WE WRITE TESTS? 29 Tests reduce the Cost of Change Change? Are you kidding? I will need a week to change my framework and make sure everything works!
30. WHY WE WRITE TESTS? 30 Tests allow refactoring
31. WHY WE WRITE TESTS? 31 Testing helps in faster development. Testing is slowing me down :( I am not sure when to stop writing code. Tests keep randomly failing and succeeding.
32. WHY WE WRITE TESTS? 32 Tests reduce fear Don’t touch that code and tests. You will break them!
33. WHY? 33
34. NON-DETERMINISTIC TESTS 34 A.K.A. “Flaky tests”
35. WHY FLAKY TESTS ARE PROBLEM 35 THEY ARE HARMFUL TESTS!
36. WHY FLAKY TESTS ARE PROBLEM 36 THEY BREAK YOUR TRUST! A red build doesn’t mean anything anymore!
37. WHY FLAKY TESTS ARE PROBLEM 37 THEY DESTROY THE VALUE OF AUTOMATED REGRESSION
38. WHY FLAKY TESTS ARE PROBLEM 38 THEY GIVE YOU ILLUSION OF SUCCESS!
39. WHY FLAKY TESTS ARE PROBLEM 39 THEY GIVE YOU ILLUSION OF FAILURE!
40. WHY FLAKY TESTS ARE PROBLEM 40 THEY PROMOTE BAD CULTURE! Culture of living in failures!
41. CAUSES OF FLAKINESS 41
42. CAUSES OF FLAKINESS 42 LACK OF ISOLATION
43. CAUSES OF FLAKINESS 43 SHARED STATE IN APPLICATION
44. CAUSES OF FLAKINESS 44 ASYNCHRONOUS BEHAVIOR
45. CAUSES OF FLAKINESS 45 EXTERNAL SYSTEM CALLS
46. CAUSES OF FLAKINESS 46 TIME
47. HOW TO FIX FLAKINESS 47
51. HOW TO FIX FLAKY TESTS 51 DO NOT THINK BUILD RANDOMLY FAILED.
52. HOW TO FIX FLAKY TESTS 52 THINK BUILD RANDOMLY PASSED!
53. HOW TO FIX FLAKY TESTS 53 QUARANTINE
54. HOW TO FIX FLAKY TESTS 54 KEEP ONLY N TESTS IN QUARANTINE Let’s say, 5 tests at any given point
55. HOW TO FIX FLAKY TESTS 55 ALLOCATE TIME TO FIX QUARANTINED TESTS
56. HOW TO FIX FLAKY TESTS 56 UNDERSTAND NATURE OF FLAKINESS Most flakiness is due to our own mistakes!
57. HOW TO FIX FLAKY TESTS 57 YOU KNOW THERE MIGHT BE BUGS IN PRODUCTION CODE
58. HOW TO FIX FLAKY TESTS 58 YOU KNOW THERE MIGHT BE BUGS IN PRODUCTION CODE TEST
59. HOW TO FIX FLAKY TESTS 59 REFACTOR YOUR TESTS!
60. HOW TO FIX FLAKY TESTS 60 RANDOM WAITS DUPLICATE CODE GOD OBJECTS COMPLICATED CODE
61. REMEMBER! 61
62. 62 A BUG IS A TEST YOU FORGOT TO WRITE
63. 63 TESTER IS A ROLE NOT A PERSON
64. 64 TESTER IS NOT A FAILED DEVELOPER
65. 65 LOVE YOUR TEST CODE. Treat your test code as production code!
66. 66 DO NOT TOLERATE BAD TEST CODE
67. 67 TALK TO THE TEAM IF WRITING THE TEST IS HARD There might a code smell lying somewhere
68. TESTING IS FUN. DON’T MAKE PEOPLE HATE IT! 68
69. ANY QUESTIONS? 69
70. RESOURCES ▫︎Eradicating Non-Determinism in Tests - by Martin Fowler ▫︎No more flaky tests on the Go team - by Pavan Sudarshan ▫︎Top 12 Reasons to Write Unit Tests - by Eric M. Burke and Brian M. Coyner ▫︎Flaky crusts: Good for pies; bad for test suites ▫︎Creating Maintainable Automated Acceptance Tests - By Jez humble and Badri 70
71. ThoughtWorks is hiring. http://www.thoughtworks.com/join THANK YOU!