Who will Test your tests!

This is the talk given at first XP conf, Bangalore (http://xpconference.in/) by me and Yahya Poonawala (https://www.linkedin.com/in/yahyapoonawala).

We all know advantages of writing tests, but when it becomes unmaintainable and we get into a situation, where we start hating our test suite, this is talk for those people who are plagued with disease of flakiness!

Transcript

1. WHO WILL TEST YOUR TESTS ? Yahya Poonawala @meetykp Priti Biyani @pritibiyani
2. 2 https://flic.kr/p/9uZhDS 2
3. TESTS Code works Reduce Bugs Reduce cost of change Allow Refactoring Faster development Reduce fear 3
4. NOW WHEN YOU HEAR …. 4
5. Oh, it failed? Don’t worry. Just re-trigger it! 5
6. Oh, it failed? Strange; it passed on my machine! 6
7. Oh, it failed? Run it on agent 007! It’s an agent issue! 7
8. Oh, it failed? I know. It fails at 12AM! 8
10. Now, again, WHY DO WE WRITE TESTS? 1 0
11. Tests prove that the code actually works Tests prove that the code sometimes works 1 1
12. Tests reduce bugs Tests itself might have bugs 1 2
13. Tests reduce the cost of change Change? Are you kidding? 1 3
14. Tests helps in faster development. Testing is slowing me down L 1 4
15. NON-DETERMINISTIC TESTS A.K.A “Flaky Tests” 1 5
16. 1 6 http://www.trovatten.com/long-game/
17. 1 7
18. 1 8 http://tom-doherty.com/wp-content/uploads/2014/02/craigslist-waste-time.jpg
19. 1 9
20. ILLUSION OF SUCCESS! 2 0
21. ILLUSION OF FAILURE! 2 1
22. 2 2
23. 2 3
24. DO NOT DELETE YOUR FLAKY TESTS!! 2 4
25. CAUSES OF FLAKINESS 2 5
26. Thread 1::! if( name is not Yahya ) {! …! fputs( name, file )! …! }! ! Thread 2::! Initialize name to Yahya! !
27. CONCURRENCY 2 7
28. Initialize lock;! ! Thread 1::! Acquire lock! if( name is not Yahya) {! . . . ! fputs( name, file )! . . .! }! Release lock! ! Thread 2::! Acquire lock! Initialize name to Yahya! Release lock! ! !
29. TIME 2 9 http://imgs.xkcd.com/comics/bug.png
30. EXTERNAL SYSTEM CALLS 3 0
31. UNORDERED COLLECTIONS 3 1
32. 3 2 https://www.vardot.com/sites/default/files/2.jpg
33. # pseudo-code
 makeAsyncCall;
 sleep(2000);
 readResponse;

34. ASYNCHRONOUS WAITING 3 4
35. ! ! make asynchronous call;! ! While (response received?)! if timeout then ! throw Test Timeout Exception ! ! else! sleep(10)! . . . ! ! read response! ! !
36. RESOURCE LEAKS 3 6
37. # //pseudo-code! FILE_NAME = “content.txt”! ! // Test 1:! ! FileReader.read(FILE_NAME).to be empty! ! // Test2: Write to the file! ! data = “Priti”! FileWriter.write(FILE_NAME, data)! FileReader.read(FILE_NAME) == data! ! // Test 3: Updates the file! ! append_data = “Biyani”! complete_data = “Priti Biyani”! FileWrite.append(FILE_NAME, append_data)! FileReader.read(FILE_NAME) == complete_data! !
38. TEST ORDER DEPENDENCY 3 8
39. INPUT / OUTPUT 3 9
40. 4 0
41. WHAT’S YOUR STRATEGY? 4 1
42. STOP CALLING YOUR BUILD FLAKY!! 4 2
43. Identify Flaky tests Quarantine Plan Fix Tests Monitor 4 3
44. IT’S NOT ALWAYS YOUR TESTS 4 4
45. DO NOT GET OVERWHELMED BY FAILING TESTS 4 5
46. REMEMBER! 4 6
47. LOVE YOUR TEST CODE ! 4 7
48. There is no such thing as FLAKY TEST! If its flaky, its not a test ! 4 8
49. AND THE FUTURE… 4 9
50. REFERENCES •  An Empirical Analysis of Flaky tests •  Eradicating Non-Determinism in Tests – Martin Fowler •  No more flaky tests on the GO team - Pavan Sudarshan •  Top 12 reasons to write unit tests •  http://tech.just-eat.com/2015/03/30/how-to- fix-flaky-tests/ 5 0
51. THANK YOU yahya.poonawala@gmail.com priti.biyani6@gmail.com