Are Your Tests Slowing You Down?

Testing is a Good Thing, right? Especially automated testing. But "Good things come to those who wait" is not something that's going to appeal to the busy developer. You want results, and you want them now. You're in The Zone working on a problem, and the last thing you want is to break your flow wrestling with your testing framework or waiting for the tests to finish running.

More code means more tests. More coverage means more tests. More tests mean more time. Time that you want to spend being productive, creative, innovative. How can you balance the need for quality with the need for speed?

In this talk, Trisha will identify issues that slow down developers when writing, running and debugging tests, and look at tools that can help developers with each of these problems. There will be live coding, analysis of social media poll results, an overview of solutions in this space, "best practice" recommendations, and machine learning will be mentioned at some point.

Transcript

1. Trisha Gee Head of Developer Advocacy, Gradle Are Your Tests Slowing You Down?
2. • Head of Developer Advocacy at Gradle • Java Champion & JetBrains Community Contributor • 20+ years development experience • Author & YouTuber Trisha Gee
3. Tests Make You More Productive?
6. So it’s OK not to write tests, right?
7. Testing is Xanax for programmers, it’s an anxiety reducing technique.” https://youtu.be/guycIP56YeY Kent Beck
8. What slows down your testing? Writing tests Running tests locally Running tests in CI Troubleshooting tests Unreliable test results
9. 1. Writing 2. Troubleshooting 3. Running
10. 1. Writing Tests Takes Time
11. Use Your IDE
12. AI Will Make You Super Productive
13. AI is good for generating mundane or predictable code
14. AI is good for generating test data
15. AI can consider edge cases we haven’t thought of
16. AI can test what’s there. It can’t tell you how the system should behave
17. AI is a decent companion …but it doesn’t replace you
18. The keyboard is not the bottleneck.” Dave Farley
19. The productivity myth suggests that anything we spend time on is up for automation…. The importance and value of thinking about our work and why we do it is waved away as a distraction.” https://www.techpolicy.press/challenging-the-myths-of-generative-ai/ Eryk Salvaggio
20. Bring a friend
21. We recommend that the developers, who are good at writing efficient, maintainable code, work together with the testers, who are good at specifying test cases, to automate tests.” Agile Testing Condensed: A Brief Introduction Lisa Crispin, Janet Gregory
22. https://bit.ly/3RG85vc
23. Creativity is a wild mind and a disciplined eye.” Dorothy Parker
24. Invest the time in writing to save time troubleshooting
25. 2. Troubleshooting Takes Time
26. What slows down your testing? Writing tests Running tests locally Running tests in CI Troubleshooting tests Unreliable test results
27. What is this test anyway?
29. Your test needs to tell you what is expected and why, so you understand what to do when it fails.
32. Why do tests fail? Body text
33. ‣ Code is wrong ‣ Test is wrong
34. Something changed
35. • Production code • Test code • Refactoring/structure • Data • Infrastructure • Load or usage
36. 1. Whose fault is it?
37. 1. Triage
38. • Production code • Test code • Refactoring/structure • Data • Infrastructure • Load or usage
39. You need data
43. 2. Fix it
44. Your best and wisest refuge from all troubles is in your science.” Ada Lovelace
52. Learn how to use your debugger!
56. Aim for fast feedback and easy reproduction
57. Fixed! ✅
58. What slows down your testing? Writing tests Running tests locally Running tests in CI Troubleshooting tests Unreliable test results
59. “Unreliable test results”
60. I have made the decision to trust you Body text
61. 1. Find Your Flaky Tests
64. This is not a fix!
66. So now you can ignore those flaky tests
68. 2. Quarantine your flaky tests
69. 3. Fix your flaky tests!!!
70. If you add a new test and it’s flaky
71. If you add a new test and it’s flaky Or are working on a test and it becomes flaky
72. If you add a new test and it’s flaky Or are working on a test and it becomes flaky Fix it now!! (Future you will thank you)
73. Make Time To Fix Flaky Tests • When they impact you directly • Attach them to related functionality • Flaky Test Days • Add flaky tests to the backlog • 20% time, tech debt time, cleanup time…
74. Causes of flakiness 1. Environment, Network, and Resources 2. Integration Points 3. Concurrency, Asynchronous Programming, and Waiting 4. Setup/Teardown and Test Data 5. System Behaviour Known good practices and tools Not always in my control Try to minimise these Can be REALLY HARD
75. https://youtu.be/vHBzZHE4tJ0 1. Environment 2. Test Data 3. Versioning 4. Resource Use 5. System Behaviour
76. 3. Running Tests Takes Time
77. What slows down your testing? Writing tests Running tests locally Running tests in CI Troubleshooting tests Unreliable test results
79. Use a Build Cache
80. Spring Boot Build Times
81. Very effective for multi-module projects
82. No cache: Full build required
83. Build cache: Changing a module’s implementation
84. Remote Build Cache
85. Run tests in parallel (local)
87. Run tests in parallel (CI)
89. Test Distribution
91. Test Distribution Results ‑ ~50% ‑ ~50% ‑ ~50% Doubling the number of executors cuts build time in half
92. Let’s use AI!
94. The test selection procedure selects fewer than a third of the tests that would be selected on the basis of build dependencies” Predictive Test Selection - Meta Research https://research.facebook.com/publications/predictive-test-selection/ Mateusz Machalica, Alex Samylkin, Meredith Porth, Satish Chandra
95. Predictive Test Selection time savings
97. What if we just make our tests faster?
98. Like any performance problem, focus on the worst first
104. “Once things like databases become a hindrance, optimise the testing loop, mock things out to run things fast. You can run locally and you don't have all the same infra dependencies as someone else.” On DevEx, PaaS, and Testing in Production https://bit.ly/3ZA0uQM Kelsey Hightower
105. Look out for expensive setup and tear down
106. Invent Time Travel
107. Write Better Tests
109. • Separation of concerns • Encapsulation • High cohesion and loose coupling • SOLID • DRY • Descriptive naming • Modularisation • Designing for performance
110. Tension between these ideals
111. Refactor and Delete
112. “Yes, I regularly delete tests that no longer add unique value.” Angie Jones (@techgirl1908)
113. In Summary
114. Automate as much as possible
115. Tooling can help you
116. There is a time investment in writing tests, because there should be thinking involved
117. Invest time in writing better tests. They will be easier to troubleshoot, less flaky, and better document your system’s expected behaviour
118. Testing is not only for checking the system does what it’s supposed to, it’s for documenting the expected behaviour.” Israel Boza Rodriguez (@IsraKaos)
119. Thank you! bit.ly/slow-tests tinyurl.com/dv-trial