The mysterious unit tests

This is a story about the unit tests suite at the National Film Board of Canada. An analysis and presentation was done on September 2013. The suite was too long so we were not always running our tests locally.

During the presentation, a summary of the analysis will be given and you will be told a series of tricks that helped us accelerate the performance of these mysterious tests.

Transcript

1. The mysterious unit tests Confoo 2016 Melanie Dubois @duboismelanie
4. Statistics Analysis / Solutions Actions Retrospective
5. Statistics
6. Tests from Sept. 9, 2011 to Sept. 19, 2013
7. Jenkins build 808 1095 tests 28:34 min.
8. Tests Length (sec.)
9. More Statistics % N Length (sec) 0.4 5 >=20 0.9 10 >=10 2.1 23 >=5 2.6 29 >=4 10.2 112 >=3 22.7 249 >=2 38.4 420 >=1
10. Unit Test vs Integration Test What is a unit test? • Covers a small unit of code • Includes the least possible paths What is an integration test? • Tests the contracts between units
11. Analysis / Solutions
12. The first step to faster unit tests: write unit tests The test client covers way more than what we are interested in testing URL Routing, Request Middleware, ORM, Template Rendering, Response Middleware, etc. 11 of the 16 longest tests use Django Client
13. Possible Solutions • Create an equivalent integration test • Move integration tests to “slow tests” suite • Call controllers directly, reducing overhead
14. Be Judicious About How Using setUp/tearDown Too many objects created in DB 8/16 (16166 DB queries) Touching the settings and the file system
15. Possible Solutions 1. Use Test Double (Fake It ‘Til You Make It) • Dummy • Stub • Spy • Fake • Mock • Patch decorator
16. Possible Solutions 2. Take advantage of SetUpClass/ tearDownClass • Runs once per test case • Data will persist between tests
17. Possible Solutions 3. Avoid the database, work with read-only, non persisted data
18. Possible Solutions 4. Remove explicit delete
19. Multiple Levels and Scopes of Concern
20. Possible Solutions 1. Separation of concern • Tests should cover code that is limited in functionality/scope • Ideally, a single method with limited external calls • If you can’t test the code efficiently, refactor the code 2. Use Mock in more complex situation • Mock patching • Track the way objects/methods interact
21. Using Unit Tests for Integration Testing
22. Possible Solutions 1. Write integration tests instead of unit tests 2. Move these tests in the “slow tests” suite
23. setUp/tearDown Instructions Inside the Test Function
24. Possible Solutions Move instructions to setUp/setUpClass or tearDown/tearDownClass
25. Actions
26. Simple Actions • Use setUpClass and tearDownClass • Remove duplicate tests • Move integration tests in the “slow tests” suite • Remove tearDown delete
27. More Demanding Actions • Test the controller instead of 200 • Use Mock, Patch, Factories, Avoid the database, Work with read-only, non persisted data • Break down big test classes into smaller ones with more focused setups and teardowns
28. Results • 4 tests suites • Unit tests suite runs in ~6 min. • We run our tests locally :)
29. Retrospective
30. A test is not a unit test if: • It talks to the database • It communicates across the network • It touches the file system • It can't run at the same time as any of your other unit tests • You have to do special things to your environment (such as editing config files) to run it.
31. Michael Feathers Tests that do these things aren't bad. Often they are worth writing, and they can be written in a unit test harness. However, it is important to be able to separate them from true unit tests so that we can keep a set of tests that we can run fast whenever we make our changes.
32. Test Pyramid
33. TDD Cycle • Think • Write a failing test first • Write enough production code to make it pass (no more than that) • Refactor • Repeat
34. Rigour Where? • Team • From team to team How? • Pairing/Code review • Presentation/Code reading • Follow up
35. References http://www.artima.com/weblogs/viewpost.jsp?thread=126923 http://www.aptest.com/testtypes.html http://www.softwaretestinghelp.com/types-of-software-testing/ http://www.codeproject.com/Tips/351122/What-is-software-testing-What-are-the-different-ty Architecture the lost years by Robert Martin EP19..21 of cleancoders by Robert Martin Test driven development by Kent Beck Let’s play TDD by James Shore The Art of Agile by James Shore Destroy All Software by Gary Bernhardt http://martinfowler.com/articles/mocksArentStubs.html Succeeding with Agile: Software Development Using Scrum by Mike Cohn xUnit Test Patterns: Refactoring Test Code by Gerard Meszaros https://www.nfb.ca
36. Thank you!
37. Questions?