[DjangoCon US] Testing in Django

Keynote about testing in/with/within Django presented at DjangoCon US 2017.

Transcript

1. TESTING in DJANGO by Ana Balica
3. @anabalica
5. Questions?
6. Django archeology 1.0
7. Django archeology 1.0 1.11 1.1 1.10 …
8. #2333 Add unit test framework for end-user Django applications
9. #2333 As an added incentive, this is a feature that is present in Rails. Add unit test framework for end-user Django applications ”
10. ./manage.py test 1.0
11. ./manage.py test app.TestClass.test_method 1.0
12. TEST RUNNER 1.0 setup test environment tests.py models.py teardown & results
13. Client 1.0 get post login logout
14. testcase 1.0 1.0 assert* Redirects (Not)Contains FormError Template(Not)Used
15. 1.1
16. Client 1.1 put head delete options
17. TestCase 1.1 TransactionTestCase
18. 1.1 my test enter transaction rollback transaction I am a TestCase burger
19. 1.1 I am a TransactionTestCase. I flush the database before each test.
20. 1.2
21. DjangoTestSuiteRunner from function to class
22. 1.2 failures = test_runner(test_labels, verbosity, interactive) if failures: sys.exit(failures)
23. 1.2 failures = test_runner(test_labels, verbosity, interactive) if failures: sys.exit(failures)
24. 1.2 0 success failures = test_runner(test_labels, verbosity, interactive) if failures: sys.exit(failures)
25. 1.2 1 failure failures = test_runner(test_labels, verbosity, interactive) if failures: sys.exit(failures)
26. 1.2 42 failure failures = test_runner(test_labels, verbosity, interactive) if failures: sys.exit(failures)
27. 1.2 256 success failures = test_runner(test_labels, verbosity, interactive) if failures: sys.exit(failures)
28. 1.2 failures = TestRunner(verbosity, interactive, failfast) if failures: sys.exit(1)
29. 1.2 failures = TestRunner(verbosity, interactive, failfast) if failures: sys.exit(1)
30. 1.2 0 / 1 success failure
31. failures = TestRunner(verbosity, interactive, failfast) if failures: sys.exit(1) 1.2 256 failure
32. failures = TestRunner(verbosity, interactive, failfast) if failures: sys.exit(1) 1.2 256 failure
33. failures = TestRunner(verbosity, interactive, failfast) if failures: sys.exit(1) 1.2 256 failure
34. failures = TestRunner(verbosity, interactive, failfast) if failures: sys.exit(1) 1.2 256 failure
35. failures = TestRunner(verbosity, interactive, failfast) if failures: sys.exit(1) 1.2 256 failure
36. 1.2 multiple databases
37. 1.2 multiple databases
38. 1.2 multiple databases primary replica
39. 1.2 multiple databases DATABASES = { 'default': { 'HOST': 'dbprimary', # ... plus other settings }, 'replica': { 'HOST': 'dbreplica', 'TEST_MIRROR': 'default', # ... plus other settings } }
40. 1.2 multiple databases DATABASES = { 'default': { 'HOST': 'dbprimary', # ... plus other settings }, 'replica': { 'HOST': 'dbreplica', 'TEST_MIRROR': 'default', # ... plus other settings } }
41. multiple databases DATABASES = { 'default': { 'HOST': 'dbprimary', # ... plus other settings }, 'replica': { 'HOST': 'dbreplica', 'TEST': { 'MIRROR': 'default', }, # ... plus other settings } }
42. multiple databases DATABASES = { 'default': { 'HOST': 'dbprimary', # ... plus other settings }, 'replica': { 'HOST': 'dbreplica', 'TEST': { 'MIRROR': 'default', }, # ... plus other settings } }
43. 1.2 multiple databases primary replica
44. 1.2 multiple databases primary replica
45. 1.2 multiple databases primary replica
46. 1.3
47. testcase 1.0 1.3 assert* QuerysetEqual NumQueries
48. 1.3 RequestFactory Client
49. 1.3 class RequestFactory(object): def request(self, **request): return WSGIRequest(self._base_environ(**request))
50. 1.3 doctests = tests + documentation
51. 1.3 doctests = tests + documentation
52. 1.4
53. 1.4 Transaction TestCase TestCase
54. 1.4 Transaction TestCase TestCase Simple TestCase LiveServer TestCase
55. 1.4 Transaction TestCase TestCase Simple TestCase LiveServer TestCase doesn’t hit the database
56. 1.4 Transaction TestCase TestCase Simple TestCase LiveServer TestCase doesn’t hit the database runs an http server
57. 1.4 Transaction TestCase TestCase Simple TestCase LiveServer TestCase doesn’t hit the database runs an http server
58. 1.5
59. 1.5 I am a TransactionTestCase. I flush the database before each test.
60. 1.5 I am a TransactionTestCase. I flush the database before each test. after
61. 1.5 flush run TransactionTestCase enter transaction run TestCase rollback transaction {
62. 1.5 flush run TransactionTestCase enter transaction run TestCase rollback transaction { dirty state
63. 1.5 flush {run TransactionTestCase enter transaction run TestCase rollback transaction
64. 1.5 flush { run TransactionTestCase enter transaction run TestCase rollback transaction
65. 1.5 flush {run TransactionTestCase enter transaction run TestCase rollback transaction
66. 1.5 flush {run TransactionTestCase enter transaction run TestCase rollback transaction
67. 1.5 flush {run TransactionTestCase enter transaction run TestCase rollback transaction problem solved
68. 1.6
69. Client 1.6 patch
70. 1.6 TEST_RUNNER improvements Test discovery Full paths vs pseudo paths Doctests discovery
71. 1.6 TEST_RUNNER improvements Test discovery Full paths vs pseudo paths Doctests discovery
72. 1.6 TEST_RUNNER improvements Test discovery Full paths vs pseudo paths Doctests discovery
73. 1.7
74. 1.7 unittest2
75. 1.7 unittest2 unittest
76. 1.7 LiveServerTestCase StaticLiveServerTestCase
77. 1.8
78. Client 1.8 trace
79. TestCase before enter atomic load fixtures … exit atomic close connections 1.8
80. TestCase before enter atomic load fixtures … exit atomic close connections 1.8
81. TestCase before enter atomic load fixtures … exit atomic close connections } times # of tests 1.8
82. TestCase after enter atomic load fixtures enter atomic … exit atomic exit atomic close connections 1.8
83. TestCase after enter atomic load fixtures enter atomic … exit atomic exit atomic close connections { once 1.8
84. TestCase after enter atomic load fixtures enter atomic … exit atomic exit atomic close connections } times # of tests { once 1.8
85. 1.9
86. 1.9 --parallel
87. 1.9
88. 1.9 workers
89. 1.9 workers databases
90. 1.9 workers databases suite
91. 1.9 workers databases partitions
92. 1.9 workers databases partitions
93. 1.9 workers databases partitions
94. 1.9 workers databases partitions
95. 1.9 workers databases partitions
96. 1.9 workers databases partitions
97. 1.9 workers databases partitions
98. nose multiprocess plugin
99. 1.10
100. 1.10 class SampleTestCase(TestCase): @tag('slow') def test_slow(self): ...
101. 1.10 ./manage.py test --tag=slow class SampleTestCase(TestCase): @tag('slow') def test_slow(self): ...
102. 1.10 ./manage.py test --tag=slow ./manage.py test --exclude-tag=slow class SampleTestCase(TestCase): @tag('slow') def test_slow(self): ...
103. nose attrib plugin py.test markers
104. nose attrib plugin py.test markers will do the same
105. 1.11
106. # Starting with Python 3.4 def test_even(self): for i in range(0, 6): with self.subTest(i=i): self.assertEqual(i % 2, 0)
107. # Starting with Python 3.4 def test_even(self): for i in range(0, 6): with self.subTest(i=i): self.assertEqual(i % 2, 0)
108. # Starting with Python 3.4 def test_even(self): for i in range(0, 6): with self.subTest(i=i): self.assertEqual(i % 2, 0) --parallel
109. TEST bed or what happens when you run ./manage.py test
112. $ ./manage.py test 1
113. TestRunner = get_runner(settings, options['testrunner']) test_runner = TestRunner(**options) failures = test_runner.run_tests(test_labels) ../management/commands/test.py 2
114. TestRunner = get_runner(settings, options['testrunner']) test_runner = TestRunner(**options) failures = test_runner.run_tests(test_labels) ../management/commands/test.py 2
115. 3 self.setup_test_environment()
116. 3 self.setup_test_environment() locmem email backend
117. 3 self.setup_test_environment() locmem email backend instrumented test renderer
118. 3 self.setup_test_environment() locmem email backend instrumented test renderer deactivate translations
119. 4 self.build_suite(test_labels, extra_tests)
120. 4 self.build_suite(test_labels, extra_tests)
121. 5 self.setup_databases() 7 self.run_suite(suite) 8 self.teardown_databases(old_config) 6 self.run_checks()
122. 5 self.setup_databases() 7 self.run_suite(suite) 8 self.teardown_databases(old_config) 6 self.run_checks()
123. 5 self.setup_databases() 7 self.run_suite(suite) 8 self.teardown_databases(old_config) 6 self.run_checks()
124. 5 self.setup_databases() 7 self.run_suite(suite) 8 self.teardown_databases(old_config) 6 self.run_checks()
125. 9 self.teardown_test_environment()
126. original email backend 9 self.teardown_test_environment()
127. original email backend 9 original test renderer self.teardown_test_environment()
128. original email backend 9 original test renderer delete state and mailbox self.teardown_test_environment()
129. 10 self.suite_result(suite, result) len(result.failures) + len(result.errors) if failures: sys.exit(1)
130. test classes
131. SimpleTestCase TransactionTestCase TestCase LiveServerTestCase StaticLiveServerTestCase
132. SimpleTestCase no database queries access to test client fast
133. TransactionTestCase allows database queries access to test client fast allows database transactions flushes database after each test
134. TestCase allows database queries access to test client faster restricts database transactions runs each test in a transaction
135. LiveServerTestCase acts like TransactionTestCase launches a live HTTP server in a separate thread
136. StaticLiveServerTestCase acts like TransactionTestCase launches a live HTTP server in a separate thread serves static files
137. Client
138. Client RequestFactory ClientHandler
139. Client RequestFactory ClientHandler constructs requests encodes data
140. Client RequestFactory ClientHandler constructs requests encodes data
141. Client RequestFactory ClientHandler constructs requests encodes data
142. Client RequestFactory ClientHandler stateful response++ handles redirects constructs requests encodes data
143. Client RequestFactory ClientHandler stateful response++ handles redirects constructs requests encodes data
144. Client RequestFactory ClientHandler stateful response++ handles redirects constructs requests encodes data
145. Client RequestFactory ClientHandler stateful response++ handles redirects constructs requests encodes data
146. Client RequestFactory ClientHandler stateful response++ handles redirects loads middleware emulates disables CSRF constructs requests encodes data
147. Client RequestFactory ClientHandler stateful response++ handles redirects loads middleware emulates disables CSRF constructs requests encodes data
148. Client RequestFactory ClientHandler stateful response++ handles redirects loads middleware emulates disables CSRF constructs requests encodes data
149. Client RequestFactory ClientHandler stateful response++ handles redirects loads middleware emulates disables CSRF constructs requests encodes data
150. Client RequestFactory ClientHandler stateful response++ handles redirects loads middleware emulates disables CSRF constructs requests encodes data
151. Quality
152. Factory Boy fixtures replacement with random and realistic values
153. Factory Boy fixtures replacement with random and realistic values generated by Faker
154. property based testing with Hypothesis
155. from hypothesis import given from hypothesis.strategies import text @given(text()) def test_decode_inverts_encode(s): assert decode(encode(s)) == s
156. from hypothesis import given from hypothesis.strategies import text @given(text()) def test_decode_inverts_encode(s): assert decode(encode(s)) == s
157. from hypothesis import given from hypothesis.strategies import text @given(text()) def test_decode_inverts_encode(s): assert decode(encode(s)) == s
158. from hypothesis import given from hypothesis.strategies import text @given(text()) def test_decode_inverts_encode(s): assert decode(encode(s)) == s
159. from hypothesis import given from hypothesis.strategies import text @given(text()) def test_decode_inverts_encode(s): assert decode(encode(s)) == s
160. from hypothesis import given from hypothesis.strategies import text @given(text()) def test_decode_inverts_encode(s): assert decode(encode(s)) == s rerun for different values
161. from hypothesis.extra.django.models import models from hypothesis.strategies import integers models(Customer).example() models(Customer, age=integers( min_value=0, max_value=120) ).example()
162. from hypothesis.extra.django.models import models from hypothesis.strategies import integers models(Customer).example() models(Customer, age=integers( min_value=0, max_value=120) ).example()
163. from hypothesis.extra.django import TestCase from hypothesis import given from hypothesis.extra.django.models import models from hypothesis.strategies import lists, integers class TestProjectManagement(TestCase): @given( models(Project, collaborator_limit=integers(min_value=0, max_value=20)), lists(models(User), max_size=20)) def test_can_add_users_up_to_collaborator_limit(self, project, collaborators): for c in collaborators: if project.at_collaboration_limit(): with self.assertRaises(LimitReached): project.add_user(c) self.assertFalse(project.team_contains(c)) else: project.add_user(c) self.assertTrue(project.team_contains(c))
164. from hypothesis.extra.django import TestCase from hypothesis import given from hypothesis.extra.django.models import models from hypothesis.strategies import lists, integers class TestProjectManagement(TestCase): @given( models(Project, collaborator_limit=integers(min_value=0, max_value=20)), lists(models(User), max_size=20)) def test_can_add_users_up_to_collaborator_limit(self, project, collaborators): for c in collaborators: if project.at_collaboration_limit(): with self.assertRaises(LimitReached): project.add_user(c) self.assertFalse(project.team_contains(c)) else: project.add_user(c) self.assertTrue(project.team_contains(c))
165. class TestProjectManagement(TestCase): @given( models(Project, collaborator_limit=integers(min_value=0, max_value=20)), lists(models(User), max_size=20)) def test_can_add_users_up_to_collaborator_limit(self, project, collaborators): for c in collaborators: if project.at_collaboration_limit(): with self.assertRaises(LimitReached): project.add_user(c) self.assertFalse(project.team_contains(c)) else: project.add_user(c) self.assertTrue(project.team_contains(c))
166. class TestProjectManagement(TestCase): @given( models(Project, collaborator_limit=integers(min_value=0, max_value=20)), lists(models(User), max_size=20)) def test_can_add_users_up_to_collaborator_limit(self, project, collaborators): for c in collaborators: if project.at_collaboration_limit(): with self.assertRaises(LimitReached): project.add_user(c) self.assertFalse(project.team_contains(c)) else: project.add_user(c) self.assertTrue(project.team_contains(c))
167. class TestProjectManagement(TestCase): @given( models(Project, collaborator_limit=integers(min_value=0, max_value=20)), lists(models(User), max_size=20)) def test_can_add_users_up_to_collaborator_limit(self, project, collaborators): for c in collaborators: if project.at_collaboration_limit(): with self.assertRaises(LimitReached): project.add_user(c) self.assertFalse(project.team_contains(c)) else: project.add_user(c) self.assertTrue(project.team_contains(c))
168. class TestProjectManagement(TestCase): @given( models(Project, collaborator_limit=integers(min_value=0, max_value=20)), lists(models(User), max_size=20)) def test_can_add_users_up_to_collaborator_limit(self, project, collaborators): for c in collaborators: if project.at_collaboration_limit(): with self.assertRaises(LimitReached): project.add_user(c) self.assertFalse(project.team_contains(c)) else: project.add_user(c) self.assertTrue(project.team_contains(c))
169. Falsifying example: test_can_add_users_up_to_collaborator_limit( self=TestProjectManagement(), project=Project('', 1), collaborators=[ User(.@.com), User(.@.com) ] ) Traceback (most recent call last): ... raise LimitReached() manager.models.LimitReached
170. Property based testing is more complicated, yet more valuable
171. Django test code coverage is 75%
172. Deceptive metric
173. High coverage != high quality
174. Mutation testing available Python implementation: mutpy
175. mut.py --target node --unit-test test_node target unit test
176. if foo and bar: do_this() target unit test
177. if foo and bar: do_this() if foo or bar: do_this() mutant unit test
178. mutant killed
179. mutant killed mutant survived
180. AOR - arithmetic operator replacement BCR - break continue replacement COI - conditional operator insertion CRP - constant replacement DDL - decorator deletion LOR - logical operator replacement
181. [*] Start mutation process: - targets: django.utils.encoding - tests: tests.utils_tests.test_encoding [*] 10 tests passed: - tests.utils_tests.test_encoding [0.00533 s] [*] Start mutants generation and execution: ... [*] Mutation score [12.19066 s]: 32.1% - all: 88 - killed: 24 (27.3%) - survived: 55 (62.5%) - incompetent: 7 (8.0%) - timeout: 2 (2.3%)
182. [*] Start mutation process: - targets: django.utils.encoding - tests: tests.utils_tests.test_encoding [*] 10 tests passed: - tests.utils_tests.test_encoding [0.00533 s] [*] Start mutants generation and execution: ... [*] Mutation score [12.19066 s]: 32.1% - all: 88 - killed: 24 (27.3%) - survived: 55 (62.5%) - incompetent: 7 (8.0%) - timeout: 2 (2.3%)
183. Django duration utils mutation score is 89%
184. Django duration utils mutation score is 89% and the coverage is 100%
185. Django encoding utils mutation score is 64%
186. Django encoding utils mutation score is 64% while the coverage is 100%
187. django Testing tutorial
188. my_app ├── __init__.py ├── admin.py ├── migrations │ └── __init__.py ├── models.py ├── tests.py └── views.py
189. my_app ├── __init__.py ├── admin.py ├── migrations │ └── __init__.py ├── models.py ├── tests.py └── views.py
190. When testing, more is better
191. # of tests total execution time
192. slow tests, slow feedback loop
193. 7 tips on how to speed up your tests
194. 7 tips on how to speed up your tests #4 will shock you
196. 1. use MD5PasswordHasher
197. 1. use MD5PasswordHasher 2. have more SimpleTestCase
198. 1. use MD5PasswordHasher 2. have more SimpleTestCase 3. use setUpTestData()
199. 1. use MD5PasswordHasher 2. have more SimpleTestCase 3. use setUpTestData() 4. use mocks EVERYWHERE
200. 1. use MD5PasswordHasher 2. have more SimpleTestCase 3. use setUpTestData() 4. use mocks EVERYWHERE
201. 1. use MD5PasswordHasher 2. have more SimpleTestCase 3. use setUpTestData() 4. use mocks EVERYWHERE 5. be vigilant of what gets created in setUp()
202. 1. use MD5PasswordHasher 2. have more SimpleTestCase 3. use setUpTestData() 4. use mocks EVERYWHERE 5. be vigilant of what gets created in setUp() 6. don’t save model objects if not necessary
203. 1. use MD5PasswordHasher 2. have more SimpleTestCase 3. use setUpTestData() 4. use mocks EVERYWHERE 5. be vigilant of what gets created in setUp() 6. don’t save model objects if not necessary 7. isolate unit tests
204. 1. use MD5PasswordHasher 2. have more SimpleTestCase 3. use setUpTestData() 4. use mocks EVERYWHERE 5. be vigilant of what gets created in setUp() 6. don’t save model objects if not necessary 7. isolate unit tests
205. class SimpleTest(TestCase): def setUp(self): for _ in range(10): Robot.objects.create() 5. be vigilant of what gets created in setUp()
206. class SimpleTest(TestCase): def setUp(self): for _ in range(10): Robot.objects.create() 5. be vigilant of what gets created in setUp()
207. 6. don’t save model objects if not necessary
208. 6. don’t save model objects if not necessary Robot.objects.create() instead of
209. 6. don’t save model objects if not necessary Robot.objects.create() Robot() instead of maybe do
210. 6. don’t save model objects if not necessary Robot.objects.create() Robot() RobotFactory.build() instead of maybe do or
211. 6. don’t save model objects if not necessary Robot.objects.create() Robot() RobotFactory.build() RobotFactory.stub() instead of maybe do or or
212. 6. don’t save model objects if not necessary Robot.objects.create() Robot() RobotFactory.build() RobotFactory.stub() factory boy instead of maybe do or or }
213. 7. isolate unit tests
214. 7. isolate unit tests
215. 7. isolate unit tests
216. 7. isolate unit tests unit tests functional tests
217. 7. isolate unit tests unit tests functional tests ./manage.py test --tag=unit
219. Ugh, taxes!
220. Product ProductQuestionnaire name ingredients price product category drink_category food_category has_decorations is_coated_in_chocolate is_warm is_cold
222. Production code Test
223. class ProductQuestionnaireCreate(CreateView): def form_valid(self, form): if is_biscuit and is_coated_in_chocolate: set_vat_20() return super().form_valid(form) class ProductQuestionnaireCreateTestCase(TestCase): def test_20p_vat_if_coated_in_chocolate_biscuit(self): product = ProductFactory() response = self.client.post(self.url, {'q1': 'a1', 'q2': 'a2'}) product.refresh_from_db() self.assertEqual(product.vat, 20) solution #1
224. class ProductQuestionnaireCreate(CreateView): def form_valid(self, form): if is_biscuit and is_coated_in_chocolate: set_vat_20() return super().form_valid(form) class ProductQuestionnaireCreateTestCase(TestCase): def test_20p_vat_if_coated_in_chocolate_biscuit(self): product = ProductFactory() response = self.client.post(self.url, {'q1': 'a1', 'q2': 'a2'}) product.refresh_from_db() self.assertEqual(product.vat, 20) solution #1
225. class ProductQuestionnaireCreate(CreateView): def form_valid(self, form): return super().form_valid(form)
226. class ProductQuestionnaireCreateTestCase(TestCase): def test_20p_vat_if_coated_in_chocolate_biscuit(self): def test_0p_vat_if_baguette(self): def test_0p_vat_if_flapjack(self): def test_20p_vat_if_cereal_bar(self):
227. To test if I need to pay 20% VAT for biscuits coated in chocolate, I need to: go through the router interact with database send input to receive output
228. Mocks are not a solution
229. class ProductQuestionnaireForm(forms.ModelForm): def save(self, commit=True): instance = super().save(commit) if is_biscuit and is_coated_in_chocolate: set_vat_20() return instance solution #2 class ProductQuestionnaireFormTestCase(TestCase): def test_20p_vat_if_coated_in_chocolate_biscuit(self): product = ProductFactory() form = ProductQuestionnaireForm(data={'k1': 'v1', 'k2': 'v2'}) self.assertTrue(form.is_valid()) form.save() product.refresh_from_db() self.assertEqual(product.vat, 20)
230. class ProductQuestionnaireForm(forms.ModelForm): def save(self, commit=True): instance = super().save(commit) if is_biscuit and is_coated_in_chocolate: set_vat_20() return instance solution #2 class ProductQuestionnaireFormTestCase(TestCase): def test_20p_vat_if_coated_in_chocolate_biscuit(self): product = ProductFactory() form = ProductQuestionnaireForm(data={'k1': 'v1', 'k2': 'v2'}) self.assertTrue(form.is_valid()) form.save() product.refresh_from_db() self.assertEqual(product.vat, 20)
231. To test if I need to pay 20% VAT for biscuits coated in chocolate, I need to: go through the router interact with database send input to receive output
232. solution #3 class VATCalculator(object): def calculate_vat(self, **kwargs): if is_biscuit and is_coated_in_chocolate: return 20 class VATCalculatorTestCase(SimpleTestCase): def test_20p_vat_if_coated_in_chocolate_biscuit(self): calc = VATCalculator() self.assertEqual(calc.calculate_vat( is_biscuit=True, is_coated_in_choco=True ), 20)
233. To test if I need to pay 20% VAT for biscuits coated in chocolate, I need to: go through the router interact with database send input to receive output
234. reusability testability extensibility
236. Tests have more in them than we think
237. Happy testing and big bear hug thanks!