[DUTH] Testing in Django

A talk presented at Django Under The Hood 2016 in Amsterdam about testing in/with/within Django and all that.

Transcript

1. TESTING in DJANGO by Ana Balica
2. @anabalica
3. POTATO
4. Django archeology 1.0
5. Django archeology 1.0 1.10 1.1 1.9 …
6. #2333 Add unit test framework for end-user Django applications
7. #2333 As an added incentive, this is a feature that is present in Rails. Add unit test framework for end-user Django applications ”
8. ./manage.py test 1.0
9. ./manage.py test app.TestClass.test_method 1.0
10. TEST RUNNER 1.0 setup test environment tests.py models.py teardown & results
11. Client 1.0 get post login logout
12. testcase 1.0 1.0 assert* Redirects (Not)Contains FormError Template(Not)Used
13. 1.1
14. Client 1.1 put head delete options
15. TestCase 1.1 TransactionTestCase
16. 1.1 my test enter transaction rollback transaction I am a TestCase burger
17. 1.1 I am a TransactionTestCase. I flush the database before each test.
18. 1.2
19. DjangoTestSuiteRunner from function to class
20. 1.2 failures = test_runner(test_labels, verbosity, interactive) if failures: sys.exit(failures)
21. 1.2 failures = test_runner(test_labels, verbosity, interactive) if failures: sys.exit(failures)
22. 1.2 0 success failures = test_runner(test_labels, verbosity, interactive) if failures: sys.exit(failures)
23. 1.2 1 failure failures = test_runner(test_labels, verbosity, interactive) if failures: sys.exit(failures)
24. 1.2 42 failure failures = test_runner(test_labels, verbosity, interactive) if failures: sys.exit(failures)
25. 1.2 256 success failures = test_runner(test_labels, verbosity, interactive) if failures: sys.exit(failures)
26. 1.2 failures = TestRunner(verbosity, interactive, failfast) if failures: sys.exit(bool(failures))
27. 1.2 failures = TestRunner(verbosity, interactive, failfast) if failures: sys.exit(bool(failures))
28. 1.2 0 / 1 success failure
29. failures = TestRunner(verbosity, interactive, failfast) if failures: sys.exit(1) 1.2 256 failure
30. failures = TestRunner(verbosity, interactive, failfast) if failures: sys.exit(1) 1.2 256 failure
31. failures = TestRunner(verbosity, interactive, failfast) if failures: sys.exit(1) 1.2 256 failure
32. failures = TestRunner(verbosity, interactive, failfast) if failures: sys.exit(1) 1.2 256 failure
33. failures = TestRunner(verbosity, interactive, failfast) if failures: sys.exit(1) 1.2 256 failure
34. 1.2 multiple databases
35. 1.2 multiple databases
36. 1.2 multiple databases primary replica
37. 1.2 multiple databases DATABASES = { 'default': { 'HOST': 'dbprimary', # ... plus other settings }, 'replica': { 'HOST': 'dbreplica', 'TEST_MIRROR': 'default', # ... plus other settings } }
38. 1.2 multiple databases DATABASES = { 'default': { 'HOST': 'dbprimary', # ... plus other settings }, 'replica': { 'HOST': 'dbreplica', 'TEST_MIRROR': 'default', # ... plus other settings } }
39. multiple databases DATABASES = { 'default': { 'HOST': 'dbprimary', # ... plus other settings }, 'replica': { 'HOST': 'dbreplica', 'TEST': { 'MIRROR': 'default', }, # ... plus other settings } }
40. multiple databases DATABASES = { 'default': { 'HOST': 'dbprimary', # ... plus other settings }, 'replica': { 'HOST': 'dbreplica', 'TEST': { 'MIRROR': 'default', }, # ... plus other settings } }
41. 1.2 multiple databases primary replica
42. 1.2 multiple databases primary replica
43. 1.2 multiple databases primary replica
44. 1.3
45. testcase 1.0 1.3 assert* QuerysetEqual NumQueries
46. 1.3 RequestFactory Client
47. 1.3 RequestFactory Client
48. 1.3 class RequestFactory(object): def request(self, **request): return WSGIRequest(self._base_environ(**request))
49. 1.3 doctests = tests + documentation
50. 1.3 doctests = tests + documentation
51. 1.3 @skipIfDBFeature(feature) @skipUnlessDBFeature(feature)
52. 1.4
53. 1.4 Transaction TestCase TestCase
54. 1.4 Transaction TestCase TestCase Simple TestCase LiveServer TestCase
55. 1.4 Transaction TestCase TestCase Simple TestCase LiveServer TestCase doesn’t hit the database
56. 1.4 Transaction TestCase TestCase Simple TestCase LiveServer TestCase doesn’t hit the database runs an http server
57. 1.4 Transaction TestCase TestCase Simple TestCase LiveServer TestCase doesn’t hit the database runs an http server
58. 1.5
59. 1.5 TEST_RUNNER improvements Python 3 Tutorial on testing New assertions
60. 1.5 I am a TransactionTestCase. I flush the database before each test.
61. 1.5 I am a TransactionTestCase. I flush the database before each test. after
62. 1.5 flush run TransactionTestCase enter transaction run TestCase rollback transaction {
63. 1.5 flush run TransactionTestCase enter transaction run TestCase rollback transaction { dirty state
64. 1.5 flush {run TransactionTestCase enter transaction run TestCase rollback transaction
65. 1.5 flush { run TransactionTestCase enter transaction run TestCase rollback transaction
66. 1.5 flush {run TransactionTestCase enter transaction run TestCase rollback transaction
67. 1.5 flush {run TransactionTestCase enter transaction run TestCase rollback transaction
68. 1.5 flush {run TransactionTestCase enter transaction run TestCase rollback transaction problem solved
69. 1.6
70. Client 1.6 patch
71. 1.6 TEST_RUNNER improvements Test discovery Full paths vs pseudo paths Doctests discovery
72. 1.6 TEST_RUNNER improvements Test discovery Full paths vs pseudo paths Doctests discovery
73. 1.6 TEST_RUNNER improvements Test discovery Full paths vs pseudo paths Doctests discovery
74. 1.7
75. 1.7 unittest2
76. 1.7 unittest2 unittest
77. 1.7 LiveServerTestCase StaticLiveServerTestCase
78. 1.8
79. Client 1.8 trace
80. TestCase before enter atomic load fixtures … exit atomic close connections 1.8
81. TestCase before enter atomic load fixtures … exit atomic close connections 1.8
82. TestCase before enter atomic load fixtures … exit atomic close connections } times # of tests 1.8
83. TestCase after enter atomic load fixtures enter atomic … exit atomic exit atomic close connections 1.8
84. TestCase after enter atomic load fixtures enter atomic … exit atomic exit atomic close connections 1.8
85. TestCase after enter atomic load fixtures enter atomic … exit atomic exit atomic close connections } times # of tests 1.8
86. TestCase after enter atomic load fixtures enter atomic … exit atomic exit atomic close connections } times # of tests { once 1.8
87. 1.9
88. 1.9 --parallel
89. 1.9
90. 1.9 workers
91. 1.9 workers databases
92. 1.9 workers databases suite
93. 1.9 workers databases partitions
94. 1.9 workers databases partitions
95. 1.9 workers databases partitions
96. 1.9 workers databases partitions
97. 1.9 workers databases partitions
98. 1.9 workers databases partitions
99. 1.9 workers databases partitions
100. nose multiprocess plugin
101. 1.10
102. 1.10 class SampleTestCase(TestCase): @tag('slow') def test_slow(self): ...
103. 1.10 ./manage.py test --tag=slow class SampleTestCase(TestCase): @tag('slow') def test_slow(self): ...
104. 1.10 ./manage.py test --tag=slow ./manage.py test --exclude-tag=slow class SampleTestCase(TestCase): @tag('slow') def test_slow(self): ...
105. nose attrib plugin py.test markers
106. nose attrib plugin py.test markers will do the same
107. TEST bed or what happens when you run ./manage.py test
110. $ ./manage.py test 1
111. TestRunner = get_runner(settings, options['testrunner']) test_runner = TestRunner(**options) failures = test_runner.run_tests(test_labels) ../management/commands/test.py 2
112. TestRunner = get_runner(settings, options['testrunner']) test_runner = TestRunner(**options) failures = test_runner.run_tests(test_labels) ../management/commands/test.py 2
113. 3 self.setup_test_environment()
114. 3 self.setup_test_environment() locmem email backend
115. 3 self.setup_test_environment() locmem email backend instrumented test renderer
116. 3 self.setup_test_environment() locmem email backend instrumented test renderer deactivate translations
117. 4 self.build_suite(test_labels, extra_tests)
118. 4 self.build_suite(test_labels, extra_tests)
119. 5 self.setup_databases() 6 self.run_suite(suite) 7 self.teardown_databases(old_config)
120. 5 self.setup_databases() 6 self.run_suite(suite) 7 self.teardown_databases(old_config)
121. 5 self.setup_databases() 6 self.run_suite(suite) 7 self.teardown_databases(old_config)
122. 8 self.teardown_test_environment()
123. original email backend 8 self.teardown_test_environment()
124. original email backend 8 original test renderer self.teardown_test_environment()
125. original email backend 8 original test renderer delete state and mailbox self.teardown_test_environment()
126. 9 self.suite_result(suite, result) len(result.failures) + len(result.errors) if failures: sys.exit(1)
127. all the test classes
128. SimpleTestCase TransactionTestCase TestCase LiveServerTestCase StaticLiveServerTestCase
129. SimpleTestCase no database queries access to test client fast
130. TransactionTestCase allows database queries access to test client fast allows database transactions flushes database after each test
131. TestCase allows database queries access to test client faster restricts database transactions runs each test in a transaction
132. LiveServerTestCase acts like TransactionTestCase launches a live HTTP server in a separate thread
133. StaticLiveServerTestCase acts like TransactionTestCase launches a live HTTP server in a separate thread serves static files
134. Client
135. Client RequestFactory ClientHandler
136. Client RequestFactory ClientHandler constructs requests encodes data
137. Client RequestFactory ClientHandler constructs requests encodes data
138. Client RequestFactory ClientHandler constructs requests encodes data
139. Client RequestFactory ClientHandler stateful response++ handles redirects constructs requests encodes data
140. Client RequestFactory ClientHandler stateful response++ handles redirects constructs requests encodes data
141. Client RequestFactory ClientHandler stateful response++ handles redirects constructs requests encodes data
142. Client RequestFactory ClientHandler stateful response++ handles redirects constructs requests encodes data
143. Client RequestFactory ClientHandler stateful response++ handles redirects loads middleware emulates disables CSRF constructs requests encodes data
144. Client RequestFactory ClientHandler stateful response++ handles redirects loads middleware emulates disables CSRF constructs requests encodes data
145. Client RequestFactory ClientHandler stateful response++ handles redirects loads middleware emulates disables CSRF constructs requests encodes data
146. Client RequestFactory ClientHandler stateful response++ handles redirects loads middleware emulates disables CSRF constructs requests encodes data
147. Client RequestFactory ClientHandler stateful response++ handles redirects loads middleware emulates disables CSRF constructs requests encodes data
148. Quality
149. Factory Boy fixtures replacement with random and realistic values
150. Factory Boy fixtures replacement with random and realistic values generated by Faker
151. property based testing with Hypothesis
152. from hypothesis import given from hypothesis.strategies import text @given(text()) def test_decode_inverts_encode(s): assert decode(encode(s)) == s
153. from hypothesis import given from hypothesis.strategies import text @given(text()) def test_decode_inverts_encode(s): assert decode(encode(s)) == s
154. from hypothesis import given from hypothesis.strategies import text @given(text()) def test_decode_inverts_encode(s): assert decode(encode(s)) == s
155. from hypothesis import given from hypothesis.strategies import text @given(text()) def test_decode_inverts_encode(s): assert decode(encode(s)) == s
156. from hypothesis import given from hypothesis.strategies import text @given(text()) def test_decode_inverts_encode(s): assert decode(encode(s)) == s
157. from hypothesis import given from hypothesis.strategies import text @given(text()) def test_decode_inverts_encode(s): assert decode(encode(s)) == s rerun for different values
158. from hypothesis.extra.django.models import models from hypothesis.strategies import integers models(Customer).example() models(Customer, age=integers( min_value=0, max_value=120) ).example()
159. from hypothesis.extra.django.models import models from hypothesis.strategies import integers models(Customer).example() models(Customer, age=integers( min_value=0, max_value=120) ).example()
160. from hypothesis.extra.django import TestCase from hypothesis import given from hypothesis.extra.django.models import models from hypothesis.strategies import lists, integers class TestProjectManagement(TestCase): @given( models(Project, collaborator_limit=integers(min_value=0, max_value=20)), lists(models(User), max_size=20)) def test_can_add_users_up_to_collaborator_limit(self, project, collaborators): for c in collaborators: if project.at_collaboration_limit(): with self.assertRaises(LimitReached): project.add_user(c) self.assertFalse(project.team_contains(c)) else: project.add_user(c) self.assertTrue(project.team_contains(c))
161. from hypothesis.extra.django import TestCase from hypothesis import given from hypothesis.extra.django.models import models from hypothesis.strategies import lists, integers class TestProjectManagement(TestCase): @given( models(Project, collaborator_limit=integers(min_value=0, max_value=20)), lists(models(User), max_size=20)) def test_can_add_users_up_to_collaborator_limit(self, project, collaborators): for c in collaborators: if project.at_collaboration_limit(): with self.assertRaises(LimitReached): project.add_user(c) self.assertFalse(project.team_contains(c)) else: project.add_user(c) self.assertTrue(project.team_contains(c))
162. class TestProjectManagement(TestCase): @given( models(Project, collaborator_limit=integers(min_value=0, max_value=20)), lists(models(User), max_size=20)) def test_can_add_users_up_to_collaborator_limit(self, project, collaborators): for c in collaborators: if project.at_collaboration_limit(): with self.assertRaises(LimitReached): project.add_user(c) self.assertFalse(project.team_contains(c)) else: project.add_user(c) self.assertTrue(project.team_contains(c))
163. class TestProjectManagement(TestCase): @given( models(Project, collaborator_limit=integers(min_value=0, max_value=20)), lists(models(User), max_size=20)) def test_can_add_users_up_to_collaborator_limit(self, project, collaborators): for c in collaborators: if project.at_collaboration_limit(): with self.assertRaises(LimitReached): project.add_user(c) self.assertFalse(project.team_contains(c)) else: project.add_user(c) self.assertTrue(project.team_contains(c))
164. class TestProjectManagement(TestCase): @given( models(Project, collaborator_limit=integers(min_value=0, max_value=20)), lists(models(User), max_size=20)) def test_can_add_users_up_to_collaborator_limit(self, project, collaborators): for c in collaborators: if project.at_collaboration_limit(): with self.assertRaises(LimitReached): project.add_user(c) self.assertFalse(project.team_contains(c)) else: project.add_user(c) self.assertTrue(project.team_contains(c))
165. class TestProjectManagement(TestCase): @given( models(Project, collaborator_limit=integers(min_value=0, max_value=20)), lists(models(User), max_size=20)) def test_can_add_users_up_to_collaborator_limit(self, project, collaborators): for c in collaborators: if project.at_collaboration_limit(): with self.assertRaises(LimitReached): project.add_user(c) self.assertFalse(project.team_contains(c)) else: project.add_user(c) self.assertTrue(project.team_contains(c))
166. Falsifying example: test_can_add_users_up_to_collaborator_limit( self=TestProjectManagement(), project=Project('', 1), collaborators=[ User(.@.com), User(.@.com) ] ) Traceback (most recent call last): ... raise LimitReached() manager.models.LimitReached
167. Property based testing is more complicated, yet more valuable
168. Django test code coverage is 76%
169. Deceptive metric
170. High coverage != high quality
171. Mutation testing available Python implementation: mutpy
172. mut.py --target node --unit-test test_node target unit test
173. if foo and bar: do_this() target unit test
174. if foo and bar: do_this() if foo or bar: do_this() mutant unit test
175. mutant killed
176. mutant killed mutant survived
177. AOR - arithmetic operator replacement BCR - break continue replacement COI - conditional operator insertion CRP - constant replacement DDL - decorator deletion LOR - logical operator replacement
178. [*] Start mutation process: - targets: django.utils.encoding - tests: tests.utils_tests.test_encoding [*] 10 tests passed: - tests.utils_tests.test_encoding [0.00533 s] [*] Start mutants generation and execution: ... [*] Mutation score [12.19066 s]: 32.1% - all: 88 - killed: 24 (27.3%) - survived: 55 (62.5%) - incompetent: 7 (8.0%) - timeout: 2 (2.3%)
179. [*] Start mutation process: - targets: django.utils.encoding - tests: tests.utils_tests.test_encoding [*] 10 tests passed: - tests.utils_tests.test_encoding [0.00533 s] [*] Start mutants generation and execution: ... [*] Mutation score [12.19066 s]: 32.1% - all: 88 - killed: 24 (27.3%) - survived: 55 (62.5%) - incompetent: 7 (8.0%) - timeout: 2 (2.3%)
180. Django duration utils mutation score is 89%
181. Django duration utils mutation score is 89% and the coverage is 91%
182. Django encoding utils mutation score is 32%
183. Django encoding utils mutation score is 32% while the coverage is 63%
184. django Testing tutorial
185. my_app ├── __init__.py ├── admin.py ├── migrations │ └── __init__.py ├── models.py ├── tests.py └── views.py
186. my_app ├── __init__.py ├── admin.py ├── migrations │ └── __init__.py ├── models.py ├── tests.py └── views.py
187. When testing, more is better
188. time # of tests
189. # of tests total execution time
190. slow tests, slow feedback loop
191. 8 tips on how to speed up your tests
192. 8 tips on how to speed up your tests #5 will shock you
194. 1. use MD5PasswordHasher
195. 1. use MD5PasswordHasher 2. consider in-memory sqlite3
196. 1. use MD5PasswordHasher 2. consider in-memory sqlite3 3. have more SimpleTestCase
197. 1. use MD5PasswordHasher 2. consider in-memory sqlite3 3. have more SimpleTestCase 4. use setUpTestData()
198. 1. use MD5PasswordHasher 2. consider in-memory sqlite3 3. have more SimpleTestCase 4. use setUpTestData() 5. use mocks EVERYWHERE
199. 1. use MD5PasswordHasher 2. consider in-memory sqlite3 3. have more SimpleTestCase 4. use setUpTestData() 5. use mocks EVERYWHERE
200. 1. use MD5PasswordHasher 2. consider in-memory sqlite3 3. have more SimpleTestCase 4. use setUpTestData() 5. use mocks EVERYWHERE 6. be vigilant of what gets created in setUp()
201. 1. use MD5PasswordHasher 2. consider in-memory sqlite3 3. have more SimpleTestCase 4. use setUpTestData() 5. use mocks EVERYWHERE 6. be vigilant of what gets created in setUp() 7. don’t save model objects if not necessary
202. 1. use MD5PasswordHasher 2. consider in-memory sqlite3 3. have more SimpleTestCase 4. use setUpTestData() 5. use mocks EVERYWHERE 6. be vigilant of what gets created in setUp() 7. don’t save model objects if not necessary 8. isolate unit tests
203. 1. use MD5PasswordHasher 2. consider in-memory sqlite3 3. have more SimpleTestCase 4. use setUpTestData() 5. use mocks EVERYWHERE 6. be vigilant of what gets created in setUp() 7. don’t save model objects if not necessary 8. isolate unit tests
204. 1. use MD5PasswordHasher 2. consider in-memory sqlite3 3. have more SimpleTestCase 4. use setUpTestData() 5. use mocks EVERYWHERE 6. be vigilant of what gets created in setUp() 7. don’t save model objects if not necessary 8. isolate unit tests
205. 1. use MD5PasswordHasher 2. consider in-memory sqlite3 3. have more SimpleTestCase 4. use setUpTestData() 5. use mocks EVERYWHERE 6. be vigilant of what gets created in setUp() 7. don’t save model objects if not necessary 8. isolate unit tests
206. class SimpleTest(TestCase): def setUp(self): for _ in range(10): Robot.objects.create() 6. be vigilant of what gets created in setUp()
207. class SimpleTest(TestCase): def setUp(self): for _ in range(10): Robot.objects.create() 6. be vigilant of what gets created in setUp()
208. 7. don’t save model objects if not necessary
209. 7. don’t save model objects if not necessary Robot.objects.create() instead of
210. 7. don’t save model objects if not necessary Robot.objects.create() Robot() instead of maybe do
211. 7. don’t save model objects if not necessary Robot.objects.create() Robot() RobotFactory.build() instead of maybe do or
212. 7. don’t save model objects if not necessary Robot.objects.create() Robot() RobotFactory.build() RobotFactory.stub() instead of maybe do or or
213. 7. don’t save model objects if not necessary Robot.objects.create() Robot() RobotFactory.build() RobotFactory.stub() factory boy instead of maybe do or or }
214. 8. isolate unit tests
215. 8. isolate unit tests
216. 8. isolate unit tests
217. 8. isolate unit tests unit tests functional tests
218. 8. isolate unit tests unit tests functional tests ./manage.py test --tag=unit
220. Ugh, taxes!
221. Product ProductQuestionnaire name ingredients price product category drink_category food_category has_decorations is_coated_in_chocolate is_warm is_cold
223. Production code Test
224. class ProductQuestionnaireCreate(CreateView): def form_valid(self, form): if is_biscuit and is_coated_in_chocolate: set_vat_20() return super().form_valid(form) class ProductQuestionnaireCreateTestCase(TestCase): def test_20p_vat_if_coated_in_chocolate_biscuit(self): product = ProductFactory() response = self.client.post(self.url, {'q1': 'a1', 'q2': 'a2'}) product.refresh_from_db() self.assertEqual(product.vat, 20) take 1
225. class ProductQuestionnaireCreate(CreateView): def form_valid(self, form): if is_biscuit and is_coated_in_chocolate: set_vat_20() return super().form_valid(form) class ProductQuestionnaireCreateTestCase(TestCase): def test_20p_vat_if_coated_in_chocolate_biscuit(self): product = ProductFactory() response = self.client.post(self.url, {'q1': 'a1', 'q2': 'a2'}) product.refresh_from_db() self.assertEqual(product.vat, 20) take 1
226. class ProductQuestionnaireCreate(CreateView): def form_valid(self, form): return super().form_valid(form)
227. class ProductQuestionnaireCreateTestCase(TestCase): def test_20p_vat_if_coated_in_chocolate_biscuit(self): def test_0p_vat_if_baguette(self): def test_0p_vat_if_flapjack(self): def test_20p_vat_if_cereal_bar(self):
228. To test if I need to pay 20% VAT for biscuits coated in chocolate, I need to: go through the router interact with database send input to receive output
229. Mocks are not a solution
230. class ProductQuestionnaireForm(forms.ModelForm): def save(self, commit=True): instance = super().save(commit) if is_biscuit and is_coated_in_chocolate: set_vat_20() return instance take 2 class ProductQuestionnaireFormTestCase(TestCase): def test_20p_vat_if_coated_in_chocolate_biscuit(self): product = ProductFactory() form = ProductQuestionnaireForm(data={'k1': 'v1', 'k2': 'v2'}) self.assertTrue(form.is_valid()) form.save() product.refresh_from_db() self.assertEqual(product.vat, 20)
231. class ProductQuestionnaireForm(forms.ModelForm): def save(self, commit=True): instance = super().save(commit) if is_biscuit and is_coated_in_chocolate: set_vat_20() return instance take 2 class ProductQuestionnaireFormTestCase(TestCase): def test_20p_vat_if_coated_in_chocolate_biscuit(self): product = ProductFactory() form = ProductQuestionnaireForm(data={'k1': 'v1', 'k2': 'v2'}) self.assertTrue(form.is_valid()) form.save() product.refresh_from_db() self.assertEqual(product.vat, 20)
232. To test if I need to pay 20% VAT for biscuits coated in chocolate, I need to: go through the router interact with database send input to receive output
233. take 3 class VATCalculator(object): def calculate_vat(self, **kwargs): if is_biscuit and is_coated_in_chocolate: return 20 class VATCalculatorTestCase(SimpleTestCase): def test_20p_vat_if_coated_in_chocolate_biscuit(self): calc = VATCalculator() self.assertEqual(calc.calculate_vat( is_biscuit=True, is_coated_in_choco=True ))
234. reusability testability extensibility
235. To test if I need to pay 20% VAT for biscuits coated in chocolate, I need to: go through the router interact with database send input to receive output
237. Tests have more in them than we think
238. Happy testing and big bear hug thanks!