Ana Balica - To mock, or not to mock, that is the question

Mocking is a very powerful testing concept that has some dangerous pitfalls. There are obvious use cases where mocks are an absolute requirement to be able to test a part of the app. Nevertheless sometimes apparently useful mocks can yield erroneous test results. This talk goes into deeper detail on the trade-off of using mocks in testing.

https://us.pycon.org/2016/schedule/presentation/2014/

Transcript

1. T o
 
 or not to
 
 that is the question MO CK MO CK
2. T o
 
 or not to
 
 that is the question MO CK MO CK
3. @anabalica a treatise narrated by
 
 from Potato of Londontowne. ANA BALICA
4. Thou 
 shalt 
 write 
 tests. “ ” William Shakespeare
5. Chapter one
6. Mocks simulate
 the looks and behaviour 
 of real objects
7. REAL fig.1 fig.2 MOCK
8. mocks != stubs
9. ✓ Setup ✓ Test ✓ Verify state ✓ Teardown ✓ Setup ✓ Setup expectations ✓ Test ✓ Verify expectations ✓ Verify state ✓ Teardown Stubs Mocks
10. unittest.mock # Python 3.3 mock # Python 2.x
11. Mock()
12. Mock >>> from mock import Mock >>> m = Mock() >>> m.foo = 1 >>> m.foo 1 >>> m.bar <Mock name='mock.bar' id='4310136016'>
13. Mock() MagicMock()
14. __lt__ __gt__ __len__ __iter__ __bool__ __str__ __int__ __hash__ __exit__ __sizeof__
15. Mock() MagicMock() patch()
16. from mock import patch with patch('rainbow.Pony') as MockPony: MockPony.return_value = 42 Patching
17. Patching 'rainbow.Pony' # creatures.py class Pony: pass # rainbow.py from creatures import Pony pony = Pony()
18. Mock the object where it’s used, 
 not where it came from
19. Chapter two
20. Good mocks
21. with patch.dict('os.environ', {'ANDROID_ARGUMENT': ''}): pf = Platform() self.assertTrue(pf == ‘android') os.environ System calls
22. @mock.patch('sys.stdout', new_callable=six.StringIO) def test_print_live_refs_empty(self, stdout): trackref.print_live_refs() self.assertEqual(stdout.getvalue(), 'Live References\n\n\n') sys.stdout Streams
23. request.urlopen @patch('django.utils.six.moves.urllib.request.urlopen') def test_oembed_photo_request(self, urlopen): urlopen.return_value = self.dummy_response result = wagtail_oembed("http://www.youtube.com/watch/") self.assertEqual(result['type'], 'photo') Networking
24. @patch.object(DataLoader, '_get_file_contents') def test_parse_json_from_file(self, mock_def): mock_def.return_value = ('{"a": 1, "b": 2, "c": 3}', True) output = self._loader.load_from_file('dummy_json.txt') self.assertEqual(output, dict(a=1, b=2, c=3)) json.loads IO operations
25. @mock.patch('time.sleep') def test_500_retry(self, sleep_mock): self.set_http_response(status_code=500) # Create a bucket, a key and a file with self.assertRaises(BotoServerError): k.send_file(fail_file) Clocks, time, timezones time.sleep
26. with mock.patch('random.random', return_value=0.0): with self.assertChanges(get_timeline_size, before=10, after=5): backend.add(timeline, next(self.records)) random.random Unpredictable results
27. ✓ System calls ✓ Streams ✓ Networking ✓ IO operations ✓ Clocks, time, timezones ✓ Unpredictable results
28. ✓ Save time ✓ Make impossible possible ✓ Exclude external dependencies Why we like them
29. Chapter three
30. mocks Bad
31. with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertTrue(saved_pony.age, 3) mock_save.assert_called_once() Problems?
32. with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertTrue(saved_pony.age, 3) mock_save.assert_called_once() Problems?
33. with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.assert_called_once() Problems?
34. with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.assert_called_once() Problems?
35. with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.assert_called_twice() Problems?
36. with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.make_me_sandwich() Problems?
37. ¯\_(ϑ)_/¯
38. Solution 1 with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.assert_called_once_with()
39. Solution 2 with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) self.assertEqual(mock_save.call_count, 1)
40. Failure with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) self.assertEqual(mock_save.sandwich_count, 1)
41. Solution 3 Test Driven Development
42. Problems? with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) self.assertEqual(mock_save.call_count, 1)
43. Tests pass? SHIP IT!
44. Maybe 
 it’s incomplete?
46. Integration tests c = Client() response = c.post('/pony/', {'age': 1}) self.assertEqual(response.status_code, 201)
47. Unit tests Integration tests Mocks
48. Only mock types 
 that you own
49. Building on
 Third-Party
 Code
50. Adapter
 layer 3rd party API Application
 objects
51. Adapter
 layer 3rd party API Application
 objects Test this cluster Mock this
52. Conclusions
53. Mocks
 can be
 dangerous
54. Passing faulty tests give a 
 false sense of security
55. The end