Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[PyCon US 2016] To mock or not to mock, that is the question

[PyCon US 2016] To mock or not to mock, that is the question

Mocking is a very powerful testing concept that has some dangerous pitfalls. There are obvious use cases where mocks are an absolute requirement to be able to test a part of the app. Nevertheless sometimes apparently useful mocks can yield erroneous test results. This talk goes into deeper detail on the trade-off of using mocks in testing.

2a3082799c3df9a58d06bc1b81107752?s=128

Ana Balica

May 30, 2016
Tweet

Transcript

  1. T o
 
 or not to
 
 that is the

    question MO CK MO CK
  2. @anabalica a treatise narrated by
 
 from Potato of Londontowne.

    ANA BALICA
  3. Thou 
 shalt 
 write 
 tests. “ ” William

    Shakespeare
  4. Chapter one

  5. Mocks simulate
 the looks and behaviour 
 of real objects

  6. REAL fig.1 fig.2 MOCK

  7. mocks != stubs

  8. ✓ Setup ✓ Test ✓ Verify state ✓ Teardown ✓

    Setup ✓ Setup expectations ✓ Test ✓ Verify expectations ✓ Verify state ✓ Teardown Stubs Mocks
  9. unittest.mock # Python 3.3 mock # Python 2.x

  10. Mock()

  11. Mock >>> from mock import Mock >>> m = Mock()

    >>> m.foo = 1 >>> m.foo 1 >>> m.bar <Mock name='mock.bar' id='4310136016'>
  12. Mock() MagicMock()

  13. __lt__ __gt__ __len__ __iter__ __bool__ __str__ __int__ __hash__ __exit__ __sizeof__

  14. Mock() MagicMock() patch()

  15. from mock import patch with patch('rainbow.Pony') as MockPony: MockPony.return_value =

    42 Patching
  16. Patching 'rainbow.Pony' # creatures.py class Pony: pass # rainbow.py from

    creatures import Pony pony = Pony()
  17. Mock the object where it’s used, 
 not where it

    came from
  18. Chapter two

  19. Good mocks

  20. with patch.dict('os.environ', {'ANDROID_ARGUMENT': ''}): pf = Platform() self.assertTrue(pf == ‘android')

    os.environ System calls
  21. @mock.patch('sys.stdout', new_callable=six.StringIO) def test_print_live_refs_empty(self, stdout): trackref.print_live_refs() self.assertEqual(stdout.getvalue(), 'Live References\n\n\n') sys.stdout

    Streams
  22. request.urlopen @patch('django.utils.six.moves.urllib.request.urlopen') def test_oembed_photo_request(self, urlopen): urlopen.return_value = self.dummy_response result =

    wagtail_oembed("http://www.youtube.com/watch/") self.assertEqual(result['type'], 'photo') Networking
  23. @patch.object(DataLoader, '_get_file_contents') def test_parse_json_from_file(self, mock_def): mock_def.return_value = ('{"a": 1, "b":

    2, "c": 3}', True) output = self._loader.load_from_file('dummy_json.txt') self.assertEqual(output, dict(a=1, b=2, c=3)) json.loads IO operations
  24. @mock.patch('time.sleep') def test_500_retry(self, sleep_mock): self.set_http_response(status_code=500) # Create a bucket, a

    key and a file with self.assertRaises(BotoServerError): k.send_file(fail_file) Clocks, time, timezones time.sleep
  25. with mock.patch('random.random', return_value=0.0): with self.assertChanges(get_timeline_size, before=10, after=5): backend.add(timeline, next(self.records)) random.random

    Unpredictable results
  26. ✓ System calls ✓ Streams ✓ Networking ✓ IO operations

    ✓ Clocks, time, timezones ✓ Unpredictable results
  27. ✓ Save time ✓ Make impossible possible ✓ Exclude external

    dependencies Why we like them
  28. Chapter three

  29. mocks Bad

  30. with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid())

    saved_pony = form.save() self.assertTrue(saved_pony.age, 3) mock_save.assert_called_once() Problems?
  31. with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid())

    saved_pony = form.save() self.assertTrue(saved_pony.age, 3) mock_save.assert_called_once() Problems?
  32. with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid())

    saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.assert_called_once() Problems?
  33. with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid())

    saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.assert_called_once() Problems?
  34. with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid())

    saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.assert_called_twice() Problems?
  35. with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid())

    saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.make_me_sandwich() Problems?
  36. ¯\_(ϑ)_/¯

  37. Solution 1 with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony,

    data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.assert_called_once_with()
  38. Solution 2 with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony,

    data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) self.assertEqual(mock_save.call_count, 1)
  39. Failure with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data)

    self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) self.assertEqual(mock_save.sandwich_count, 1)
  40. Solution 3 Test Driven Development

  41. Problems? with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data)

    self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) self.assertEqual(mock_save.call_count, 1)
  42. Tests pass? SHIP IT!

  43. Maybe 
 it’s incomplete?

  44. None
  45. Integration tests c = Client() response = c.post('/pony/', {'age': 1})

    self.assertEqual(response.status_code, 201)
  46. Unit tests Integration tests Mocks

  47. Only mock types 
 that you own

  48. Building on
 Third-Party
 Code

  49. Adapter
 layer 3rd party API Application
 objects

  50. Adapter
 layer 3rd party API Application
 objects Test this cluster

    Mock this
  51. Conclusions

  52. Mocks
 can be
 dangerous

  53. Passing faulty tests give a 
 false sense of security

  54. The end