Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mock considered harmful

410e3353165c33043ab69be7fc366428?s=47 Boris Feld
October 18, 2015

Mock considered harmful

Why mock are often considered harmful and lead to bad test quality ? Is there other ways to test code without mocks?

410e3353165c33043ab69be7fc366428?s=128

Boris Feld

October 18, 2015
Tweet

Transcript

  1. Mock considered harmful Boris FELD - Pycon FR 2015

  2. Let’s test def send_billing_email(): last_month = datetime.date.today() - datetime.timedelta(month=1) for

    user in USERS.find(is_active=True, last_paid__lt=last_month): UserEmailer.send(user, 'billing_reminder')
  3. Let’s unit test it! Test UserEmailer Users • Faster to

    run • Easier to write • Devs are eager to write tests
  4. How to test it? class TestBillingEmail(unittest.TestCase): @patch('code.USERS') @patch('code.UserEmailer') def test_send_billing_email(self,

    user_email_patch, users_patch): users = [sentinel.USER1, sentinel.USER2] users_patch.find.return_value = users code.send_billing_email() last_month = datetime.date.today() - datetime.timedelta(days=30) self.assertEqual(users_patch.find.call_args_list, [call(is_active=True, last_paid__lt=last_month)]) self.assertEqual(user_email_patch.send.call_args_list, [call(users[0], 'billing_reminder'), call(users[1], 'billing_reminder')])
  5. Good? • Fast to run ✓ • Easy to write

    ✓ • Devs are eager to write tests ✓
  6. All good • Add more tests • Add more code

    • Enjoy!
  7. Start refactoring • Change parameters order • Forbid a parameter

    value • Change output format • Add a new parameter • Add a new parameter with default value • Rename method
  8. Bug in production • But it’s tested, right?! • Argh,

    f*cking Mock hides the bug
  9. Why?

  10. Take a look back class TestBillingEmail(unittest.TestCase): @patch('code.USERS') @patch('code.UserEmailer') def test_send_billing_email(self,

    user_email_patch, users_patch): users = [sentinel.USER1, sentinel.USER2] users_patch.find.return_value = users code.send_billing_email() last_month = datetime.date.today() - datetime.timedelta(days=30) self.assertEqual(users_patch.find.call_args_list, [call(is_active=True, last_paid__lt=last_month)]) self.assertEqual(user_email_patch.send.call_args_list, [call(users[0], 'billing_reminder'), call(users[1], 'billing_reminder')])
  11. Coupling

  12. Test design • Test should not break if: • USERS

    changes its API • UserEmailer changes its API • => Change detector tests
  13. Mock considered harmful

  14. The big lie

  15. Mock utilizations considered harmful

  16. - Wikipedia « In a unit test, mock objects can

    simulate the behavior of complex, real objects and are therefore useful when a real object is impractical or impossible to incorporate into a unit test. »
  17. Four common usage of Mock • Test internal implementation, Cache

    system. • Simulate some weird errors, network errors. • Reduce randomness. • « Light-weight » test double for DB, Network, Disk.
  18. With great powers…

  19. Side-effect • As it test behaviors, it couples the test

    to the mocked object contract. • The test will fail every time the contract is changed. • The test may pass while real code breaks due to refactoring. • « White-box » testing with monkey patching.
  20. White-box testing Test Inputs White box testing Test Outputs

  21. Black box testing Test Inputs White box testing Test Outputs

    Black box testing
  22. Ideal black-box test • For a given « SUT »,

    given a set of input, we could assert the state of outputs. • => Functional programming
  23. How to test it then? • Two patterns: • Dependency

    Injection • Interface Testing
  24. - James Shore « Dependency injection means giving an object

    its instance variables. Really. That's it. »
  25. Let’s try again def send_billing_email(user_model=USERS, user_emailer=UserEmailer): last_month = datetime.date.today() -

    datetime.timedelta(days=30) for user in user_model.find(is_active=True, last_paid__lt=last_month): user_emailer.send(user, 'billing_reminder') class TestBillingEmail(unittest.TestCase): def test_send_billing_email(self): users_mock = Mock() users = [sentinel.USER1, sentinel.USER2] users_mock.find.return_value = users user_emailer_mock = Mock() code.send_billing_email(users_mock, user_emailer_mock) last_month = datetime.date.today() - datetime.timedelta(days=30) self.assertEqual(users_mock.find.call_args_list, [call(is_active=True, last_paid__lt=last_month)]) self.assertEqual(user_emailer_mock.send.call_args_list, [call(users[0], 'billing_reminder'), call(users[1], 'billing_reminder')])
  26. A little better

  27. Design • The function need to get back filtered Users:

    • And send an Email to each user: user_model.find(is_active=True, last_paid__lt=last_month) user_emailer.send(user, 'billing_reminder')
  28. Contract based dependencies • Think about replaceable dependencies • If

    it implements the contract… • We can interchange them • As if they were plugin or micro-services
  29. Create a test per interface class _BaseUserModelTestCase(unittest.TestCase): def setUp(self): self.model

    = self.get_model() def test_insert(self): user = {'username': 'bfe'} self.model.insert(user) self.assertEqual(self.model.list(), [user]) def test_filter(self): users = [{'username': 'bfe', 'post': 42}, {'username': 'ohe', 'post': 25}] self.model.insert(users[0]) self.model.insert(users[1]) self.assertEqual(self.model.list(), users) self.assertEqual(self.model.find(username='bfe'), users[0]) self.assertEqual(self.model.find(post__lt=40), users[1])
  30. Then a test per implementation class DBUserModelTestCase(_BaseUserModelTestCase): def get_model(self): return

    USERS class StubUserModelTestCase(_BaseUserModelTestCase): def get_model(self): return StubUserModel()
  31. For reference, test implementation class StubUserModel(object): OPERATORS = { None:

    operator.eq, 'lt': operator.lt } def __init__(self): self.users = [] def insert(self, user): self.users.append(user) def list(self): return self.users def find(self, **kwargs): matching_users = [] for user in self.users: matching = True for kwarg, value in kwargs.items(): field = kwarg.split('__') if len(field) == 1: operator = None else: operator = field[1] if not self.OPERATORS[operator](user.get(field[0]), value): matching = False if matching: matching_users.append(user) return matching_users
  32. And emailer class _BaseUserEmailerTestCase(unittest.TestCase): def setUp(self): self.user_emailer = self.get_user_emailer() def

    test_send(self): user = {'username': 'bfe'} topic = 'test_topic' self.user_emailer(user, topic) self.assertMailSent(user, topic) self.assertNotMailSent(user, topic)
  33. Implementations tests class ImapUserEmailerTestCase(_BaseUserEmailerTestCase): def get_user_emailer(self): return UserEmailer(settings.LOCAL_IMAP_SERVER) class TestUserEmailerTestCase(_BaseUserEmailerTestCase):

    def get_user_emailer(self): return TestUserEmailer()
  34. Test double implementation class TestUserEmailer(object): def __init__(self): self.sent_emails = []

    def send(self, user, topic): self.sent_emails.append((user, topic)) def assertMailSent(self, user, topic): return (user, topic) in self.sent_emails def assertNotMailSent(self, user, topic): return not self.assertMailSent(user, topic)
  35. And now? class TestBillingEmail(unittest.TestCase): def test_send_billing_email(self): model = StubUserModel() emailer

    = TestUserEmailer() # Fixtures one_month_ago = date.today() - timedelta(days=30) valid_user = {'username': 'bfe', 'is_active': True, 'last_paid': one_month_ago + timedelta(days=2)} inactive_user = {'username': 'foo', 'is_active': False, 'last_paid': one_month_ago - timedelta(days=2)} bad_payer_user = {'username': 'foo', 'is_active': True, 'last_paid': one_month_ago - timedelta(days=2)} for u in [valid_user, inactive_user, bad_payer_user]: model.insert(u) code.send_billing_email(model, emailer) emailer.assertMailSent(bad_payer_user, 'billing_reminder')
  36. Result • Less coupled to implementation • More friendly to

    refactoring • More lines of tests
  37. Both models are worth it • Different needs • Mocks

    are cheaper • But stub requires less maintenance
  38. Different design approachs • Mock is good for top-down approach

    • London-Style, mockist-style • Stubs is good for bottom-up approach • Chicago school, classic TDD • Both are complementary
  39. Reflexion • « Interface » testing is good for design

    reflexion, how I want to call it and how the state is changed? • Helps to have modular design and clear contract.
  40. Intermediary • Code call a proxy: • Which will forward

    to a real object for production • Or a STUB in tests • Ex: Django email backends / password backends • => django-nopassword
  41. Thank you ! • http://googletesting.blogspot.fr/2015/01/testing- on-toilet-change-detector-tests.html