Mock considered harmful

Why mock are often considered harmful and lead to bad test quality ? Is there other ways to test code without mocks?

Transcript

1. Mock considered harmful Boris FELD - Pycon FR 2015
2. Let’s test def send_billing_email(): last_month = datetime.date.today() - datetime.timedelta(month=1) for user in USERS.find(is_active=True, last_paid__lt=last_month): UserEmailer.send(user, 'billing_reminder')
3. Let’s unit test it! Test UserEmailer Users • Faster to run • Easier to write • Devs are eager to write tests
4. How to test it? class TestBillingEmail(unittest.TestCase): @patch('code.USERS') @patch('code.UserEmailer') def test_send_billing_email(self, user_email_patch, users_patch): users = [sentinel.USER1, sentinel.USER2] users_patch.find.return_value = users code.send_billing_email() last_month = datetime.date.today() - datetime.timedelta(days=30) self.assertEqual(users_patch.find.call_args_list, [call(is_active=True, last_paid__lt=last_month)]) self.assertEqual(user_email_patch.send.call_args_list, [call(users[0], 'billing_reminder'), call(users[1], 'billing_reminder')])
5. Good? • Fast to run ✓ • Easy to write ✓ • Devs are eager to write tests ✓
6. All good • Add more tests • Add more code • Enjoy!
7. Start refactoring • Change parameters order • Forbid a parameter value • Change output format • Add a new parameter • Add a new parameter with default value • Rename method
8. Bug in production • But it’s tested, right?! • Argh, f*cking Mock hides the bug
9. Why?
10. Take a look back class TestBillingEmail(unittest.TestCase): @patch('code.USERS') @patch('code.UserEmailer') def test_send_billing_email(self, user_email_patch, users_patch): users = [sentinel.USER1, sentinel.USER2] users_patch.find.return_value = users code.send_billing_email() last_month = datetime.date.today() - datetime.timedelta(days=30) self.assertEqual(users_patch.find.call_args_list, [call(is_active=True, last_paid__lt=last_month)]) self.assertEqual(user_email_patch.send.call_args_list, [call(users[0], 'billing_reminder'), call(users[1], 'billing_reminder')])
11. Coupling
12. Test design • Test should not break if: • USERS changes its API • UserEmailer changes its API • => Change detector tests
13. Mock considered harmful
14. The big lie
15. Mock utilizations considered harmful
16. - Wikipedia « In a unit test, mock objects can simulate the behavior of complex, real objects and are therefore useful when a real object is impractical or impossible to incorporate into a unit test. »
17. Four common usage of Mock • Test internal implementation, Cache system. • Simulate some weird errors, network errors. • Reduce randomness. • « Light-weight » test double for DB, Network, Disk.
18. With great powers…
19. Side-effect • As it test behaviors, it couples the test to the mocked object contract. • The test will fail every time the contract is changed. • The test may pass while real code breaks due to refactoring. • « White-box » testing with monkey patching.
20. White-box testing Test Inputs White box testing Test Outputs
21. Black box testing Test Inputs White box testing Test Outputs Black box testing
22. Ideal black-box test • For a given « SUT », given a set of input, we could assert the state of outputs. • => Functional programming
23. How to test it then? • Two patterns: • Dependency Injection • Interface Testing
24. - James Shore « Dependency injection means giving an object its instance variables. Really. That's it. »
25. Let’s try again def send_billing_email(user_model=USERS, user_emailer=UserEmailer): last_month = datetime.date.today() - datetime.timedelta(days=30) for user in user_model.find(is_active=True, last_paid__lt=last_month): user_emailer.send(user, 'billing_reminder') class TestBillingEmail(unittest.TestCase): def test_send_billing_email(self): users_mock = Mock() users = [sentinel.USER1, sentinel.USER2] users_mock.find.return_value = users user_emailer_mock = Mock() code.send_billing_email(users_mock, user_emailer_mock) last_month = datetime.date.today() - datetime.timedelta(days=30) self.assertEqual(users_mock.find.call_args_list, [call(is_active=True, last_paid__lt=last_month)]) self.assertEqual(user_emailer_mock.send.call_args_list, [call(users[0], 'billing_reminder'), call(users[1], 'billing_reminder')])
26. A little better
27. Design • The function need to get back filtered Users: • And send an Email to each user: user_model.find(is_active=True, last_paid__lt=last_month) user_emailer.send(user, 'billing_reminder')
28. Contract based dependencies • Think about replaceable dependencies • If it implements the contract… • We can interchange them • As if they were plugin or micro-services
29. Create a test per interface class _BaseUserModelTestCase(unittest.TestCase): def setUp(self): self.model = self.get_model() def test_insert(self): user = {'username': 'bfe'} self.model.insert(user) self.assertEqual(self.model.list(), [user]) def test_filter(self): users = [{'username': 'bfe', 'post': 42}, {'username': 'ohe', 'post': 25}] self.model.insert(users[0]) self.model.insert(users[1]) self.assertEqual(self.model.list(), users) self.assertEqual(self.model.find(username='bfe'), users[0]) self.assertEqual(self.model.find(post__lt=40), users[1])
30. Then a test per implementation class DBUserModelTestCase(_BaseUserModelTestCase): def get_model(self): return USERS class StubUserModelTestCase(_BaseUserModelTestCase): def get_model(self): return StubUserModel()
31. For reference, test implementation class StubUserModel(object): OPERATORS = { None: operator.eq, 'lt': operator.lt } def __init__(self): self.users = [] def insert(self, user): self.users.append(user) def list(self): return self.users def find(self, **kwargs): matching_users = [] for user in self.users: matching = True for kwarg, value in kwargs.items(): field = kwarg.split('__') if len(field) == 1: operator = None else: operator = field[1] if not self.OPERATORS[operator](user.get(field[0]), value): matching = False if matching: matching_users.append(user) return matching_users
32. And emailer class _BaseUserEmailerTestCase(unittest.TestCase): def setUp(self): self.user_emailer = self.get_user_emailer() def test_send(self): user = {'username': 'bfe'} topic = 'test_topic' self.user_emailer(user, topic) self.assertMailSent(user, topic) self.assertNotMailSent(user, topic)
33. Implementations tests class ImapUserEmailerTestCase(_BaseUserEmailerTestCase): def get_user_emailer(self): return UserEmailer(settings.LOCAL_IMAP_SERVER) class TestUserEmailerTestCase(_BaseUserEmailerTestCase): def get_user_emailer(self): return TestUserEmailer()
34. Test double implementation class TestUserEmailer(object): def __init__(self): self.sent_emails = [] def send(self, user, topic): self.sent_emails.append((user, topic)) def assertMailSent(self, user, topic): return (user, topic) in self.sent_emails def assertNotMailSent(self, user, topic): return not self.assertMailSent(user, topic)
35. And now? class TestBillingEmail(unittest.TestCase): def test_send_billing_email(self): model = StubUserModel() emailer = TestUserEmailer() # Fixtures one_month_ago = date.today() - timedelta(days=30) valid_user = {'username': 'bfe', 'is_active': True, 'last_paid': one_month_ago + timedelta(days=2)} inactive_user = {'username': 'foo', 'is_active': False, 'last_paid': one_month_ago - timedelta(days=2)} bad_payer_user = {'username': 'foo', 'is_active': True, 'last_paid': one_month_ago - timedelta(days=2)} for u in [valid_user, inactive_user, bad_payer_user]: model.insert(u) code.send_billing_email(model, emailer) emailer.assertMailSent(bad_payer_user, 'billing_reminder')
36. Result • Less coupled to implementation • More friendly to refactoring • More lines of tests
37. Both models are worth it • Different needs • Mocks are cheaper • But stub requires less maintenance
38. Different design approachs • Mock is good for top-down approach • London-Style, mockist-style • Stubs is good for bottom-up approach • Chicago school, classic TDD • Both are complementary
39. Reflexion • « Interface » testing is good for design reflexion, how I want to call it and how the state is changed? • Helps to have modular design and clear contract.
40. Intermediary • Code call a proxy: • Which will forward to a real object for production • Or a STUB in tests • Ex: Django email backends / password backends • => django-nopassword
41. Thank you ! • http://googletesting.blogspot.fr/2015/01/testing- on-toilet-change-detector-tests.html