Did your mocks read the Terms of Service

When writing unit tests, we are used to inject mocks instead of concrete implementations. The problem is that the mock behavior is not tied at all to the actual implementation, and so even though all unit tests are green, you can still have bugs inside your codebase.

But tests are meant to define the contract of the feature under test. It unambiguously verifies that the feature behaves as expected no matter what. So there must be a way to use the power of testing to ensure the mocks also follow the same contract.

This talk will introduce the concept of contract based testing, which allows describing mocks, and ensure that the concrete implementation actually behave the same way. It will then show examples design patterns and best practices on how this can be used to enhance the unit tests in a project.

Transcript

1. DID YOUR MOCKS READ THE TERMS OF SERVICE ? CONTRACTS FOR YOUR MOCKS AND FAKES CodeMobile UK 2018 1
2. ABOUT… XAVIER F. GOUCHET LEAD ANDROID ENGINEER AT WORKWELL FLUENT IN ANDROID SINCE CUPCAKE ON ALL SOCIAL NETWORKS @XGOUCHET 2
4. Previously, in CodeMobileUk 2017 … 4
7. OBJECTIVE MAKING SURE THE TOOLS WE USE ENHANCE OUR TESTS You must be as confident in the test you code as you are in the code you test. 7
8. PUT YOUR HANDS IN THE AIR 8
9. PUT YOUR HANDS IN THE AIR Unit tests ? 8.1
10. PUT YOUR HANDS IN THE AIR Unit tests ? Mocks Mockito / KotlinMockito? 8.2
11. PUT YOUR HANDS IN THE AIR Unit tests ? Mocks Mockito / KotlinMockito? Easymock ? 8.3
12. PUT YOUR HANDS IN THE AIR Unit tests ? Mocks Mockito / KotlinMockito? Easymock ? JMockit ? 8.4
13. PUT YOUR HANDS IN THE AIR Unit tests ? Mocks Mockito / KotlinMockito? Easymock ? JMockit ? Mockk ? 8.5
14. WHAT ARE WE TALKING ABOUT HERE MOCKS ? STUBS ? FAKES ? DUMMIES ? 9
15. FAKES (AKA DUMMIES) Dumb POJO / Data Class, actual value is not relevant @Test(expected = IndexOutOfBoundsException::class) fun testThrowsException() { val dummyUser = User(name = "Bob", id = 42) testedList.put(-666, dummy) } 10
16. STUBS / MOCKS Objects with scripted answers @Test(expected = IllegalArgumentException::class) fun testReturnsNull() { whenever(mockProvider.getData()) doReturn null testedObject.addDataFrom(mockProvider) } 11
17. SPIES Can be mock or real objects, just tracking each call @Test fun testCallsListener() { val spiedListener = spy(realListener) testedObject.addListener(spiedListener) testedObject.doSomething() verify(spiedListener).somethingDone() } 12
18. FAKES WHAT'S THE PROBLEM WITH FAKES 13
19. ALL MY TESTS INVOLVED USERS NAMED ALICE AND BOB. 14
20. ALL MY TESTS INVOLVED USERS NAMED ALICE AND BOB. AGED 42. 14.1
21. ALL MY TESTS INVOLVED USERS NAMED ALICE AND BOB. AGED 42. WORKING AT FOOBAR INC. 14.2
22. WHEN ALL YOUR FAKE DATA ARE “foo” AND 42, HOW CAN YOU BE SURE THAT YOUR TESTS ARE VALID ? 15
23. WHEN ALL YOUR FAKE DATA ARE “foo” AND 42, HOW CAN YOU BE SURE THAT YOUR TESTS ARE VALID ? Where does the `42` come from ? 15.1
24. WHEN ALL YOUR FAKE DATA ARE “foo” AND 42, HOW CAN YOU BE SURE THAT YOUR TESTS ARE VALID ? Where does the `42` come from ? Lots of hardcoded values ! 15.2
25. WHEN ALL YOUR FAKE DATA ARE “foo” AND 42, HOW CAN YOU BE SURE THAT YOUR TESTS ARE VALID ? Where does the `42` come from ? Lots of hardcoded values ! Only one value is tested. Ever. 15.3
26. SOLUTION ? 16
27. SOLUTION ? USE RANDOM DATA… 16.1
28. @Test fun invalidateDataAfterTimeout() { testedObject.setLastCallTimestamp(0) testedObject.setDataTTL(42) long ts = 666 boolean result = testedObject.isDataValid(ts) assertFalse(result) } 17
29. @Test fun invalidateDataAfterTimeout() { val fakeTS = rand.nextLong() val fakeTTL = rand.nextLong() val fakeDelay = rand.nextLong() testedObject.setLastCallTimestamp(fakeTS) testedObject.setDataTTL(fakeTTL) val ts = fakeTS + fakeTTL + fakeDelay val result = testedObject.isDataValid(ts) assertFalse(result) } 18
30. … BUT NOT TOO RANDOM (LONG OVERFLOW, NEGATIVE TTL, …) 19
31. INTRODUCING ELMYR class FooTest { @JvmField @Rule val forger = new JUnitForger() // … } 20
32. ELMYR FEATURES : FORGING NUMBERS // works with ints longs, floats and doubles forger.anInt(min = 0, max =100) forger.aPositiveLong() forger.aGaussianFloat(mean = 42.0f, standardDeviation = 100.0f) forger.aDoubleArray(DoubleConstraint.NEGATIVE_STRICT) 21
33. ELMYR FEATURES : FORGING STRINGS forger.anHexadecimalString() forger.aWord() forger.aSentence() forger.anEmail() forger.aUrl() forger.aStringArray(StringConstraint.WORD) forger.aStringMatching("(0|+44)\\d{10}") 22
34. ELMYR FEATURES : FORGING FROM COLLECTIONS, ENUM forger.anElementFrom(myCollection) forger.anElementFrom(value1, value2, value3, …) forger.aValueFrom(MyEnum::class) forger.aNullableFrom(nonNullValue) 23
35. ELMYR FEATURES : FAILING TESTS ‘testSomething’ failed with fake seed = 0x4815162342 @Before fun forceSeed() { forger.reset(4815162342L) } 24
36. @Test fun invalidateDataAfterTimeout(){ val fakeTS = forger.aTimestamp() val fakeTTL = forger.aLong(1, 86400000) val fakeDelay = forger.aLong(1, 86400000) testedObject.setLastCallTimestamp(fakeTS) testedObject.setDataTTL(fakeTTL) val ts = fakeTS + fakeTTL + fakeDelay val result = testedObject.isDataValid(ts) assertFalse(result) } 25
37. DEALING WITH MOCKS WHAT'S THE PROBLEM WITH MOCKS 26
38. FOREWORD “There are two types of mocks: inputs and outputs” 27
39. INPUT @Test fun testSomething() { val fakeData = forger.anInt() val inputMock = mock() whenever (inputMock.getData()) doReturn fakeData // Call to getData is not verified directly } 28
40. OUTPUT @Test fun testSomething() { val outputMock = mock() // mock is not stubbed verify(outputMock).onSomethingDone() } 29
41. THE PROBLEM: MOCKS ARE DESIGNED TO MAKE THE TESTS PASS 30
42. ARE YOUR MOCKS CONSISTENT… 31
43. ARE YOUR MOCKS CONSISTENT… … BETWEEN TWO TEST METHODS ? 31.1
44. @Test fun testFoo() { whenever(mockQueue.isEmpty()) doReturn(true) whenever(mockQueue.getFirst() doThrow(new Exception()) // … } @Test fun testBar() { whenever(mockQueue.isEmpty()) doReturn(true) whenever(mockQueue.getFirst() doReturn(null) // … } 32
45. SOLUTION ? 33
46. SOLUTION ? PREPARE ALL THE MOCKS IN THE SETUP METHOD… 33.1
47. SOLUTION ? PREPARE ALL THE MOCKS IN THE SETUP METHOD… Mocks have single configuration 33.2
48. SOLUTION ? PREPARE ALL THE MOCKS IN THE SETUP METHOD… Mocks have single configuration All stubbing is done in one place 33.3
49. ARE YOUR MOCKS CONSISTENT… 34
50. ARE YOUR MOCKS CONSISTENT… … BETWEEN TWO TEST CLASSES ? 34.1
51. class FooTest { @Before fun setupMock() { whenever(mockQueue.isEmpty()) doReturn(true) whenever(mockQueue.getFirst() doThrow(new Exception()) } } class FizTest { @Before fun setupMock() { whenever(mockQueue.isEmpty()) doReturn(true) whenever(mockQueue.getFirst() doReturn(null) // … } } 35
52. SOLUTION ? 36
53. SOLUTION ? USE A SEPARATE CLASS TO SETUP THE MOCKS 36.1
54. SOLUTION ? USE A SEPARATE CLASS TO SETUP THE MOCKS INTRODUCING CONTRACT BASED MOCKING 36.2
55. DEFINING THE CONTRACT class QueueContract { val mockedQueue : Queue = mock() fun prepareEmpty() { whenever(mockedQueue.isEmpty()).thenReturn(true) whenever(mockedQueue.getFirst().thenThrow(new Exception()) } } 37
56. USING THE CONTRACT class FooTest { lateinit var queueContract : QueueContract @Before fun setUpQueue() { queueContract = new QueueContract() } @Test fun testWithEmptyQueue() { queueContract.prepareEmpty() val mock = queueContract.mockedQueue // … } } 38
57. ARE YOUR MOCKS CONSISTENT… 39
58. ARE YOUR MOCKS CONSISTENT… … WITH THE CONCRETE IMPLEMENTATIONS ? 39.1
59. ARE YOUR MOCKS CONSISTENT… … WITH THE CONCRETE IMPLEMENTATIONS ? Fuzzy specs 39.2
60. ARE YOUR MOCKS CONSISTENT… … WITH THE CONCRETE IMPLEMENTATIONS ? Fuzzy specs 3rd party implementation 39.3
61. ARE YOUR MOCKS CONSISTENT… … WITH THE CONCRETE IMPLEMENTATIONS ? Fuzzy specs 3rd party implementation Undocumented behavior 39.4
62. SOLUTION ? 40
63. SOLUTION ? USE THE CONTRACT AS TEST DEFINITION 40.1
64. INTRODUCING (Still in αlphα) MESMAEKER 41
65. DEFINING THE CONTRACT open class QueueContract : MockitoContract<Queue>(Queue::class.java) { @Clause fun prepareEmpty() { whenever { it.isEmpty() }.thenReturn(true) whenever { it.getFirst() }.thenThrow(new Exception()) } } 42
66. USING THE CONTRACT class FooTest { @Rule public BaseContractRule contracts = new BaseContractRule(); @Contract lateinit var queueContract: QueueContract @Test fun testWithEmptyQueue() { queueContract.prepareEmpty() val mock = queueContract.getMock() // … } } 43
67. TESTING THE CONTRACT class BarTest (clause : String) : ContractValidator<Queue, QueueContract> (clause) { override fun instantiateContract() : QueueContract = QueueContract() override fun instantiateSubject() : Queue = Queue() companion object { @JvmStatic @Parameterized.Parameters() fun data(): Collection<Array<Any?>> { return generateTestParameters(QueueContract::class.java) } } } 44
68. WRITING THE CLAUSES OF THE CONTRACT @Clause fun prepareWithSize(size: Int) { applyIfImplementation { it.resize(size) } whenever { it.size() }.thenReturn(size) whenever { it.get(-1) }.thenThrow(IooBException()) whenever { it.get(size) }.thenThrow(IooBException()) } 45
69. DETAIL'S IN THE FINE PRINT override fun getClauseParams(clause: String): Array<Any?>? { when (contractClause) { "prepareWithSize" -> return arrayOf(forger.aSmallInt()) else -> return emptyArray() } } 46
70. CONCLUSION 47
71. CONCLUSION Remember, tests can have bugs and code smells too 47.1
72. CONCLUSION Remember, tests can have bugs and code smells too Try to think of what could go wrong 47.2
73. CONCLUSION Remember, tests can have bugs and code smells too Try to think of what could go wrong Look for the edge cases 47.3
74. “You need to be as confident in the code you test as you are in the test you code ” 48
75. THANKS FOR YOUR ATTENTION ANY QUESTION ? (αlphα) https://github.com/xgouchet/Elmyr https://github.com/xgouchet/Mesmaeker 49