What's the Fuzz about Unit Testing

All my tests used to involve users named Alice and Bob. Aged 42. Working at FooBar Inc. The problem when using hard-coded values in your tests, means that you only test one single path of your code. Ever. During this talk, you will discover a fuzzy testing library to make your test use a wider range of data.

Transcript

1. What's the Fuzz with Unit Tests Droidcon London 2018
2. About me… • Xavier F. Gouchet • Writing Android apps since Android 1.5 (Cupcake) • Lead Android Engineer at WorkWell • @xgouchet on Github / StackOverflow / LinkedIn
4. When the tests go wrong
5. The test method… @Test fun testItemClicked() { val user = User(id = 42 /* … */) testedPresenter.onUserSelected(user) verify(mockView).navigateToChat(user.id) }
6. The tested method… fun onUserSelected(user: User) { view?.navigateToChat(company.id) }
7. … and the test setup method @Before fun setUp() { val company = Company(id = 42 /* … */) testedPresenter = Presenter(company) testedPresenter.view = mockView }
8. All my tests involved users named Bob. Aged 42. Working at FooBar Inc.
9. When all your fake data are “foo” and 42, how can you be sure that your tests are valid ?
10. The problem when using static test data • Where does the 42 come from ? • Lots of hardcoded values ! • Only one value is tested. Ever.
11. Is there a solution ?
12. What we can do about it • Keep track of test values ? • Parameterized Tests ? • Random test data !
13. Using fuzzy data in your tests • Random data • Help define boundaries of valid / invalid inputs • Help find edge cases
14. Without fuzz @Test fun invalidateDataAfterTimeout() { testedObject.setLastCallTimestamp(0) testedObject.setDataTTL(42) val ts = 666 val result = testedObject.isDataValid(ts) assertFalse(result) }
15. With fuzz @Test fun invalidateDataAfterTimeout() { val fakeTS = rand.nextLong() val fakeTTL = rand.nextLong() val fakeDelay = rand.nextLong() testedObject.setLastCallTimestamp(fakeTS) testedObject.setDataTTL(fakeTTL) val ts = fakeTS + fakeTTL + fakeDelay val result = testedObject.isDataValid(ts) assertFalse(result) }
16. • Long overflow • negative TTL • Invalid values • … … but not too random
17. https://github.com/xgouchet/Elmyr
18. Elmyr setup class FooTest { @JvmField @Rule val forger = new JUnitForger() // … }
19. Elmyr forging numbers forger.anInt(min = 0, max =100) forger.aSmallInt() forger.aPositiveLong() forger.aGaussianFloat(mean = 42f, standardDeviation = 100f) forger.aDoubleArray(DoubleConstraint.NEGATIVE_STRICT)
20. Elmyr forging Strings forger.anHexadecimalString() forger.aWord() forger.aSentence() forger.anEmail() forger.aUrl() forger.aStringArray(StringConstraint.WORD) forger.aStringMatching("(0|+44)\\d{10}")
21. Elmyr forging (from) Collections forger.aList { aUrl() } forger.aSubListOf(myList) forger.anElementFrom(myCollection) forger.anElementFrom("foo", "bar", "baz" /* … */) forger.aValueFrom(MyEnum::class) forger.aNullableFrom { aUrl() }
22. Elmyr forging (from) Collections forger.aList { aUrl() } forger.aSubListOf(myList) forger.anElementFrom(myCollection) forger.anElementFrom("foo", "bar", "baz" /* … */) forger.aValueFrom(MyEnum::class) forger.aNullableFrom { aUrl() }
23. Without fuzz @Test fun invalidateDataAfterTimeout() { val fakeTS = forger.aTimestamp() val fakeTTL = forger.aLong(1, 86400000) val fakeDelay = forger.aLong(1, 86400000) testedObject.setLastCallTimestamp(fakeTS) testedObject.setDataTTL(fakeTTL) val ts = fakeTS + fakeTTL + fakeDelay val result = testedObject.isDataValid(ts) assertFalse(result) }
24. Making tests reproducible
25. <FooTest.testSomething()> failed with fake seed 0x4815162342 Add the following line in your setUp method to reproduce : forger.reset(0x4815162342L) Elmyr libary
26. Adapting to your use cases
27. Custom POJO / Data Class data class User( val id: String, val firstName: String, val lastName: String, val jobTitle: String?, val avatarUrl: String?, val email: String?, val tags: List<String>? )
28. Custom extension function fun Forger.aUser(): User { return User( id = aNumericalString(), firstName = aWord(Case.CAPITALIZE), lastName = aWord(Case.CAPITALIZE), jobTitle = aNullable { aSentence() }, avatarUrl = aNullable { aUrl() }, email = aNullable { anEmail() }, tags = aList { aWord() } ) }
29. Our original test… @Before fun setUp() { val company = forger.aCompany() testedPresenter = Presenter(company) testedPresenter.view = mockView } @Test fun testItemClicked() { val user = forger.aUser() testedPresenter.onUserSelected(user) verify(mockView).navigateToChat(user.id) }
30. “You need to be as confident in the tests we code as we are in the code we test.” Xavier F. Gouchet
31. Thanks for your attention Any question ? @xgouchet http://github.com/xgouchet/Elmyr