What's the Fuzz about Unit Testing

All my tests used to involve users named Alice and Bob. Aged 42. Working at FooBar Inc. The problem when using hard-coded values in your tests, means that you only test one single path of your code. Ever. During this talk, you will discover a fuzzy testing library to make your test use a wider range of data.

Transcript

1. What's the Fuzz
   with Unit Tests
   Droidcon London 2018
   

   View Slide

2. About me…
   ● Xavier F. Gouchet
   ● Writing Android apps since Android 1.5 (Cupcake)
   ● Lead Android Engineer at WorkWell
   ● @xgouchet on Github / StackOverflow / LinkedIn
   

   View Slide

3. View Slide

4. When the tests go wrong
   

   View Slide

5. The test method…
   @Test fun testItemClicked() {
   val user = User(id = 42 /* … */)
   testedPresenter.onUserSelected(user)
   verify(mockView).navigateToChat(user.id)
   }
   

   View Slide

6. The tested method…
   fun onUserSelected(user: User) {
   view?.navigateToChat(company.id)
   }
   

   View Slide

7. … and the test setup method
   @Before fun setUp() {
   val company = Company(id = 42 /* … */)
   testedPresenter = Presenter(company)
   testedPresenter.view = mockView
   }
   

   View Slide

8. All my tests involved
   users named Bob.
   Aged 42.
   Working at FooBar Inc.
   

   View Slide

9. When all your fake data
   are “foo” and 42, how
   can you be sure that
   your tests are valid ?
   

   View Slide

10. The problem when using static test data
    ● Where does the 42 come from ?
    ● Lots of hardcoded values !
    ● Only one value is tested. Ever.
    

    View Slide

11. Is there a solution ?
    

    View Slide

12. What we can do
    about it ● Keep track of test values ?
    ● Parameterized Tests ?
    ● Random test data !
    

    View Slide

13. Using fuzzy data in your tests
    ● Random data
    ● Help define boundaries of valid / invalid inputs
    ● Help find edge cases
    

    View Slide

14. Without fuzz
    @Test fun invalidateDataAfterTimeout() {
    testedObject.setLastCallTimestamp(0)
    testedObject.setDataTTL(42)
    val ts = 666
    val result = testedObject.isDataValid(ts)
    assertFalse(result)
    }
    

    View Slide

15. With fuzz
    @Test fun invalidateDataAfterTimeout() {
    val fakeTS = rand.nextLong()
    val fakeTTL = rand.nextLong()
    val fakeDelay = rand.nextLong()
    testedObject.setLastCallTimestamp(fakeTS)
    testedObject.setDataTTL(fakeTTL)
    val ts = fakeTS + fakeTTL + fakeDelay
    val result = testedObject.isDataValid(ts)
    assertFalse(result)
    }
    

    View Slide

16. ● Long overflow
    ● negative TTL
    ● Invalid values
    ● …
    … but not too
    random
    

    View Slide

17. https://github.com/xgouchet/Elmyr
    

    View Slide

18. Elmyr setup
    class FooTest {
    @JvmField @Rule val forger = new JUnitForger()
    // …
    }
    

    View Slide

19. Elmyr forging numbers
    forger.anInt(min = 0, max =100)
    forger.aSmallInt()
    forger.aPositiveLong()
    forger.aGaussianFloat(mean = 42f, standardDeviation = 100f)
    forger.aDoubleArray(DoubleConstraint.NEGATIVE_STRICT)
    

    View Slide

20. Elmyr forging Strings
    forger.anHexadecimalString()
    forger.aWord()
    forger.aSentence()
    forger.anEmail()
    forger.aUrl()
    forger.aStringArray(StringConstraint.WORD)
    forger.aStringMatching("(0|+44)\\d{10}")
    

    View Slide

21. Elmyr forging (from) Collections
    forger.aList { aUrl() }
    forger.aSubListOf(myList)
    forger.anElementFrom(myCollection)
    forger.anElementFrom("foo", "bar", "baz" /* … */)
    forger.aValueFrom(MyEnum::class)
    forger.aNullableFrom { aUrl() }
    

    View Slide

22. Elmyr forging (from) Collections
    forger.aList { aUrl() }
    forger.aSubListOf(myList)
    forger.anElementFrom(myCollection)
    forger.anElementFrom("foo", "bar", "baz" /* … */)
    forger.aValueFrom(MyEnum::class)
    forger.aNullableFrom { aUrl() }
    

    View Slide

23. Without fuzz
    @Test fun invalidateDataAfterTimeout() {
    val fakeTS = forger.aTimestamp()
    val fakeTTL = forger.aLong(1, 86400000)
    val fakeDelay = forger.aLong(1, 86400000)
    testedObject.setLastCallTimestamp(fakeTS)
    testedObject.setDataTTL(fakeTTL)
    val ts = fakeTS + fakeTTL + fakeDelay
    val result = testedObject.isDataValid(ts)
    assertFalse(result)
    }
    

    View Slide

24. Making tests reproducible
    

    View Slide

25. failed with fake seed 0x4815162342
    Add the following line in your setUp method to reproduce :
    forger.reset(0x4815162342L)
    Elmyr libary
    

    View Slide

26. Adapting to your use cases
    

    View Slide

27. Custom POJO / Data Class
    data class User(
    val id: String,
    val firstName: String,
    val lastName: String,
    val jobTitle: String?,
    val avatarUrl: String?,
    val email: String?,
    val tags: List?
    )
    

    View Slide

28. Custom extension function
    fun Forger.aUser(): User {
    return User(
    id = aNumericalString(),
    firstName = aWord(Case.CAPITALIZE),
    lastName = aWord(Case.CAPITALIZE),
    jobTitle = aNullable { aSentence() },
    avatarUrl = aNullable { aUrl() },
    email = aNullable { anEmail() },
    tags = aList { aWord() }
    )
    }
    

    View Slide

29. Our original test…
    @Before fun setUp() {
    val company = forger.aCompany()
    testedPresenter = Presenter(company)
    testedPresenter.view = mockView
    }
    @Test fun testItemClicked() {
    val user = forger.aUser()
    testedPresenter.onUserSelected(user)
    verify(mockView).navigateToChat(user.id)
    }
    

    View Slide

30. “You need to be as confident in
    the tests we code as we are in
    the code we test.”
    Xavier F. Gouchet
    

    View Slide

31. Thanks for your
    attention Any question ?
    @xgouchet
    http://github.com/xgouchet/Elmyr
    

    View Slide