What's the Fuzz about Unit Testing

What's the Fuzz about Unit Testing

All my tests used to involve users named Alice and Bob. Aged 42. Working at FooBar Inc. The problem when using hard-coded values in your tests, means that you only test one single path of your code. Ever. During this talk, you will discover a fuzzy testing library to make your test use a wider range of data.

Cf95f93e78f6d6dd0630049396f723c6?s=128

Xavier Gouchet

October 25, 2018
Tweet

Transcript

  1. What's the Fuzz with Unit Tests Droidcon London 2018

  2. About me… • Xavier F. Gouchet • Writing Android apps

    since Android 1.5 (Cupcake) • Lead Android Engineer at WorkWell • @xgouchet on Github / StackOverflow / LinkedIn
  3. None
  4. When the tests go wrong

  5. The test method… @Test fun testItemClicked() { val user =

    User(id = 42 /* … */) testedPresenter.onUserSelected(user) verify(mockView).navigateToChat(user.id) }
  6. The tested method… fun onUserSelected(user: User) { view?.navigateToChat(company.id) }

  7. … and the test setup method @Before fun setUp() {

    val company = Company(id = 42 /* … */) testedPresenter = Presenter(company) testedPresenter.view = mockView }
  8. All my tests involved users named Bob. Aged 42. Working

    at FooBar Inc.
  9. When all your fake data are “foo” and 42, how

    can you be sure that your tests are valid ?
  10. The problem when using static test data • Where does

    the 42 come from ? • Lots of hardcoded values ! • Only one value is tested. Ever.
  11. Is there a solution ?

  12. What we can do about it • Keep track of

    test values ? • Parameterized Tests ? • Random test data !
  13. Using fuzzy data in your tests • Random data •

    Help define boundaries of valid / invalid inputs • Help find edge cases
  14. Without fuzz @Test fun invalidateDataAfterTimeout() { testedObject.setLastCallTimestamp(0) testedObject.setDataTTL(42) val ts

    = 666 val result = testedObject.isDataValid(ts) assertFalse(result) }
  15. With fuzz @Test fun invalidateDataAfterTimeout() { val fakeTS = rand.nextLong()

    val fakeTTL = rand.nextLong() val fakeDelay = rand.nextLong() testedObject.setLastCallTimestamp(fakeTS) testedObject.setDataTTL(fakeTTL) val ts = fakeTS + fakeTTL + fakeDelay val result = testedObject.isDataValid(ts) assertFalse(result) }
  16. • Long overflow • negative TTL • Invalid values •

    … … but not too random
  17. https://github.com/xgouchet/Elmyr

  18. Elmyr setup class FooTest { @JvmField @Rule val forger =

    new JUnitForger() // … }
  19. Elmyr forging numbers forger.anInt(min = 0, max =100) forger.aSmallInt() forger.aPositiveLong()

    forger.aGaussianFloat(mean = 42f, standardDeviation = 100f) forger.aDoubleArray(DoubleConstraint.NEGATIVE_STRICT)
  20. Elmyr forging Strings forger.anHexadecimalString() forger.aWord() forger.aSentence() forger.anEmail() forger.aUrl() forger.aStringArray(StringConstraint.WORD) forger.aStringMatching("(0|+44)\\d{10}")

  21. Elmyr forging (from) Collections forger.aList { aUrl() } forger.aSubListOf(myList) forger.anElementFrom(myCollection)

    forger.anElementFrom("foo", "bar", "baz" /* … */) forger.aValueFrom(MyEnum::class) forger.aNullableFrom { aUrl() }
  22. Elmyr forging (from) Collections forger.aList { aUrl() } forger.aSubListOf(myList) forger.anElementFrom(myCollection)

    forger.anElementFrom("foo", "bar", "baz" /* … */) forger.aValueFrom(MyEnum::class) forger.aNullableFrom { aUrl() }
  23. Without fuzz @Test fun invalidateDataAfterTimeout() { val fakeTS = forger.aTimestamp()

    val fakeTTL = forger.aLong(1, 86400000) val fakeDelay = forger.aLong(1, 86400000) testedObject.setLastCallTimestamp(fakeTS) testedObject.setDataTTL(fakeTTL) val ts = fakeTS + fakeTTL + fakeDelay val result = testedObject.isDataValid(ts) assertFalse(result) }
  24. Making tests reproducible

  25. <FooTest.testSomething()> failed with fake seed 0x4815162342 Add the following line

    in your setUp method to reproduce : forger.reset(0x4815162342L) Elmyr libary
  26. Adapting to your use cases

  27. Custom POJO / Data Class data class User( val id:

    String, val firstName: String, val lastName: String, val jobTitle: String?, val avatarUrl: String?, val email: String?, val tags: List<String>? )
  28. Custom extension function fun Forger.aUser(): User { return User( id

    = aNumericalString(), firstName = aWord(Case.CAPITALIZE), lastName = aWord(Case.CAPITALIZE), jobTitle = aNullable { aSentence() }, avatarUrl = aNullable { aUrl() }, email = aNullable { anEmail() }, tags = aList { aWord() } ) }
  29. Our original test… @Before fun setUp() { val company =

    forger.aCompany() testedPresenter = Presenter(company) testedPresenter.view = mockView } @Test fun testItemClicked() { val user = forger.aUser() testedPresenter.onUserSelected(user) verify(mockView).navigateToChat(user.id) }
  30. “You need to be as confident in the tests we

    code as we are in the code we test.” Xavier F. Gouchet
  31. Thanks for your attention Any question ? @xgouchet http://github.com/xgouchet/Elmyr