It's time to up your test game

On paper, everyone agrees that tests are important. And yet, more often than not, tests are neglected. In the rare case where we take the time to write them, more often than not it’s considered a cumbersome task, and given less attention than the "real code".

Tests are often as important, if not more so, as the code that will be published on the Play Store. This talk will start with describing a productive mindset when writing tests, followed by a few examples of common pitfalls and how to avoid them, and will conclude with a couple of advanced testing paradigms that can help make your tests cleaner and better.

Transcript

1. It’s time to UP
   your test game
   Xavier F. Gouchet, Mobile RUM Team Lead
   @xgouchet / @datadoghq
   

   View Slide

2. Monitor your Web and Mobile
   applications from your
   users’ perspective
   ● Performance Improvement
   ● Error Management and Alerting
   ● Customer Support
   ● Application Analytics
   About… Datadog Real User Monitoring
   2
   

   View Slide

3. Agenda
   ● Common testing* pitfalls
   ● Advanced topics :
   ○ Fuzzy Testing
   ○ Mutation Testing
   3
   

   View Slide

4. 1. Common Testing Pitfalls
   Writing tests is hard!
   

   View Slide

5. Tests are hard to maintain
   5
   

   View Slide

6. Causes for poor tests
   Lazy code
   It’s “just
   test”, why
   should we put
   too much
   effort in it?
   Not thought
   through
   Let’s just
   test the
   nominal case,
   what could go
   wrong?
   Tests without
   intent
   We need to
   increase our
   coverage, but
   what are we
   testing
   exactly ?
   6
   

   View Slide

7. Write clean code tests
   7
   

   View Slide

8. Keep your best practices
   ● Code style and naming
   conventions
   ● KISS
   ● SOLID
   ● Use Patterns
   ● Avoid Hardcoded Values✘
   8
   

   View Slide

9. Keep in mind the goals of testing
   ● Verify that the code works
   ● Make the contract explicit
   ● Guide the development (TDD)
   Watch your tests fail✘
   ● Prevent regressions
   ● Ease maintenance
   9
   

   View Slide

10. Covering all cases
    10
    

    View Slide

11. What are you actually testing
    ● Each test has a single purpose
    ● Test name reflects the intent
    ○ `given X when Y then Z`
    ○ `A performs B on C event`
    11
    

    View Slide

12. Think of any way the code might break
    ● Rule of thumb
    ○ 1+ test for nominal case
    ○ 1+ test for each input (invalid, null)
    ○ 1+ test for error state(s)
    ● Add a test for every new bug
    12
    

    View Slide

13. 2. Fuzzy Testing
    Real users aren’t all named
    Alice or Bob…
    

    View Slide

14. ✘Hardcoded Values
    14
    

    View Slide

15. What could go wrong here?
    @Test
    fun `test get host`() {
    // Given
    val email = "username@example.com"
    // When
    val host = extractHost(email)
    // Then
    assertThat(host).isEqualTo("example.com")
    }
    15
    

    View Slide

16. Constants are especially bad in Tests
    ● You’re only testing one value
    … Every
    … single
    … time!
    ● Humans are bad at generating
    “randomness”
    16
    

    View Slide

17. Fuzzy testing
    17
    

    View Slide

18. The main concept
    ARRANGE
    Generate
    “random” data
    ACT
    Perform some
    test
    ASSERT
    Verify the
    output /
    callbacks
    matches the
    test data
    18
    

    View Slide

19. Using random test data…
    ● Overall a wider range of data is
    tested
    ● Understand boundaries of
    valid/invalid inputs
    ● Write generic tests without
    assumptions
    ● Find edge cases
    19
    

    View Slide

20. Using the Elmyr library
    https://github.com/xgouchet/Elmyr
    20
    

    View Slide

21. Basic tests
    class FooTest {
    @get:Rule
    val forge = ForgeRule()
    @IntForgery
    var i: Int = 0
    // …
    }
    21
    

    View Slide

22. Basic tests
    class FooTest {
    @get:Rule
    val forge = ForgeRule()
    @IntForgery
    var i: Int = 0
    // …
    }
    22
    

    View Slide

23. Basic tests
    class FooTest {
    @get:Rule
    val forge = ForgeRule()
    @IntForgery
    var i: Int = 0
    // …
    }
    23
    

    View Slide

24. Generate any kind of data
    ● Boolean
    ● Numeric
    ● String
    ● Collections (List, Set, Map)
    ● Enums
    24
    

    View Slide

25. Adapting to your use cases
    ● Keep the control over the
    generation of primitives
    ● Generate custom class instances
    25
    

    View Slide

26. Controlling your random data
    class FooTest {
    @get:Rule
    val forge = ForgeRule()
    @IntForgery(min = 13, max = 42)
    var i: Int = 0
    // …
    }
    26
    

    View Slide

27. Controlling your random data
    class FooTest {
    @get:Rule
    val forge = ForgeRule()
    @IntForgery(min = 13, max = 42)
    var i: Int = 0
    // …
    }
    27
    

    View Slide

28. Randomize your own classes
    class BookForgeryFactory : ForgeryFactory {
    override fun getForgery(forge: Forge): Book {
    return Book(
    bookId = forge.getForgery(),
    title = forge.anAlphabeticalString(),
    author = forge.anAlphabeticalString(),
    email = forge.aStringMatching("\w+@\w+.com")
    )
    }
    }
    28
    

    View Slide

29. Randomize your own classes
    class BookForgeryFactory : ForgeryFactory {
    override fun getForgery(forge: Forge): Book {
    return Book(
    bookId = forge.getForgery(),
    title = forge.anAlphabeticalString(),
    author = forge.anAlphabeticalString(),
    email = forge.aStringMatching("\w+@\w+.com")
    )
    }
    }
    29
    

    View Slide

30. Randomize your own classes
    class BookForgeryFactory : ForgeryFactory {
    override fun getForgery(forge: Forge): Book {
    return Book(
    bookId = forge.getForgery(),
    title = forge.anAlphabeticalString(),
    author = forge.anAlphabeticalString(),
    email = forge.aStringMatching("\w+@\w+.com")
    )
    }
    }
    30
    

    View Slide

31. Randomize your own classes
    class BookForgeryFactory : ForgeryFactory {
    override fun getForgery(forge: Forge): Book {
    return Book(
    bookId = forge.getForgery(),
    title = forge.anAlphabeticalString(),
    author = forge.anAlphabeticalString(),
    email = forge.aStringMatching("\w+@\w+.com")
    )
    }
    }
    31
    

    View Slide

32. Randomize your own classes
    class BookForgeryFactory : ForgeryFactory {
    override fun getForgery(forge: Forge): Book {
    return Book(
    bookId = forge.getForgery(),
    title = forge.anAlphabeticalString(),
    author = forge.anAlphabeticalString(),
    email = forge.aStringMatching("\w+@\w+.com")
    )
    }
    }
    32
    

    View Slide

33. Keep the random within a controlled range
    class FooTest {
    @get:Rule
    val forge = ForgeRule()
    .withFactory(BookFactory())
    @Forgery
    lateinit var book: Book
    // …
    }
    33
    

    View Slide

34. Keep the random within a controlled range
    class FooTest {
    @get:Rule
    val forge = ForgeRule()
    .withFactory(BookFactory())
    @Forgery
    lateinit var book: Book
    // …
    }
    34
    

    View Slide

35. Keep the random within a controlled range
    class FooTest {
    @get:Rule
    val forge = ForgeRule()
    .withFactory(BookFactory())
    @Forgery
    lateinit var book: Book
    // …
    }
    35
    

    View Slide

36. Making tests reproducible
    failed
    with Forge seed 0xdeadfa11
    36
    

    View Slide

37. Making tests reproducible
    class FooTest {
    @get:Rule
    val forge = ForgeRule(0xdeadfa11)
    @IntForgery
    var i: Int = 0
    // …
    }
    37
    

    View Slide

38. Making tests reproducible
    class FooTest {
    @get:Rule
    val forge = ForgeRule(0xdeadfa11)
    @IntForgery
    var i: Int = 0
    // …
    }
    38
    

    View Slide

39. 3. Mutation Testing
    How confident are you about
    your coverage?
    

    View Slide

40. SUCCESS: 26/26 (100%) Tests passed
    — @bloerwald
    40
    

    View Slide

41. Code coverage only tells you
    which lines are visited,
    not which lines are tested
    41
    

    View Slide

42. An 80% code coverage doesn’t mean
    “80% of your code is tested”;
    it means
    “at least 20% of your code is not tested”
    42
    

    View Slide

43. ✘Watch your tests fail
    43
    

    View Slide

44. Code review your tests
    Code has bugs
    Tests are Code
    ∴
    Tests have bugs
    44
    

    View Slide

45. The Mutation Testing Algorithm
    ● Have some (valid) tests
    ● Mutate the production code
    i.e. introduce some bugs
    ● Run the tests again
    ● …
    ● Profit
    45
    

    View Slide

46. Using the PITest library
    https://github.com/koral--/gradle-pitest-plugin
    46
    

    View Slide

47. Add the plugin to your build.gradle
    apply plugin: 'pl.droidsonroids.pitest'
    pitest {
    targetClasses = ['com.example.*']
    outputFormats = ['XML', 'HTML']
    }
    47
    

    View Slide

48. 48
    Mutation Coverage
    

    View Slide

49. if (x > y) …
    if (a && b) …
    return t
    i++
    Before / After Mutation
    if (x ≤ y) …
    if (a || b) …
    return null
    i--
    49
    

    View Slide

50. Think about your tests as an
    incident response team;
    Mutation testing is similar to
    running a drill emergency.
    50
    

    View Slide

51. It won’t find
    bugs in the code,
    just reveal test
    issues
    Useful but not critical
    Only simulates
    atomic faults
    51
    

    View Slide

52. Key takeaway
    52
    

    View Slide

53. “You need to be as confident
    in the tests you code as you are in
    the code you test.”
    53
    

    View Slide

54. ● Test is code
    ● Vary your test inputs
    ● Make sure your tests are
    actually testing something
    Being confident in your tests
    54
    

    View Slide

55. https://github.com/DataDog/dd-sdk-android
    55
    

    View Slide

56. 56
    Free Trial
    dtdg.co/droidcon
    Win a Google Pixel 4
    

    View Slide

57. Thanks!
    Any questions?
    @xgouchet / @datadoghq
    57
    Presentation template by SlidesCarnival · Icons by Aleksandra Wolska @tutsii159
    

    View Slide