It's time to up your test game

It's time to up your test game

On paper, everyone agrees that tests are important. And yet, more often than not, tests are neglected. In the rare case where we take the time to write them, more often than not it’s considered a cumbersome task, and given less attention than the "real code".

Tests are often as important, if not more so, as the code that will be published on the Play Store. This talk will start with describing a productive mindset when writing tests, followed by a few examples of common pitfalls and how to avoid them, and will conclude with a couple of advanced testing paradigms that can help make your tests cleaner and better.

Cf95f93e78f6d6dd0630049396f723c6?s=128

Xavier Gouchet

September 10, 2020
Tweet

Transcript

  1. It’s time to UP your test game Xavier F. Gouchet,

    Mobile RUM Team Lead @xgouchet / @datadoghq
  2. Monitor your Web and Mobile applications from your users’ perspective

    • Performance Improvement • Error Management and Alerting • Customer Support • Application Analytics About… Datadog Real User Monitoring 2
  3. Agenda • Common testing* pitfalls • Advanced topics : ◦

    Fuzzy Testing ◦ Mutation Testing 3
  4. 1. Common Testing Pitfalls Writing tests is hard!

  5. Tests are hard to maintain 5

  6. Causes for poor tests Lazy code It’s “just test”, why

    should we put too much effort in it? Not thought through Let’s just test the nominal case, what could go wrong? Tests without intent We need to increase our coverage, but what are we testing exactly ? 6
  7. Write clean code tests 7

  8. Keep your best practices • Code style and naming conventions

    • KISS • SOLID • Use Patterns • Avoid Hardcoded Values✘ 8
  9. Keep in mind the goals of testing • Verify that

    the code works • Make the contract explicit • Guide the development (TDD) Watch your tests fail✘ • Prevent regressions • Ease maintenance 9
  10. Covering all cases 10

  11. What are you actually testing • Each test has a

    single purpose • Test name reflects the intent ◦ `given X when Y then Z` ◦ `A performs B on C event` 11
  12. Think of any way the code might break • Rule

    of thumb ◦ 1+ test for nominal case ◦ 1+ test for each input (invalid, null) ◦ 1+ test for error state(s) • Add a test for every new bug 12
  13. 2. Fuzzy Testing Real users aren’t all named Alice or

    Bob…
  14. ✘Hardcoded Values 14

  15. What could go wrong here? @Test fun `test get host`()

    { // Given val email = "username@example.com" // When val host = extractHost(email) // Then assertThat(host).isEqualTo("example.com") } 15
  16. Constants are especially bad in Tests • You’re only testing

    one value … Every … single … time! • Humans are bad at generating “randomness” 16
  17. Fuzzy testing 17

  18. The main concept ARRANGE Generate “random” data ACT Perform some

    test ASSERT Verify the output / callbacks matches the test data 18
  19. Using random test data… • Overall a wider range of

    data is tested • Understand boundaries of valid/invalid inputs • Write generic tests without assumptions • Find edge cases 19
  20. Using the Elmyr library https://github.com/xgouchet/Elmyr 20

  21. Basic tests class FooTest { @get:Rule val forge = ForgeRule()

    @IntForgery var i: Int = 0 // … } 21
  22. Basic tests class FooTest { @get:Rule val forge = ForgeRule()

    @IntForgery var i: Int = 0 // … } 22
  23. Basic tests class FooTest { @get:Rule val forge = ForgeRule()

    @IntForgery var i: Int = 0 // … } 23
  24. Generate any kind of data • Boolean • Numeric •

    String • Collections (List, Set, Map) • Enums 24
  25. Adapting to your use cases • Keep the control over

    the generation of primitives • Generate custom class instances 25
  26. Controlling your random data class FooTest { @get:Rule val forge

    = ForgeRule() @IntForgery(min = 13, max = 42) var i: Int = 0 // … } 26
  27. Controlling your random data class FooTest { @get:Rule val forge

    = ForgeRule() @IntForgery(min = 13, max = 42) var i: Int = 0 // … } 27
  28. Randomize your own classes class BookForgeryFactory : ForgeryFactory<Book> { override

    fun getForgery(forge: Forge): Book { return Book( bookId = forge.getForgery<UUID>(), title = forge.anAlphabeticalString(), author = forge.anAlphabeticalString(), email = forge.aStringMatching("\w+@\w+.com") ) } } 28
  29. Randomize your own classes class BookForgeryFactory : ForgeryFactory<Book> { override

    fun getForgery(forge: Forge): Book { return Book( bookId = forge.getForgery<UUID>(), title = forge.anAlphabeticalString(), author = forge.anAlphabeticalString(), email = forge.aStringMatching("\w+@\w+.com") ) } } 29
  30. Randomize your own classes class BookForgeryFactory : ForgeryFactory<Book> { override

    fun getForgery(forge: Forge): Book { return Book( bookId = forge.getForgery<UUID>(), title = forge.anAlphabeticalString(), author = forge.anAlphabeticalString(), email = forge.aStringMatching("\w+@\w+.com") ) } } 30
  31. Randomize your own classes class BookForgeryFactory : ForgeryFactory<Book> { override

    fun getForgery(forge: Forge): Book { return Book( bookId = forge.getForgery<UUID>(), title = forge.anAlphabeticalString(), author = forge.anAlphabeticalString(), email = forge.aStringMatching("\w+@\w+.com") ) } } 31
  32. Randomize your own classes class BookForgeryFactory : ForgeryFactory<Book> { override

    fun getForgery(forge: Forge): Book { return Book( bookId = forge.getForgery<UUID>(), title = forge.anAlphabeticalString(), author = forge.anAlphabeticalString(), email = forge.aStringMatching("\w+@\w+.com") ) } } 32
  33. Keep the random within a controlled range class FooTest {

    @get:Rule val forge = ForgeRule() .withFactory(BookFactory()) @Forgery lateinit var book: Book // … } 33
  34. Keep the random within a controlled range class FooTest {

    @get:Rule val forge = ForgeRule() .withFactory(BookFactory()) @Forgery lateinit var book: Book // … } 34
  35. Keep the random within a controlled range class FooTest {

    @get:Rule val forge = ForgeRule() .withFactory(BookFactory()) @Forgery lateinit var book: Book // … } 35
  36. Making tests reproducible <FooTest.testSomething()> failed with Forge seed 0xdeadfa11 36

  37. Making tests reproducible class FooTest { @get:Rule val forge =

    ForgeRule(0xdeadfa11) @IntForgery var i: Int = 0 // … } 37
  38. Making tests reproducible class FooTest { @get:Rule val forge =

    ForgeRule(0xdeadfa11) @IntForgery var i: Int = 0 // … } 38
  39. 3. Mutation Testing How confident are you about your coverage?

  40. SUCCESS: 26/26 (100%) Tests passed — @bloerwald 40

  41. Code coverage only tells you which lines are visited, not

    which lines are tested 41
  42. An 80% code coverage doesn’t mean “80% of your code

    is tested”; it means “at least 20% of your code is not tested” 42
  43. ✘Watch your tests fail 43

  44. Code review your tests Code has bugs Tests are Code

    ∴ Tests have bugs 44
  45. The Mutation Testing Algorithm • Have some (valid) tests •

    Mutate the production code i.e. introduce some bugs • Run the tests again • … • Profit 45
  46. Using the PITest library https://github.com/koral--/gradle-pitest-plugin 46

  47. Add the plugin to your build.gradle apply plugin: 'pl.droidsonroids.pitest' pitest

    { targetClasses = ['com.example.*'] outputFormats = ['XML', 'HTML'] } 47
  48. 48 Mutation Coverage

  49. if (x > y) … if (a && b) …

    return t i++ Before / After Mutation if (x ≤ y) … if (a || b) … return null i-- 49
  50. Think about your tests as an incident response team; Mutation

    testing is similar to running a drill emergency. 50
  51. It won’t find bugs in the code, just reveal test

    issues Useful but not critical Only simulates atomic faults 51
  52. Key takeaway 52

  53. “You need to be as confident in the tests you

    code as you are in the code you test.” 53
  54. • Test is code • Vary your test inputs •

    Make sure your tests are actually testing something Being confident in your tests 54
  55. https://github.com/DataDog/dd-sdk-android 55

  56. 56 Free Trial dtdg.co/droidcon Win a Google Pixel 4

  57. Thanks! Any questions? @xgouchet / @datadoghq 57 Presentation template by

    SlidesCarnival · Icons by Aleksandra Wolska @tutsii159