It's time to up your test game

Slide 1

Slide 1 text

It’s time to UP your test game Xavier F. Gouchet, Mobile RUM Team Lead @xgouchet / @datadoghq

Slide 2

Slide 2 text

Monitor your Web and Mobile applications from your users’ perspective ● Performance Improvement ● Error Management and Alerting ● Customer Support ● Application Analytics About… Datadog Real User Monitoring 2

Slide 3

Slide 3 text

Agenda ● Common testing* pitfalls ● Advanced topics : ○ Fuzzy Testing ○ Mutation Testing 3

Slide 4

Slide 4 text

1. Common Testing Pitfalls Writing tests is hard!

Slide 5

Slide 5 text

Tests are hard to maintain 5

Slide 6

Slide 6 text

Causes for poor tests Lazy code It’s “just test”, why should we put too much effort in it? Not thought through Let’s just test the nominal case, what could go wrong? Tests without intent We need to increase our coverage, but what are we testing exactly ? 6

Slide 7

Slide 7 text

Write clean code tests 7

Slide 8

Slide 8 text

Keep your best practices ● Code style and naming conventions ● KISS ● SOLID ● Use Patterns ● Avoid Hardcoded Values✘ 8

Slide 9

Slide 9 text

Keep in mind the goals of testing ● Verify that the code works ● Make the contract explicit ● Guide the development (TDD) Watch your tests fail✘ ● Prevent regressions ● Ease maintenance 9

Slide 10

Slide 10 text

Covering all cases 10

Slide 11

Slide 11 text

What are you actually testing ● Each test has a single purpose ● Test name reflects the intent ○ `given X when Y then Z` ○ `A performs B on C event` 11

Slide 12

Slide 12 text

Think of any way the code might break ● Rule of thumb ○ 1+ test for nominal case ○ 1+ test for each input (invalid, null) ○ 1+ test for error state(s) ● Add a test for every new bug 12

Slide 13

Slide 13 text

2. Fuzzy Testing Real users aren’t all named Alice or Bob…

Slide 14

Slide 14 text

✘Hardcoded Values 14

Slide 15

Slide 15 text

What could go wrong here? @Test fun `test get host`() { // Given val email = "username@example.com" // When val host = extractHost(email) // Then assertThat(host).isEqualTo("example.com") } 15

Slide 16

Slide 16 text

Constants are especially bad in Tests ● You’re only testing one value … Every … single … time! ● Humans are bad at generating “randomness” 16

Slide 17

Slide 17 text

Fuzzy testing 17

Slide 18

Slide 18 text

The main concept ARRANGE Generate “random” data ACT Perform some test ASSERT Verify the output / callbacks matches the test data 18

Slide 19

Slide 19 text

Using random test data… ● Overall a wider range of data is tested ● Understand boundaries of valid/invalid inputs ● Write generic tests without assumptions ● Find edge cases 19

Slide 20

Slide 20 text

Using the Elmyr library https://github.com/xgouchet/Elmyr 20

Slide 21

Slide 21 text

Basic tests class FooTest { @get:Rule val forge = ForgeRule() @IntForgery var i: Int = 0 // … } 21

Slide 22

Slide 22 text

Basic tests class FooTest { @get:Rule val forge = ForgeRule() @IntForgery var i: Int = 0 // … } 22

Slide 23

Slide 23 text

Basic tests class FooTest { @get:Rule val forge = ForgeRule() @IntForgery var i: Int = 0 // … } 23

Slide 24

Slide 24 text

Generate any kind of data ● Boolean ● Numeric ● String ● Collections (List, Set, Map) ● Enums 24

Slide 25

Slide 25 text

Adapting to your use cases ● Keep the control over the generation of primitives ● Generate custom class instances 25

Slide 26

Slide 26 text

Controlling your random data class FooTest { @get:Rule val forge = ForgeRule() @IntForgery(min = 13, max = 42) var i: Int = 0 // … } 26

Slide 27

Slide 27 text

Controlling your random data class FooTest { @get:Rule val forge = ForgeRule() @IntForgery(min = 13, max = 42) var i: Int = 0 // … } 27

Slide 28

Slide 28 text

Randomize your own classes class BookForgeryFactory : ForgeryFactory { override fun getForgery(forge: Forge): Book { return Book( bookId = forge.getForgery(), title = forge.anAlphabeticalString(), author = forge.anAlphabeticalString(), email = forge.aStringMatching("\w+@\w+.com") ) } } 28

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Keep the random within a controlled range class FooTest { @get:Rule val forge = ForgeRule() .withFactory(BookFactory()) @Forgery lateinit var book: Book // … } 33

Slide 34

Slide 34 text

Keep the random within a controlled range class FooTest { @get:Rule val forge = ForgeRule() .withFactory(BookFactory()) @Forgery lateinit var book: Book // … } 34

Slide 35

Slide 35 text

Keep the random within a controlled range class FooTest { @get:Rule val forge = ForgeRule() .withFactory(BookFactory()) @Forgery lateinit var book: Book // … } 35

Slide 36

Slide 36 text

Making tests reproducible failed with Forge seed 0xdeadfa11 36

Slide 37

Slide 37 text

Making tests reproducible class FooTest { @get:Rule val forge = ForgeRule(0xdeadfa11) @IntForgery var i: Int = 0 // … } 37

Slide 38

Slide 38 text

Making tests reproducible class FooTest { @get:Rule val forge = ForgeRule(0xdeadfa11) @IntForgery var i: Int = 0 // … } 38

Slide 39

Slide 39 text

3. Mutation Testing How confident are you about your coverage?

Slide 40

Slide 40 text

SUCCESS: 26/26 (100%) Tests passed — @bloerwald 40

Slide 41

Slide 41 text

Code coverage only tells you which lines are visited, not which lines are tested 41

Slide 42

Slide 42 text

An 80% code coverage doesn’t mean “80% of your code is tested”; it means “at least 20% of your code is not tested” 42

Slide 43

Slide 43 text

✘Watch your tests fail 43

Slide 44

Slide 44 text

Code review your tests Code has bugs Tests are Code ∴ Tests have bugs 44

Slide 45

Slide 45 text

The Mutation Testing Algorithm ● Have some (valid) tests ● Mutate the production code i.e. introduce some bugs ● Run the tests again ● … ● Profit 45

Slide 46

Slide 46 text

Using the PITest library https://github.com/koral--/gradle-pitest-plugin 46

Slide 47

Slide 47 text

Add the plugin to your build.gradle apply plugin: 'pl.droidsonroids.pitest' pitest { targetClasses = ['com.example.*'] outputFormats = ['XML', 'HTML'] } 47

Slide 48

Slide 48 text

48 Mutation Coverage

Slide 49

Slide 49 text

if (x > y) … if (a && b) … return t i++ Before / After Mutation if (x ≤ y) … if (a || b) … return null i-- 49

Slide 50

Slide 50 text

Think about your tests as an incident response team; Mutation testing is similar to running a drill emergency. 50

Slide 51

Slide 51 text

It won’t find bugs in the code, just reveal test issues Useful but not critical Only simulates atomic faults 51

Slide 52

Slide 52 text

Key takeaway 52

Slide 53

Slide 53 text

“You need to be as confident in the tests you code as you are in the code you test.” 53

Slide 54

Slide 54 text

● Test is code ● Vary your test inputs ● Make sure your tests are actually testing something Being confident in your tests 54

Slide 55

Slide 55 text

https://github.com/DataDog/dd-sdk-android 55

Slide 56

Slide 56 text

56 Free Trial dtdg.co/droidcon Win a Google Pixel 4

Slide 57

Slide 57 text

Thanks! Any questions? @xgouchet / @datadoghq 57 Presentation template by SlidesCarnival · Icons by Aleksandra Wolska @tutsii159