Property Based Testing in Practice

On paper, everyone agrees that tests are important. And yet, more often than not, tests are neglected. In the rare case where we take the time to write them, more often than not it’s considered a cumbersome task, and given less attention than the "real code". This leads to a lot of code smell being written in tests, one of which is using hardcoded values.

Tests are often as important, if not more so, as the code that will be published on the Play Store. This workshop will start with describing a productive mindset when writing tests, followed by an in-depth presentation of what Property based testing and Fuzzy testing are, with hands-on exercises to let attendees understand how to apply this concept to any existing Android codebase.

Transcript

1. Property Based Testing
   In Practice
   

   View Slide

2. 2
   Xavier F. Gouchet
   Senior Software Engineer at Datadog
   @xgouchet / @datadoghq
   

   View Slide

3. View Slide

4. Goal for the next 80 minutes
   Why?
   Let’s understand
   motivation behind it.
   How does it work?
   Let’s dig into the
   theory and principles.
   Concrete examples
   a.k.a. the part when
   you are working and I
   can just relax… or
   not
   4
   

   View Slide

5. https://github.com/xgouchet/PBTIP
   5
   

   View Slide

6. Introduction
   

   View Slide

7. What’s wrong with that?
   @ParameterizedTest
   @ValueSource(strings = [
   "127.0.0.1 - alice [] \"GET /report HTTP/1.0\" 200 123",
   "127.0.0.1 - bob [09/May/2018:16:00:39 +0000] \"GET HTTP/1.0\" 200 123",
   "127.0.0.1 - cat [09/May/2018:16:00:39 +0000] \"GET /report HTTP/1.0"
   ])
   fun parser_must_throw_for_invalid_logs(rawLog: String) {
   shouldThrow {
   LogParser(line = rawLog).parse()
   }
   }
   7
   

   View Slide

8. What’s wrong with that?
   @ParameterizedTest
   @ValueSource(strings = [
   "127.0.0.1 - alice [] \"GET /report HTTP/1.0\" 200 123",
   "127.0.0.1 - bob [09/May/2018:16:00:39 +0000] \"GET HTTP/1.0\" 200 123",
   "127.0.0.1 - cat [09/May/2018:16:00:39 +0000] \"GET /report HTTP/1.0"
   ])
   fun parser_must_throw_for_invalid_logs(rawLog: String) {
   shouldThrow {
   LogParser(line = rawLog).parse()
   }
   }
   8
   

   View Slide

9. What’s wrong with that?
   @ParameterizedTest
   @ValueSource(strings = [
   "127.0.0.1 - alice [] \"GET /report HTTP/1.0\" 200 123",
   "127.0.0.1 - bob [09/May/2018:16:00:39 +0000] \"GET HTTP/1.0\" 200 123",
   "127.0.0.1 - cat [09/May/2018:16:00:39 +0000] \"GET /report HTTP/1.0"
   ])
   fun parser_must_throw_for_invalid_logs(rawLog: String) {
   shouldThrow {
   LogParser(line = rawLog).parse()
   }
   }
   9
   

   View Slide

10. What’s wrong with that?
    You’re only testing the same values
    … Every
    … single
    … time!
    10
    

    View Slide

11. Fixture is there to make the tests pass
    Are you really
    trying to make
    your code break?
    Are you sure you
    tested all
    possible invalid
    values?
    11
    

    View Slide

12. Humans are bad at generating “randomness”
    12
    Credit: XKCD
    

    View Slide

13. We don’t know why the inputs are invalid
    13
    // What’s wrong with this value ?
    127.0.0.1 — james [09/May/2018:16:00:39
    +0000] "GET /report HTTP/1.0" 200 123
    

    View Slide

14. 14
    Feature
    Compliance
    Input
    Coverage
    Compiler
    Static Analysis
    Monkey Testing
    Unit Tests
    Integration
    Tests
    

    View Slide

15. Property Based Testing
    

    View Slide

16. Fuzzy Testing
    fuzzy testing /ˈfʌzɪ tɛstɪŋ/
    n. feeding a feature (function, program,
    etc.) data from a large corpus, possibly
    dynamically generated, in order to see
    whether or not it fails.
    16
    

    View Slide

17. Property (Based) Testing
    property testing /ˈprɑpɚtɪ tɛstɪŋ/
    n. testing method relying on the abstract
    properties of a feature, rather than on a
    short list of sample case.
    17
    

    View Slide

18. What is a property
    property /ˈprɑpɚtɪ/
    n. a functional property which is part of a
    feature under test. (≠ class property).
    18
    

    View Slide

19. Describe the properties of features in the test
    ● Input boundaries
    ○ For any int between 0 and 128
    ○ For any float number
    ○ For any string representing an email
    ● Describe expectations based on
    the input
    ● Generate random inputs each run
    19
    

    View Slide

20. Pros
    ● Abstraction: property instead of
    value
    ● Coverage: generate true* random
    data
    ● Confidence: make your tests
    great again
    20
    

    View Slide

21. Cons
    ● Put more effort in writing tests
    ● Needs more effort to fix flaky
    tests
    21
    

    View Slide

22. Random data caveat
    ● Define the range of valid and
    invalid inputs
    ○ Valid input range can be smaller than
    you think
    ○ Invalid input can take more form than
    you think
    ○ When you ask for an input as a String,
    you’re never expecting it to be
    “𝘈Ḇ𝖢𝕯٤Ḟԍ𝕙Ǐ𝙅ḲԸⲘ𝙉০Ρ𝗤Ɍ𝓢ȚЦṼѠ𝓧ƳȤ”
    22
    

    View Slide

23. Hands on
    

    View Slide

24. Hello world… Addition (the old way)
    @Test fun test_addition() {
    val a = 3
    val b = 7
    val res = calculator.add(a, b)
    assertThat(res).isEqualTo(10)
    }
    24
    

    View Slide

25. Hello world… Addition (Fuzzy)
    @Test fun test_addition(
    @IntForgery a: Int,
    @IntForgery b: Int
    ) {
    val res = calculator.add(a, b)
    assertThat(res).isEqualTo(a + b)
    }
    25
    

    View Slide

26. Let’s talk about math… What is ‘addition‘?
    ∀ (𝑎,𝑏) ∈ ℤ
    𝑎 + 𝑏 = 𝑏 + 𝑎
    ∀ (𝑎,𝑏,𝑐) ∈ ℤ
    (𝑎 + 𝑏) + 𝑐 = 𝑎 + (𝑏 + 𝑐)
    ∀ 𝑎 ∈ ℤ
    𝑎 + 0 = 𝑎
    ℤ = { 0, 1, 2, 3, 4, 5, … }
    26
    

    View Slide

27. Hello world… Addition (Property based 1⁄
    4
    )
    @Test fun `addition is commutative`(
    @IntForgery a: Int,
    @IntForgery b: Int
    ) {
    val add_a_b = calculator.add(a, b)
    val add_b_a = calculator.add(b, a)
    assertThat(add_a_b).isEqualTo(add_b_a)
    }
    27
    

    View Slide

28. Hello world… Addition (Property based 2⁄
    4
    )
    @Test fun property_associativity(
    @IntForgery a: Int,
    @IntForgery b: Int,
    @IntForgery c: Int
    ) {
    val add_left = calculator.add(calculator.add(a, b), c)
    val add_right = calculator.add(a, calculator.add(b, c))
    assertThat(add_left).isEqualTo(add_right)
    }
    28
    

    View Slide

29. Hello world… Addition (Property based 3⁄
    4
    )
    @Test fun property_identity(@IntForgery a: Int) {
    val res = calculator.add(a, 0)
    assertThat(res).isEqualTo(a)
    }
    29
    

    View Slide

30. Hello world… Addition (Property based 4⁄
    4
    )
    @Test fun property_successor(@IntForgery a: Int) {
    val add_a_2 = calculator.add(a, 2)
    val add_a_1_1 = calculator.add(calculator.add(a, 1), 1)
    assertThat(add_a_2).isEqualTo(add_a_1_1)
    }
    30
    

    View Slide

31. Frequently Asked Questions
    

    View Slide

32. 32
    How many times do I
    have to run my tests?
    A lot of times; but your CI can
    help you with that
    

    View Slide

33. 33
    Won’t that create
    aky tests?
    Yes!
    And that’s a good thing!
    Flaky tests mean your input
    range boundaries are
    not perfect (yet).
    

    View Slide

34. Going Further
    

    View Slide

35. Shrinking & Falsifying
    shrinking /ˈʃɹɪnkɪŋ/
    n. feature of some property testing or
    fuzzing libraries to run the tests until it
    finds the exact boundaries of valid/invalid
    inputs.
    falsifying /ˈfɔlsɪfaɪɪŋ/
    n. feature of some property testing or
    fuzzing libraries to run the tests until it
    finds at least one invalid input.
    35
    

    View Slide

36. “You need to be as confident
    in the tests you code as you are in
    the code you test.”
    36
    

    View Slide

37. Thanks!
    Any questions?
    ✘ @xgouchet
    ✘ @datadoghq
    37
    Presentation template by SlidesCarnival · Icons by Aleksandra Wolska @tutsii159
    

    View Slide