$30 off During Our Annual Pro Sale. View Details »

Property Based Testing in Practice

Property Based Testing in Practice

On paper, everyone agrees that tests are important. And yet, more often than not, tests are neglected. In the rare case where we take the time to write them, more often than not it’s considered a cumbersome task, and given less attention than the "real code". This leads to a lot of code smell being written in tests, one of which is using hardcoded values.

Tests are often as important, if not more so, as the code that will be published on the Play Store. This workshop will start with describing a productive mindset when writing tests, followed by an in-depth presentation of what Property based testing and Fuzzy testing are, with hands-on exercises to let attendees understand how to apply this concept to any existing Android codebase.

Xavier Gouchet

April 26, 2022
Tweet

More Decks by Xavier Gouchet

Other Decks in Programming

Transcript

  1. Property Based Testing
    In Practice

    View Slide

  2. 2
    Xavier F. Gouchet
    Senior Software Engineer at Datadog
    @xgouchet / @datadoghq

    View Slide

  3. View Slide

  4. Goal for the next 80 minutes
    Why?
    Let’s understand
    motivation behind it.
    How does it work?
    Let’s dig into the
    theory and principles.
    Concrete examples
    a.k.a. the part when
    you are working and I
    can just relax… or
    not
    4

    View Slide

  5. https://github.com/xgouchet/PBTIP
    5

    View Slide

  6. Introduction

    View Slide

  7. What’s wrong with that?
    @ParameterizedTest
    @ValueSource(strings = [
    "127.0.0.1 - alice [] \"GET /report HTTP/1.0\" 200 123",
    "127.0.0.1 - bob [09/May/2018:16:00:39 +0000] \"GET HTTP/1.0\" 200 123",
    "127.0.0.1 - cat [09/May/2018:16:00:39 +0000] \"GET /report HTTP/1.0"
    ])
    fun parser_must_throw_for_invalid_logs(rawLog: String) {
    shouldThrow {
    LogParser(line = rawLog).parse()
    }
    }
    7

    View Slide

  8. What’s wrong with that?
    @ParameterizedTest
    @ValueSource(strings = [
    "127.0.0.1 - alice [] \"GET /report HTTP/1.0\" 200 123",
    "127.0.0.1 - bob [09/May/2018:16:00:39 +0000] \"GET HTTP/1.0\" 200 123",
    "127.0.0.1 - cat [09/May/2018:16:00:39 +0000] \"GET /report HTTP/1.0"
    ])
    fun parser_must_throw_for_invalid_logs(rawLog: String) {
    shouldThrow {
    LogParser(line = rawLog).parse()
    }
    }
    8

    View Slide

  9. What’s wrong with that?
    @ParameterizedTest
    @ValueSource(strings = [
    "127.0.0.1 - alice [] \"GET /report HTTP/1.0\" 200 123",
    "127.0.0.1 - bob [09/May/2018:16:00:39 +0000] \"GET HTTP/1.0\" 200 123",
    "127.0.0.1 - cat [09/May/2018:16:00:39 +0000] \"GET /report HTTP/1.0"
    ])
    fun parser_must_throw_for_invalid_logs(rawLog: String) {
    shouldThrow {
    LogParser(line = rawLog).parse()
    }
    }
    9

    View Slide

  10. What’s wrong with that?
    You’re only testing the same values
    … Every
    … single
    … time!
    10

    View Slide

  11. Fixture is there to make the tests pass
    Are you really
    trying to make
    your code break?
    Are you sure you
    tested all
    possible invalid
    values?
    11

    View Slide

  12. Humans are bad at generating “randomness”
    12
    Credit: XKCD

    View Slide

  13. We don’t know why the inputs are invalid
    13
    // What’s wrong with this value ?
    127.0.0.1 — james [09/May/2018:16:00:39
    +0000] "GET /report HTTP/1.0" 200 123

    View Slide

  14. 14
    Feature
    Compliance
    Input
    Coverage
    Compiler
    Static Analysis
    Monkey Testing
    Unit Tests
    Integration
    Tests

    View Slide

  15. Property Based Testing

    View Slide

  16. Fuzzy Testing
    fuzzy testing /ˈfʌzɪ tɛstɪŋ/
    n. feeding a feature (function, program,
    etc.) data from a large corpus, possibly
    dynamically generated, in order to see
    whether or not it fails.
    16

    View Slide

  17. Property (Based) Testing
    property testing /ˈprɑpɚtɪ tɛstɪŋ/
    n. testing method relying on the abstract
    properties of a feature, rather than on a
    short list of sample case.
    17

    View Slide

  18. What is a property
    property /ˈprɑpɚtɪ/
    n. a functional property which is part of a
    feature under test. (≠ class property).
    18

    View Slide

  19. Describe the properties of features in the test
    ● Input boundaries
    ○ For any int between 0 and 128
    ○ For any float number
    ○ For any string representing an email
    ● Describe expectations based on
    the input
    ● Generate random inputs each run
    19

    View Slide

  20. Pros
    ● Abstraction: property instead of
    value
    ● Coverage: generate true* random
    data
    ● Confidence: make your tests
    great again
    20

    View Slide

  21. Cons
    ● Put more effort in writing tests
    ● Needs more effort to fix flaky
    tests
    21

    View Slide

  22. Random data caveat
    ● Define the range of valid and
    invalid inputs
    ○ Valid input range can be smaller than
    you think
    ○ Invalid input can take more form than
    you think
    ○ When you ask for an input as a String,
    you’re never expecting it to be
    “𝘈Ḇ𝖢𝕯٤Ḟԍ𝕙Ǐ𝙅ḲԸⲘ𝙉০Ρ𝗤Ɍ𝓢ȚЦṼѠ𝓧ƳȤ”
    22

    View Slide

  23. Hands on

    View Slide

  24. Hello world… Addition (the old way)
    @Test fun test_addition() {
    val a = 3
    val b = 7
    val res = calculator.add(a, b)
    assertThat(res).isEqualTo(10)
    }
    24

    View Slide

  25. Hello world… Addition (Fuzzy)
    @Test fun test_addition(
    @IntForgery a: Int,
    @IntForgery b: Int
    ) {
    val res = calculator.add(a, b)
    assertThat(res).isEqualTo(a + b)
    }
    25

    View Slide

  26. Let’s talk about math… What is ‘addition‘?
    ∀ (𝑎,𝑏) ∈ ℤ
    𝑎 + 𝑏 = 𝑏 + 𝑎
    ∀ (𝑎,𝑏,𝑐) ∈ ℤ
    (𝑎 + 𝑏) + 𝑐 = 𝑎 + (𝑏 + 𝑐)
    ∀ 𝑎 ∈ ℤ
    𝑎 + 0 = 𝑎
    ℤ = { 0, 1, 2, 3, 4, 5, … }
    26

    View Slide

  27. Hello world… Addition (Property based 1⁄
    4
    )
    @Test fun `addition is commutative`(
    @IntForgery a: Int,
    @IntForgery b: Int
    ) {
    val add_a_b = calculator.add(a, b)
    val add_b_a = calculator.add(b, a)
    assertThat(add_a_b).isEqualTo(add_b_a)
    }
    27

    View Slide

  28. Hello world… Addition (Property based 2⁄
    4
    )
    @Test fun property_associativity(
    @IntForgery a: Int,
    @IntForgery b: Int,
    @IntForgery c: Int
    ) {
    val add_left = calculator.add(calculator.add(a, b), c)
    val add_right = calculator.add(a, calculator.add(b, c))
    assertThat(add_left).isEqualTo(add_right)
    }
    28

    View Slide

  29. Hello world… Addition (Property based 3⁄
    4
    )
    @Test fun property_identity(@IntForgery a: Int) {
    val res = calculator.add(a, 0)
    assertThat(res).isEqualTo(a)
    }
    29

    View Slide

  30. Hello world… Addition (Property based 4⁄
    4
    )
    @Test fun property_successor(@IntForgery a: Int) {
    val add_a_2 = calculator.add(a, 2)
    val add_a_1_1 = calculator.add(calculator.add(a, 1), 1)
    assertThat(add_a_2).isEqualTo(add_a_1_1)
    }
    30

    View Slide

  31. Frequently Asked Questions

    View Slide

  32. 32
    How many times do I
    have to run my tests?
    A lot of times; but your CI can
    help you with that

    View Slide

  33. 33
    Won’t that create
    aky tests?
    Yes!
    And that’s a good thing!
    Flaky tests mean your input
    range boundaries are
    not perfect (yet).

    View Slide

  34. Going Further

    View Slide

  35. Shrinking & Falsifying
    shrinking /ˈʃɹɪnkɪŋ/
    n. feature of some property testing or
    fuzzing libraries to run the tests until it
    finds the exact boundaries of valid/invalid
    inputs.
    falsifying /ˈfɔlsɪfaɪɪŋ/
    n. feature of some property testing or
    fuzzing libraries to run the tests until it
    finds at least one invalid input.
    35

    View Slide

  36. “You need to be as confident
    in the tests you code as you are in
    the code you test.”
    36

    View Slide

  37. Thanks!
    Any questions?
    ✘ @xgouchet
    ✘ @datadoghq
    37
    Presentation template by SlidesCarnival · Icons by Aleksandra Wolska @tutsii159

    View Slide