Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Property Based Testing in Practice

Property Based Testing in Practice

On paper, everyone agrees that tests are important. And yet, more often than not, tests are neglected. In the rare case where we take the time to write them, more often than not it’s considered a cumbersome task, and given less attention than the "real code". This leads to a lot of code smell being written in tests, one of which is using hardcoded values.

Tests are often as important, if not more so, as the code that will be published on the Play Store. This workshop will start with describing a productive mindset when writing tests, followed by an in-depth presentation of what Property based testing and Fuzzy testing are, with hands-on exercises to let attendees understand how to apply this concept to any existing Android codebase.

Cf95f93e78f6d6dd0630049396f723c6?s=128

Xavier Gouchet

April 26, 2022
Tweet

More Decks by Xavier Gouchet

Other Decks in Programming

Transcript

  1. Property Based Testing In Practice

  2. 2 Xavier F. Gouchet Senior Software Engineer at Datadog @xgouchet

    / @datadoghq
  3. None
  4. Goal for the next 80 minutes Why? Let’s understand motivation

    behind it. How does it work? Let’s dig into the theory and principles. Concrete examples a.k.a. the part when you are working and I can just relax… or not 4
  5. https://github.com/xgouchet/PBTIP 5

  6. Introduction

  7. What’s wrong with that? @ParameterizedTest @ValueSource(strings = [ "127.0.0.1 -

    alice [] \"GET /report HTTP/1.0\" 200 123", "127.0.0.1 - bob [09/May/2018:16:00:39 +0000] \"GET HTTP/1.0\" 200 123", "127.0.0.1 - cat [09/May/2018:16:00:39 +0000] \"GET /report HTTP/1.0" ]) fun parser_must_throw_for_invalid_logs(rawLog: String) { shouldThrow<IllegalArgumentException> { LogParser(line = rawLog).parse() } } 7
  8. What’s wrong with that? @ParameterizedTest @ValueSource(strings = [ "127.0.0.1 -

    alice [] \"GET /report HTTP/1.0\" 200 123", "127.0.0.1 - bob [09/May/2018:16:00:39 +0000] \"GET HTTP/1.0\" 200 123", "127.0.0.1 - cat [09/May/2018:16:00:39 +0000] \"GET /report HTTP/1.0" ]) fun parser_must_throw_for_invalid_logs(rawLog: String) { shouldThrow<IllegalArgumentException> { LogParser(line = rawLog).parse() } } 8
  9. What’s wrong with that? @ParameterizedTest @ValueSource(strings = [ "127.0.0.1 -

    alice [] \"GET /report HTTP/1.0\" 200 123", "127.0.0.1 - bob [09/May/2018:16:00:39 +0000] \"GET HTTP/1.0\" 200 123", "127.0.0.1 - cat [09/May/2018:16:00:39 +0000] \"GET /report HTTP/1.0" ]) fun parser_must_throw_for_invalid_logs(rawLog: String) { shouldThrow<IllegalArgumentException> { LogParser(line = rawLog).parse() } } 9
  10. What’s wrong with that? You’re only testing the same values

    … Every … single … time! 10
  11. Fixture is there to make the tests pass Are you

    really trying to make your code break? Are you sure you tested all possible invalid values? 11
  12. Humans are bad at generating “randomness” 12 Credit: XKCD

  13. We don’t know why the inputs are invalid 13 //

    What’s wrong with this value ? 127.0.0.1 — james [09/May/2018:16:00:39 +0000] "GET /report HTTP/1.0" 200 123
  14. 14 Feature Compliance Input Coverage Compiler Static Analysis Monkey Testing

    Unit Tests Integration Tests
  15. Property Based Testing

  16. Fuzzy Testing fuzzy testing /ˈfʌzɪ tɛstɪŋ/ n. feeding a feature

    (function, program, etc.) data from a large corpus, possibly dynamically generated, in order to see whether or not it fails. 16
  17. Property (Based) Testing property testing /ˈprɑpɚtɪ tɛstɪŋ/ n. testing method

    relying on the abstract properties of a feature, rather than on a short list of sample case. 17
  18. What is a property property /ˈprɑpɚtɪ/ n. a functional property

    which is part of a feature under test. (≠ class property). 18
  19. Describe the properties of features in the test • Input

    boundaries ◦ For any int between 0 and 128 ◦ For any float number ◦ For any string representing an email • Describe expectations based on the input • Generate random inputs each run 19
  20. Pros • Abstraction: property instead of value • Coverage: generate

    true* random data • Confidence: make your tests great again 20
  21. Cons • Put more effort in writing tests • Needs

    more effort to fix flaky tests 21
  22. Random data caveat • Define the range of valid and

    invalid inputs ◦ Valid input range can be smaller than you think ◦ Invalid input can take more form than you think ◦ When you ask for an input as a String, you’re never expecting it to be “𝘈Ḇ𝖢𝕯٤Ḟԍ𝕙Ǐ𝙅ḲԸⲘ𝙉০Ρ𝗤Ɍ𝓢ȚЦṼѠ𝓧ƳȤ” 22
  23. Hands on

  24. Hello world… Addition (the old way) @Test fun test_addition() {

    val a = 3 val b = 7 val res = calculator.add(a, b) assertThat(res).isEqualTo(10) } 24
  25. Hello world… Addition (Fuzzy) @Test fun test_addition( @IntForgery a: Int,

    @IntForgery b: Int ) { val res = calculator.add(a, b) assertThat(res).isEqualTo(a + b) } 25
  26. Let’s talk about math… What is ‘addition‘? ∀ (𝑎,𝑏) ∈

    ℤ 𝑎 + 𝑏 = 𝑏 + 𝑎 ∀ (𝑎,𝑏,𝑐) ∈ ℤ (𝑎 + 𝑏) + 𝑐 = 𝑎 + (𝑏 + 𝑐) ∀ 𝑎 ∈ ℤ 𝑎 + 0 = 𝑎 ℤ = { 0, 1, 2, 3, 4, 5, … } 26
  27. Hello world… Addition (Property based 1⁄ 4 ) @Test fun

    `addition is commutative`( @IntForgery a: Int, @IntForgery b: Int ) { val add_a_b = calculator.add(a, b) val add_b_a = calculator.add(b, a) assertThat(add_a_b).isEqualTo(add_b_a) } 27
  28. Hello world… Addition (Property based 2⁄ 4 ) @Test fun

    property_associativity( @IntForgery a: Int, @IntForgery b: Int, @IntForgery c: Int ) { val add_left = calculator.add(calculator.add(a, b), c) val add_right = calculator.add(a, calculator.add(b, c)) assertThat(add_left).isEqualTo(add_right) } 28
  29. Hello world… Addition (Property based 3⁄ 4 ) @Test fun

    property_identity(@IntForgery a: Int) { val res = calculator.add(a, 0) assertThat(res).isEqualTo(a) } 29
  30. Hello world… Addition (Property based 4⁄ 4 ) @Test fun

    property_successor(@IntForgery a: Int) { val add_a_2 = calculator.add(a, 2) val add_a_1_1 = calculator.add(calculator.add(a, 1), 1) assertThat(add_a_2).isEqualTo(add_a_1_1) } 30
  31. Frequently Asked Questions

  32. 32 How many times do I have to run my

    tests? A lot of times; but your CI can help you with that
  33. 33 Won’t that create aky tests? Yes! And that’s a

    good thing! Flaky tests mean your input range boundaries are not perfect (yet).
  34. Going Further

  35. Shrinking & Falsifying shrinking /ˈʃɹɪnkɪŋ/ n. feature of some property

    testing or fuzzing libraries to run the tests until it finds the exact boundaries of valid/invalid inputs. falsifying /ˈfɔlsɪfaɪɪŋ/ n. feature of some property testing or fuzzing libraries to run the tests until it finds at least one invalid input. 35
  36. “You need to be as confident in the tests you

    code as you are in the code you test.” 36
  37. Thanks! Any questions? ✘ @xgouchet ✘ @datadoghq 37 Presentation template

    by SlidesCarnival · Icons by Aleksandra Wolska @tutsii159