Property Based Testing in Practice

On paper, everyone agrees that tests are important. And yet, more often than not, tests are neglected. In the rare case where we take the time to write them, more often than not it’s considered a cumbersome task, and given less attention than the "real code". This leads to a lot of code smell being written in tests, one of which is using hardcoded values.

Tests are often as important, if not more so, as the code that will be published on the Play Store. This workshop will start with describing a productive mindset when writing tests, followed by an in-depth presentation of what Property based testing and Fuzzy testing are, with hands-on exercises to let attendees understand how to apply this concept to any existing Android codebase.

Transcript

1. Property Based Testing In Practice
2. 2 Xavier F. Gouchet Senior Software Engineer at Datadog @xgouchet / @datadoghq
4. Goal for the next 80 minutes Why? Let’s understand motivation behind it. How does it work? Let’s dig into the theory and principles. Concrete examples a.k.a. the part when you are working and I can just relax… or not 4
5. https://github.com/xgouchet/PBTIP 5
6. Introduction
7. What’s wrong with that? @ParameterizedTest @ValueSource(strings = [ "127.0.0.1 - alice [] \"GET /report HTTP/1.0\" 200 123", "127.0.0.1 - bob [09/May/2018:16:00:39 +0000] \"GET HTTP/1.0\" 200 123", "127.0.0.1 - cat [09/May/2018:16:00:39 +0000] \"GET /report HTTP/1.0" ]) fun parser_must_throw_for_invalid_logs(rawLog: String) { shouldThrow<IllegalArgumentException> { LogParser(line = rawLog).parse() } } 7
8. What’s wrong with that? @ParameterizedTest @ValueSource(strings = [ "127.0.0.1 - alice [] \"GET /report HTTP/1.0\" 200 123", "127.0.0.1 - bob [09/May/2018:16:00:39 +0000] \"GET HTTP/1.0\" 200 123", "127.0.0.1 - cat [09/May/2018:16:00:39 +0000] \"GET /report HTTP/1.0" ]) fun parser_must_throw_for_invalid_logs(rawLog: String) { shouldThrow<IllegalArgumentException> { LogParser(line = rawLog).parse() } } 8
9. What’s wrong with that? @ParameterizedTest @ValueSource(strings = [ "127.0.0.1 - alice [] \"GET /report HTTP/1.0\" 200 123", "127.0.0.1 - bob [09/May/2018:16:00:39 +0000] \"GET HTTP/1.0\" 200 123", "127.0.0.1 - cat [09/May/2018:16:00:39 +0000] \"GET /report HTTP/1.0" ]) fun parser_must_throw_for_invalid_logs(rawLog: String) { shouldThrow<IllegalArgumentException> { LogParser(line = rawLog).parse() } } 9
10. What’s wrong with that? You’re only testing the same values … Every … single … time! 10
11. Fixture is there to make the tests pass Are you really trying to make your code break? Are you sure you tested all possible invalid values? 11
12. Humans are bad at generating “randomness” 12 Credit: XKCD
13. We don’t know why the inputs are invalid 13 // What’s wrong with this value ? 127.0.0.1 — james [09/May/2018:16:00:39 +0000] "GET /report HTTP/1.0" 200 123
14. 14 Feature Compliance Input Coverage Compiler Static Analysis Monkey Testing Unit Tests Integration Tests
15. Property Based Testing
16. Fuzzy Testing fuzzy testing /ˈfʌzɪ tɛstɪŋ/ n. feeding a feature (function, program, etc.) data from a large corpus, possibly dynamically generated, in order to see whether or not it fails. 16
17. Property (Based) Testing property testing /ˈprɑpɚtɪ tɛstɪŋ/ n. testing method relying on the abstract properties of a feature, rather than on a short list of sample case. 17
18. What is a property property /ˈprɑpɚtɪ/ n. a functional property which is part of a feature under test. (≠ class property). 18
19. Describe the properties of features in the test • Input boundaries ◦ For any int between 0 and 128 ◦ For any float number ◦ For any string representing an email • Describe expectations based on the input • Generate random inputs each run 19
20. Pros • Abstraction: property instead of value • Coverage: generate true* random data • Confidence: make your tests great again 20
21. Cons • Put more effort in writing tests • Needs more effort to fix flaky tests 21
22. Random data caveat • Define the range of valid and invalid inputs ◦ Valid input range can be smaller than you think ◦ Invalid input can take more form than you think ◦ When you ask for an input as a String, you’re never expecting it to be “𝘈Ḇ𝖢𝕯٤Ḟԍ𝕙Ǐ𝙅ḲԸⲘ𝙉০Ρ𝗤Ɍ𝓢ȚЦṼѠ𝓧ƳȤ” 22
23. Hands on
24. Hello world… Addition (the old way) @Test fun test_addition() { val a = 3 val b = 7 val res = calculator.add(a, b) assertThat(res).isEqualTo(10) } 24
25. Hello world… Addition (Fuzzy) @Test fun test_addition( @IntForgery a: Int, @IntForgery b: Int ) { val res = calculator.add(a, b) assertThat(res).isEqualTo(a + b) } 25
26. Let’s talk about math… What is ‘addition‘? ∀ (𝑎,𝑏) ∈ ℤ 𝑎 + 𝑏 = 𝑏 + 𝑎 ∀ (𝑎,𝑏,𝑐) ∈ ℤ (𝑎 + 𝑏) + 𝑐 = 𝑎 + (𝑏 + 𝑐) ∀ 𝑎 ∈ ℤ 𝑎 + 0 = 𝑎 ℤ = { 0, 1, 2, 3, 4, 5, … } 26
27. Hello world… Addition (Property based 1⁄ 4 ) @Test fun `addition is commutative`( @IntForgery a: Int, @IntForgery b: Int ) { val add_a_b = calculator.add(a, b) val add_b_a = calculator.add(b, a) assertThat(add_a_b).isEqualTo(add_b_a) } 27
28. Hello world… Addition (Property based 2⁄ 4 ) @Test fun property_associativity( @IntForgery a: Int, @IntForgery b: Int, @IntForgery c: Int ) { val add_left = calculator.add(calculator.add(a, b), c) val add_right = calculator.add(a, calculator.add(b, c)) assertThat(add_left).isEqualTo(add_right) } 28
29. Hello world… Addition (Property based 3⁄ 4 ) @Test fun property_identity(@IntForgery a: Int) { val res = calculator.add(a, 0) assertThat(res).isEqualTo(a) } 29
30. Hello world… Addition (Property based 4⁄ 4 ) @Test fun property_successor(@IntForgery a: Int) { val add_a_2 = calculator.add(a, 2) val add_a_1_1 = calculator.add(calculator.add(a, 1), 1) assertThat(add_a_2).isEqualTo(add_a_1_1) } 30
31. Frequently Asked Questions
32. 32 How many times do I have to run my tests? A lot of times; but your CI can help you with that
33. 33 Won’t that create aky tests? Yes! And that’s a good thing! Flaky tests mean your input range boundaries are not perfect (yet).
34. Going Further
35. Shrinking & Falsifying shrinking /ˈʃɹɪnkɪŋ/ n. feature of some property testing or fuzzing libraries to run the tests until it finds the exact boundaries of valid/invalid inputs. falsifying /ˈfɔlsɪfaɪɪŋ/ n. feature of some property testing or fuzzing libraries to run the tests until it finds at least one invalid input. 35
36. “You need to be as confident in the tests you code as you are in the code you test.” 36
37. Thanks! Any questions? ✘ @xgouchet ✘ @datadoghq 37 Presentation template by SlidesCarnival · Icons by Aleksandra Wolska @tutsii159