Slide 1

Slide 1 text

Property Based Testing In Practice

Slide 2

Slide 2 text

2 Xavier F. Gouchet Senior Software Engineer at Datadog @xgouchet / @datadoghq

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Goal for the next 80 minutes Why? Let’s understand motivation behind it. How does it work? Let’s dig into the theory and principles. Concrete examples a.k.a. the part when you are working and I can just relax… or not 4

Slide 5

Slide 5 text

https://github.com/xgouchet/PBTIP 5

Slide 6

Slide 6 text

Introduction

Slide 7

Slide 7 text

What’s wrong with that? @ParameterizedTest @ValueSource(strings = [ "127.0.0.1 - alice [] \"GET /report HTTP/1.0\" 200 123", "127.0.0.1 - bob [09/May/2018:16:00:39 +0000] \"GET HTTP/1.0\" 200 123", "127.0.0.1 - cat [09/May/2018:16:00:39 +0000] \"GET /report HTTP/1.0" ]) fun parser_must_throw_for_invalid_logs(rawLog: String) { shouldThrow { LogParser(line = rawLog).parse() } } 7

Slide 8

Slide 8 text

What’s wrong with that? @ParameterizedTest @ValueSource(strings = [ "127.0.0.1 - alice [] \"GET /report HTTP/1.0\" 200 123", "127.0.0.1 - bob [09/May/2018:16:00:39 +0000] \"GET HTTP/1.0\" 200 123", "127.0.0.1 - cat [09/May/2018:16:00:39 +0000] \"GET /report HTTP/1.0" ]) fun parser_must_throw_for_invalid_logs(rawLog: String) { shouldThrow { LogParser(line = rawLog).parse() } } 8

Slide 9

Slide 9 text

What’s wrong with that? @ParameterizedTest @ValueSource(strings = [ "127.0.0.1 - alice [] \"GET /report HTTP/1.0\" 200 123", "127.0.0.1 - bob [09/May/2018:16:00:39 +0000] \"GET HTTP/1.0\" 200 123", "127.0.0.1 - cat [09/May/2018:16:00:39 +0000] \"GET /report HTTP/1.0" ]) fun parser_must_throw_for_invalid_logs(rawLog: String) { shouldThrow { LogParser(line = rawLog).parse() } } 9

Slide 10

Slide 10 text

What’s wrong with that? You’re only testing the same values … Every … single … time! 10

Slide 11

Slide 11 text

Fixture is there to make the tests pass Are you really trying to make your code break? Are you sure you tested all possible invalid values? 11

Slide 12

Slide 12 text

Humans are bad at generating “randomness” 12 Credit: XKCD

Slide 13

Slide 13 text

We don’t know why the inputs are invalid 13 // What’s wrong with this value ? 127.0.0.1 — james [09/May/2018:16:00:39 +0000] "GET /report HTTP/1.0" 200 123

Slide 14

Slide 14 text

14 Feature Compliance Input Coverage Compiler Static Analysis Monkey Testing Unit Tests Integration Tests

Slide 15

Slide 15 text

Property Based Testing

Slide 16

Slide 16 text

Fuzzy Testing fuzzy testing /ˈfʌzɪ tɛstɪŋ/ n. feeding a feature (function, program, etc.) data from a large corpus, possibly dynamically generated, in order to see whether or not it fails. 16

Slide 17

Slide 17 text

Property (Based) Testing property testing /ˈprɑpɚtɪ tɛstɪŋ/ n. testing method relying on the abstract properties of a feature, rather than on a short list of sample case. 17

Slide 18

Slide 18 text

What is a property property /ˈprɑpɚtɪ/ n. a functional property which is part of a feature under test. (≠ class property). 18

Slide 19

Slide 19 text

Describe the properties of features in the test ● Input boundaries ○ For any int between 0 and 128 ○ For any float number ○ For any string representing an email ● Describe expectations based on the input ● Generate random inputs each run 19

Slide 20

Slide 20 text

Pros ● Abstraction: property instead of value ● Coverage: generate true* random data ● Confidence: make your tests great again 20

Slide 21

Slide 21 text

Cons ● Put more effort in writing tests ● Needs more effort to fix flaky tests 21

Slide 22

Slide 22 text

Random data caveat ● Define the range of valid and invalid inputs ○ Valid input range can be smaller than you think ○ Invalid input can take more form than you think ○ When you ask for an input as a String, you’re never expecting it to be “𝘈Ḇ𝖢𝕯٤Ḟԍ𝕙Ǐ𝙅ḲԸⲘ𝙉০Ρ𝗤Ɍ𝓢ȚЦṼѠ𝓧ƳȤ” 22

Slide 23

Slide 23 text

Hands on

Slide 24

Slide 24 text

Hello world… Addition (the old way) @Test fun test_addition() { val a = 3 val b = 7 val res = calculator.add(a, b) assertThat(res).isEqualTo(10) } 24

Slide 25

Slide 25 text

Hello world… Addition (Fuzzy) @Test fun test_addition( @IntForgery a: Int, @IntForgery b: Int ) { val res = calculator.add(a, b) assertThat(res).isEqualTo(a + b) } 25

Slide 26

Slide 26 text

Let’s talk about math… What is ‘addition‘? ∀ (𝑎,𝑏) ∈ ℤ 𝑎 + 𝑏 = 𝑏 + 𝑎 ∀ (𝑎,𝑏,𝑐) ∈ ℤ (𝑎 + 𝑏) + 𝑐 = 𝑎 + (𝑏 + 𝑐) ∀ 𝑎 ∈ ℤ 𝑎 + 0 = 𝑎 ℤ = { 0, 1, 2, 3, 4, 5, … } 26

Slide 27

Slide 27 text

Hello world… Addition (Property based 1⁄ 4 ) @Test fun `addition is commutative`( @IntForgery a: Int, @IntForgery b: Int ) { val add_a_b = calculator.add(a, b) val add_b_a = calculator.add(b, a) assertThat(add_a_b).isEqualTo(add_b_a) } 27

Slide 28

Slide 28 text

Hello world… Addition (Property based 2⁄ 4 ) @Test fun property_associativity( @IntForgery a: Int, @IntForgery b: Int, @IntForgery c: Int ) { val add_left = calculator.add(calculator.add(a, b), c) val add_right = calculator.add(a, calculator.add(b, c)) assertThat(add_left).isEqualTo(add_right) } 28

Slide 29

Slide 29 text

Hello world… Addition (Property based 3⁄ 4 ) @Test fun property_identity(@IntForgery a: Int) { val res = calculator.add(a, 0) assertThat(res).isEqualTo(a) } 29

Slide 30

Slide 30 text

Hello world… Addition (Property based 4⁄ 4 ) @Test fun property_successor(@IntForgery a: Int) { val add_a_2 = calculator.add(a, 2) val add_a_1_1 = calculator.add(calculator.add(a, 1), 1) assertThat(add_a_2).isEqualTo(add_a_1_1) } 30

Slide 31

Slide 31 text

Frequently Asked Questions

Slide 32

Slide 32 text

32 How many times do I have to run my tests? A lot of times; but your CI can help you with that

Slide 33

Slide 33 text

33 Won’t that create aky tests? Yes! And that’s a good thing! Flaky tests mean your input range boundaries are not perfect (yet).

Slide 34

Slide 34 text

Going Further

Slide 35

Slide 35 text

Shrinking & Falsifying shrinking /ˈʃɹɪnkɪŋ/ n. feature of some property testing or fuzzing libraries to run the tests until it finds the exact boundaries of valid/invalid inputs. falsifying /ˈfɔlsɪfaɪɪŋ/ n. feature of some property testing or fuzzing libraries to run the tests until it finds at least one invalid input. 35

Slide 36

Slide 36 text

“You need to be as confident in the tests you code as you are in the code you test.” 36

Slide 37

Slide 37 text

Thanks! Any questions? ✘ @xgouchet ✘ @datadoghq 37 Presentation template by SlidesCarnival · Icons by Aleksandra Wolska @tutsii159