2
Xavier F. Gouchet
Senior Software Engineer at Datadog
@xgouchet / @datadoghq
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
Goal for the next 80 minutes
Why?
Let’s understand
motivation behind it.
How does it work?
Let’s dig into the
theory and principles.
Concrete examples
a.k.a. the part when
you are working and I
can just relax… or
not
4
What’s wrong with that?
You’re only testing the same values
… Every
… single
… time!
10
Slide 11
Slide 11 text
Fixture is there to make the tests pass
Are you really
trying to make
your code break?
Are you sure you
tested all
possible invalid
values?
11
Slide 12
Slide 12 text
Humans are bad at generating “randomness”
12
Credit: XKCD
Slide 13
Slide 13 text
We don’t know why the inputs are invalid
13
// What’s wrong with this value ?
127.0.0.1 — james [09/May/2018:16:00:39
+0000] "GET /report HTTP/1.0" 200 123
Fuzzy Testing
fuzzy testing /ˈfʌzɪ tɛstɪŋ/
n. feeding a feature (function, program,
etc.) data from a large corpus, possibly
dynamically generated, in order to see
whether or not it fails.
16
Slide 17
Slide 17 text
Property (Based) Testing
property testing /ˈprɑpɚtɪ tɛstɪŋ/
n. testing method relying on the abstract
properties of a feature, rather than on a
short list of sample case.
17
Slide 18
Slide 18 text
What is a property
property /ˈprɑpɚtɪ/
n. a functional property which is part of a
feature under test. (≠ class property).
18
Slide 19
Slide 19 text
Describe the properties of features in the test
● Input boundaries
○ For any int between 0 and 128
○ For any float number
○ For any string representing an email
● Describe expectations based on
the input
● Generate random inputs each run
19
Slide 20
Slide 20 text
Pros
● Abstraction: property instead of
value
● Coverage: generate true* random
data
● Confidence: make your tests
great again
20
Slide 21
Slide 21 text
Cons
● Put more effort in writing tests
● Needs more effort to fix flaky
tests
21
Slide 22
Slide 22 text
Random data caveat
● Define the range of valid and
invalid inputs
○ Valid input range can be smaller than
you think
○ Invalid input can take more form than
you think
○ When you ask for an input as a String,
you’re never expecting it to be
“𝘈Ḇ𝖢𝕯٤Ḟԍ𝕙Ǐ𝙅ḲԸⲘ𝙉০Ρ𝗤Ɍ𝓢ȚЦṼѠ𝓧ƳȤ”
22
Slide 23
Slide 23 text
Hands on
Slide 24
Slide 24 text
Hello world… Addition (the old way)
@Test fun test_addition() {
val a = 3
val b = 7
val res = calculator.add(a, b)
assertThat(res).isEqualTo(10)
}
24
Slide 25
Slide 25 text
Hello world… Addition (Fuzzy)
@Test fun test_addition(
@IntForgery a: Int,
@IntForgery b: Int
) {
val res = calculator.add(a, b)
assertThat(res).isEqualTo(a + b)
}
25
Hello world… Addition (Property based 1⁄
4
)
@Test fun `addition is commutative`(
@IntForgery a: Int,
@IntForgery b: Int
) {
val add_a_b = calculator.add(a, b)
val add_b_a = calculator.add(b, a)
assertThat(add_a_b).isEqualTo(add_b_a)
}
27
Slide 28
Slide 28 text
Hello world… Addition (Property based 2⁄
4
)
@Test fun property_associativity(
@IntForgery a: Int,
@IntForgery b: Int,
@IntForgery c: Int
) {
val add_left = calculator.add(calculator.add(a, b), c)
val add_right = calculator.add(a, calculator.add(b, c))
assertThat(add_left).isEqualTo(add_right)
}
28
Slide 29
Slide 29 text
Hello world… Addition (Property based 3⁄
4
)
@Test fun property_identity(@IntForgery a: Int) {
val res = calculator.add(a, 0)
assertThat(res).isEqualTo(a)
}
29
Slide 30
Slide 30 text
Hello world… Addition (Property based 4⁄
4
)
@Test fun property_successor(@IntForgery a: Int) {
val add_a_2 = calculator.add(a, 2)
val add_a_1_1 = calculator.add(calculator.add(a, 1), 1)
assertThat(add_a_2).isEqualTo(add_a_1_1)
}
30
Slide 31
Slide 31 text
Frequently Asked Questions
Slide 32
Slide 32 text
32
How many times do I
have to run my tests?
A lot of times; but your CI can
help you with that
Slide 33
Slide 33 text
33
Won’t that create
aky tests?
Yes!
And that’s a good thing!
Flaky tests mean your input
range boundaries are
not perfect (yet).
Slide 34
Slide 34 text
Going Further
Slide 35
Slide 35 text
Shrinking & Falsifying
shrinking /ˈʃɹɪnkɪŋ/
n. feature of some property testing or
fuzzing libraries to run the tests until it
finds the exact boundaries of valid/invalid
inputs.
falsifying /ˈfɔlsɪfaɪɪŋ/
n. feature of some property testing or
fuzzing libraries to run the tests until it
finds at least one invalid input.
35
Slide 36
Slide 36 text
“You need to be as confident
in the tests you code as you are in
the code you test.”
36
Slide 37
Slide 37 text
Thanks!
Any questions?
✘ @xgouchet
✘ @datadoghq
37
Presentation template by SlidesCarnival · Icons by Aleksandra Wolska @tutsii159