Introduction to Property-Based Testing @ COSCUP 2022

Transcript

1. Introduction to Property-Based Testing cybai @ COSCUP 2022
2. 2 sli.do #coscup22-fp
3. • @CYBAI • @_cybai • Front-End Engineer @ Autify, Inc. • Web Front-End by day, learning Functional Programming by night. whoami 3
4. Agenda 1. What is Property-Based Testing? 2. More examples for Property-Based Testing 3. When to Example-Based Testing? 4. Advanced topics? 5. Q & A 4 Note: PBT refers to Property-Based Testing in this talk.
5. What is Property-Based Testing? 5
6. What is Property-Based Testing? 6
7. What is Property? 7
8. A property is an invariant of your testing target. What is Property? 8
9. What is Property? 9 A property is an invariant of your testing target. What does that mean? Let’s start from the traditional example-based testing.
10. What is Property-Based Testing? (Cont.) 10 Haskell: QuickCheck, Hedgehog Rust: quickcheck, proptest Erlang: PropEr JavaScript: fast-check, jsverify Python: Hypothesis OCaml: popper … and many more PBT frameworks for different languages
12. The function under test 12
13. The function under test Sample Inputs 13
14. The function under test Expected Outputs 14 Sample Inputs
15. What is Property-Based Testing? (Cont.) 15 E.g. [1,2,3], [0], [], [1,2,3,...,100]
16. What is Property-Based Testing? (Cont.) A test case should be like a proof for the invariant for the testing target. 16
17. A test case should be like a proof for the invariant for the testing target. In the example of `reverse` function, the invariant here is “An input array should be kept as same if it’s reversed twice”. What is Property-Based Testing? (Cont.) 17
18. What is Property-Based Testing? (Cont.) 1. Arbitraries 2. Runners 3. Shrinkers 18
19. What is Property-Based Testing? (Cont.) 1. Arbitraries 2. Runners 3. Shrinkers fc.integer() fc.webUrl() fc.oneof() fc.record() fc.boolean() fc.nat() fc.string() .filter() fc.option() .map() 19 Generate random values
20. What values will fast-check Arbitraries generate? 20 fc.integer() => -1000, -10, …, 0, 1, 3, …, 100, … fc.nat() => 0, 1, 2, …, 100000, …
21. What values will fast-check Arbitraries generate? 21 fc.integer() => -1000, -10, …, 0, 1, 3, …, 100, … fc.boolean() => true, false fc.nat() => 0, 1, 2, …, 100000, … fc.string() => "", "C0SCup", "p#@91-+", …
22. What values will fast-check Arbitraries generate? 22 fc.integer() => -1000, -10, …, 0, 1, 3, …, 100, … fc.oneof(fc.nat(), fc.char()) => Use one of the given arbitraries fc.boolean() => true, false fc.nat() => 0, 1, 2, …, 100000, … fc.string() => "", "C0SCup", "p#@91-+", … .filter() => precondition for the arbitrary (e.g. n < 100) fc.array(fc.nat()) => [], [1], [0,2,2,3], …, [3,5, …, 10,100]
23. What values will fast-check Arbitraries generate? 23 fc.integer() => -1000, -10, …, 0, 1, 3, …, 100, … fc.webUrl() => "https://lo.ui", "https://4.xlm", … fc.oneof(fc.nat(), fc.char()) => Use one of the given arbitraries fc.record({ age: fc.nat(), name: fc.string() }) => Arbitrary for given object model, e.g. { age: 2, name: "John" } fc.boolean() => true, false fc.nat() => 0, 1, 2, …, 100000, … fc.string() => "", "C0SCup", "p#@91-+", … .filter() => precondition for the arbitrary (e.g. n < 100) fc.array(fc.nat()) => [], [1], [0,2,2,3], …, [3,5, …, 10,100] fc.option(fc.nat()) => null, 0, 1, …, 100, …, 100000, …
24. What is Property-Based Testing? (Cont.) 1. Arbitraries 2. Runners 3. Shrinkers Run tests for properties and validate specified predicates. Most of PBT frameworks default to 100 number of runs but you can change it. 24
25. What is Property-Based Testing? (Cont.) 1. Arbitraries 2. Runners 3. Shrinkers Teach the PBT framework how to shrink the failed test case. This would be helpful for knowing smallest possible case. (but not every PBT framework supports this) 25
26. More examples for Property-Based Testing 26
27. More examples - reverse 27
28. More examples - reverse (Cont.) 28
29. More examples - reverse (Cont.) 29
30. More examples - reverse (Cont.) 30 shrunk times and shrunk test case
31. More examples - Calculate Shipping Fee 31
32. More examples - Calculate Shipping Fee (Cont.) 32
33. More examples - Calculate Shipping Fee (Cont.) 33
34. More examples - Calculate Shipping Fee (Cont.) 34
35. More examples for Property-Based Testing You can find more examples on Advent of PBT 2021 created by the author of fast-check. 35
36. When to Example-Based Testing? You can still write unit tests in example-based testing way to make people easier to understand how to use them. I’d highly recommend test them with `doctest`. 36
37. When to Example-Based Testing? Let’s see how `doctest` is utilized in Rust. By default, code block in doc comments can be tested by `cargo test` or `rustdoc --test`. So, it can be tested as a part of document. 37 The calculate shipping fee example in Rust
38. Advanced Topics? 38
39. 1. PBT for End-to-End Testing (like this post) Advanced Topics? 39
40. 1. PBT for End-to-End Testing (like this post) 2. Model-based testing (like XState testing) Advanced Topics? 40
41. 1. PBT for End-to-End Testing (like this post) 2. Model-based testing (like XState testing) 3. PBT for Web Application (like Quickstrom) Advanced Topics? 41
42. Advanced Topics? 1. PBT for End-to-End Testing (like this post) 2. Model-based testing (like XState testing) 3. PBT for Web Application (like Quickstrom) 4. How to specify it! 42
43. Advanced Topics? 1. PBT for End-to-End Testing 2. Model-based testing 3. PBT for Web Application (like Quickstrom) 4. How to specify it! 43
44. Advanced Topics? 1. PBT for End-to-End Testing (like this post) 2. Model-based testing (like XState testing) 3. PBT for Web Application (like Quickstrom) 4. How to specify it! 5. ... and maybe some more I don’t know yet! 44
45. Test properties (invariants) with property-based testing and move example-based testing as document with `doctest`. Conclusion 45
46. Test properties (invariants) with property-based testing and move example-based testing as document with `doctest`. Let’s try to prove our programs by Property-Based Testing! Conclusion 46
47. Q & A 47