Property-based testing in C#

Traditional unit tests rely on carefully crafted examples, but what if you could have your computer generate and verify thousands of test cases for you? Property-based testing is a powerful technique that allows developers to define expected properties of their code and let a testing framework explore edge cases automatically.

This session introduces property-based testing in C#, demonstrating how it differs from unit testing, how it can uncover hidden bugs and when (or when not!) to apply it.

Through practical examples using FsCheck, attendees will learn how to generate random test data, define meaningful properties, and apply property-based testing to real-world scenarios. Whether you’re new to property-based testing or looking to deepen your understanding, this talk will provide actionable insights to improve the reliability and robustness of your code and tests.

Transcript

1. Property-based testing With examples in C# @jovaneyck.bsky.social
7. Replaying failed properties
8. Getting started
9. PBT pro == What input to generate + What properties to write + When (not) to PBT
10. Generating random inputs
11. Generating random inputs
12. Generating random inputs
13. Generating random inputs
14. Constraining generators
15. Combining generators
16. Combining generators
17. DnD generators
18. 1d6?
19. 1d6!
20. 1d6 distribution
21. 2d6?
22. !2d6
24. 2d6!
25. 2d6 v2
26. PBT pro == What input to generate + What properties to write + When (not) to PBT
27. Heuristics for “good properties”
28. generalizing unit tests John Hughes - Building on developers' intuitions (...) | Lambda Days 19
29. Generalizing unit tests
46. Extracting domains from an email address
47. Start at the end
48. Serialization/deserialization
49. Serialization/deserialization
50. Inverse operations There and back again
51. Fuzzing You can throw anything at it
52. Commutativity Different paths, same destination https://fsharpforfunandprofit.com/posts/property-based-testing-2/
53. Invariants Some things never change https://fsharpforfunandprofit.com/posts/property-based-testing-2/
54. Idempotence The more things change, the more they stay the same https://fsharpforfunandprofit.com/posts/property-based-testing-2/
55. Let’s build Google
56. Let’s build Google
57. Let’s build Google
58. Metamorphic testing
59. Heuristics for good properties • Generalized unit tests • “Start at the end” • Inverse operations: “There and back again” • Fuzzing: “You can throw everything at it” • Commutativity: “Different paths, same destination” • Invariants: “Some things never change” • Idempotence: “The more things change, the more they stay the same” • Metamorphic properties
60. PBT pro == What input to generate + What properties to write + When (not) to PBT
62. Effort Confidence
63. When to PBT • When high confidence is worth the effort ◦ Critical code • Problems with very large input/state spaces ◦ Parsers • Data structures, DDD aggregates/VO’s, monads ( ) ◦ anything with clear and complete laws or invariants • Fuzzy/non-deterministic problems ◦ might not have deterministic or knowable input->output relationships ◦ e.g. classifiers, LLM’s • Legacy code ◦ Writing properties will quickly (in)validate mental model of the SUT • Stateful systems that can be modeled
64. When to PBT Mysteries of Dropbox, Property-based testing of a Distributed Synchronization Service, John Hughes et al.
65. When to PBT Delhaize rekent al maanden 'spookproducten' aan bij sommige klanten: "Ik moest betalen voor bak trappist die ik nooit kocht"
66. When (not) to PBT • Test-Driven Development ◦ It’s possible & fun BUT ◦ Often even harder to do PBT in combination with TDD due to the friction between “universal properties’ and “incremental development”
67. Criteria Unit test Property Effort easy to write needs more cases you need to list them takes more effort to come up with good properties and generators One property covers lots (and lots) of cases Execution speed Fast 100/10.000x slower Bugs they catch depends on the test list edge cases, “+1 QA” in team Document the system examples business rules and invariants Keeping test code focused test data builders, custom assertions generators and properties Unit test vs. property-based test
68. Jessica Kerr’s thoughts on property-based testing “See how much thought goes into a property-based test? and this is a simple specification! I don’t recommend writing these for all your code – only the really important stuff.” “Property-based tests are best combined with example-based tests. Examples help you start organizing your thoughts, and they’re easier for future-you to read and understand when you come back to this code later. Humans think in examples. Programs don’t extrapolate. Property-based thinking and property-based testing can bridge between us and the computer. Math, it’s a tool.” https://jessitron.com/2013/04/25/property-based-testing-what-is-it/
69. OO tax
70. OO tax
71. PBT pro == What input to generate + What properties to write + When (not) to PBT
72. DIY 2
73. DIY 2 tests
74. DIY https://kata-log.rocks/mars-rover-kata https://bit.ly/pbt-workshop
75. Property-based testing With examples in C# @jovaneyck.bsky.social