Generating Your Assumptions

Tests are critical for verifying that we’re building the software we want; however, a good test suite is hard to achieve. Setting up scenarios is boring, repetitive, and time consuming. Not to mention we have to think of all these edge cases before our users exploit them.

Property-based testing introduces developers to a new way of thinking about testing. We can create thousands of clever tests with just a small amount of code. We can then spend our time and effort effectively to help make sure the assumptions of our software hold true.

Talk Recording: https://youtu.be/cVHOhTv25AU

Transcript

1. Generating Your Assumptions Michael Torres @mike__torres
2. Why Test? Code confidence Prevent regressions Spend time on features and not on bugs Improve design of code Have happier users
3. it(“gets int value from string") { } Parse Int String
4. it(“gets int value from string") { parseInt("0").shouldBe(0) } Parse Int String
5. it(“gets int value from string") { parseInt("0").shouldBe(0) parseInt("10").shouldBe(10) } Parse Int String
6. it(“gets int value from string") { parseInt("0").shouldBe(0) parseInt("10").shouldBe(10) parseInt("15").shouldBe(15) } Parse Int String
7. it(“gets int value from string") { parseInt("0").shouldBe(0) parseInt("10").shouldBe(10) parseInt("15").shouldBe(15) parseInt("100").shouldBe(100) } Parse Int String
8. it(“gets int value from string") { parseInt("0").shouldBe(0) parseInt("10").shouldBe(10) parseInt("15").shouldBe(15) parseInt("100").shouldBe(100) parseInt("-1").shouldBe(-1) } Parse Int String
9. 2,147,483,647 possible positive 32 bit integers
10. “Program testing can be used to show the presence of bugs, but never to show their absence!” - Edsger Dijkstra
11. Assumptions This works for other similar scenarios This is enough tests Others know my assumptions too
12. Many Scenarios Different inputs Different code paths Different order of operations Different time between operations
13. Write More Scenarios!
14. Write More Scenarios! Repetitive Error prone Boring Requires thought
15. “I don't think that people should write a lot of tests. I think people should run a lot of tests” - Rich Hickey
16. Generate Scenarios Computers are good at this Millions of generations Organize generated tests
17. Parse Int String For all integers in string form, they can be parsed to their value
18. it("parse int string") { assertAll(generateInt()) { int !-> parseInt("$int").shouldBe(int) } } Parse Int String
19. it("parse all int string") { assertAll(generateInt()) { int !-> parseInt("$int").shouldBe(int) } } Parse Int String
20. Property-Based Testing Generate test cases to help verify a property of our system
21. Property
 
 Under specified circumstances, something that is always true
22. Example Properties Sorting should maintain the same size Concatenating two strings should contain both strings Argument order does not matter when adding numbers
23. Expressive Document what you’re really testing More general and less prone to change
24. Edge Cases Empty collections Special characters Min/Max Integers Leap Year
25. Many Test Cases Can generate millions of scenarios for tests Continuously trying to break your assumptions
26. Makes You Think Will this always be true? Think about the code and the requirements
27. Frameworks Haskell - QuickCheck Java - junit-quickcheck C# - FsCheck Scala - ScalaCheck Kotlin - kotlintest … and many more!
28. Generators
29. Gen.int()
 
 Gen.string() Basic Types "" 0 98 -1 "abc" “ae-([&“
30. data class Product(val name: String, val price: Int) fun productGen(): Gen<Product> = Gen.bind( Gen.string(), Gen.positiveIntegers()) { name, price !-> Product(name, price) } fun productsGen(): Gen<List<Product!>> = Gen.list(productGen()) Composable
31. data class Product(val name: String, val price: Int) fun productGen(): Gen<Product> = Gen.bind( Gen.string(), Gen.positiveIntegers()) { name, price !-> Product(name, price) } fun productsGen(): Gen<List<Product!>> = Gen.list(productGen()) Composable
32. describe("existing example test") { val products = productsGen().random().first() !!... } Stubbing
33. Shrinking
34. Shrinking Process of reducing and simplifying generated data upon a failure
35. Product("~T7 KqJST\"+*'@!}iL=;1)Oczb{k\"xDRU^!\\?qfCNt)-", 734113874) Product("~T7 KqJST\"+*'@!}iL=;1)", 734113874) Product("~", 734113874) Shrinking
36. First Test Start with a small simple example Find the most recent valid order
37. it("should get most recent successful order") { assertAll(ordersGen()) { orders !-> } } Most Recent Valid Order
38. it("should get most recent successful order") { assertAll(ordersGen()) { orders !-> val result = mostRecentValidOrder(orders) } } Most Recent Valid Order
39. Common Hurdle Don’t know what to assert against Function is so simple that it’s hard to find a property
40. it("should get most recent successful order") { assertAll(ordersGen()) { orders !-> val result = mostRecentValidOrder(orders) !// ¯\_(ツ)_/¯ val expected = mostRecentValidOrder(orders) result.shouldBe(expected) } } Common Hurdle
41. Patterns
42. Modeling Test against an alternative implementation Implementation should be reliably correct Implementation can be a slower, naive version Implementation can also be a legacy version
43. it("should get most recent successful order”) { assertAll(ordersGen()) { orders !-> val result = mostRecentValidOrder(orders) val expected = orders.filter { order !-> order.isSuccess() } .sortedByDescending { order !-> order.date } .firstOrNull() result.shouldBe(expected) } } Most Recent Valid Order
44. it("should get most recent successful order”) { assertAll(ordersGen()) { orders !-> val result = mostRecentValidOrder(orders) val expected = orders.filter { order !-> order.isSuccess() } .sortedByDescending { order !-> order.date } .firstOrNull() result.shouldBe(expected) } } Most Recent Valid Order
45. Generalization Write some example cases first Generalize those into abstract properties
46. it("should always get the last id") { getLastAddedProductId(listOf(1)).shouldBe(1) getLastAddedProductId(listOf(1, 2, 3)).shouldBe(3) getLastAddedProductId(listOf(5, 4)).shouldBe(4) } Get Last Product Id
47. it("should always get the last id") { val idGen = Gen.positiveIntegers() val idsGen = Gen.list(idGen) assertAll(idsGen, idGen) { productIds, id !-> val allIds = productIds.plus(id) getLastAddedProductId(allIds).shouldBe(id) } } Get Last Product Id
48. it("should always get the last id") { val idGen = Gen.positiveIntegers() val idsGen = Gen.list(idGen) assertAll(idsGen, idGen) { productIds, id !-> val allIds = productIds.plus(id) getLastAddedProductId(allIds).shouldBe(id) } } Get Last Product Id
49. Invariants Breaking down into smaller properties 
 “Strong ropes are built from 
 smaller threads put together”
50. fun checkoutTotal(products: List<Product>, discount: Discount?): Int Checkout Total
51. it("the total should be at least zero") { assertAll(productsGen(), discountCodeGen()) { products, discount !-> val total = calculateTotal(products, discount) total.shouldBeGreaterThanOrEqual(0) } } Checkout Total
52. it("the total should be at least zero") { assertAll(productsGen(), discountCodeGen()) { products, discount !-> val total = calculateTotal(products, discount) total.shouldBeGreaterThanOrEqual(0) } } Checkout Total
53. it("zero items should always have a total of zero") { assertAll(discountCodeGen()) { discount !-> val total = calculateTotal(emptyList(), discount) total.shouldBe(0) } } Checkout Total
54. it("zero items should always have a total of zero") { assertAll(discountCodeGen()) { discount !-> val total = calculateTotal(emptyList(), discount) total.shouldBe(0) } } Checkout Total
55. it("a valid discount code should always reduce the total") { assertAll( nonEmptyProductsGen(), validDiscountCodeGen()) { products, discount !-> val totalWithoutDiscount = calculateTotal(products, null) val totalWithDiscount = calculateTotal(products, discount) totalWithDiscount.shouldBeLessThan(totalWithoutDiscount) } } Checkout Total
56. it("a valid discount code should always reduce the total") { assertAll( nonEmptyProductsGen(), validDiscountCodeGen()) { products, discount !-> val totalWithoutDiscount = calculateTotal(products, null) val totalWithDiscount = calculateTotal(products, discount) totalWithDiscount.shouldBeLessThan(totalWithoutDiscount) } } Checkout Total
57. Symmetry When you have a reversible sequence of actions Applying opposite actions together
58. it("serialized orders should be deserialzed to original") { assertAll(ordersGen()) { orders !-> val result = deserialize(serialize(orders)) result.shouldContainExactly(orders) } } JSON Parsing
59. it("serialized orders should be deserialzed to original") { assertAll(ordersGen()) { orders !-> val result = deserialize(serialize(orders)) result.shouldContainExactly(orders) } } JSON Parsing
60. it("serialized orders should be deserialzed to original") { assertAll(ordersGen()) { orders !-> val serialized = serialize(orders) assertIsJson(serialized) val deserialized = deserialize(serialized) deserialized.shouldContainExactly(orders) } } JSON Parsing
61. Stateful Testing
62. Data Generators generate data Data can be more than just for inputs Data can represent actions
63. class System { fun doSomething() fun update(value: Int) fun getState(): State } Stateful System
64. class DoThis() class Update(value: Int) class GetState() Actions
65. listOf(DoThis(), Update(5), GetState()) listOf(GetState(), Update(1), DoThis(), Update(2)) listOf(GetState(), DoThis()) … Actions
66. fun save(key: Key, value: Value)
 fun remove(key: Key) fun get(key: Key): Value? fun getAll(): Set<Value> fun clear() fun size(): Int Database
67. fun put(key: Key, value: Value)
 fun remove(key: Key) fun get(key: Key): Value? fun values(): MutableCollection<Value> fun clear() fun size(): Int HashMap
68. Model A HashMap has the same operations Simpler store We trust that it works
69. sealed class Action<Key, Value> { class Save<Key, Value>( val key: Key, val value: Value) : Action<Key, Value>() class Remove<Key, Value>(val key: Key) : Action<Key, Value>() class Get<Key, Value>(val key: Key) : Action<Key, Value>() class Clear<Key, Value> : Action<Key, Value>() class Size<Key, Value> : Action<Key, Value>() } Actions
70. fun <Key, Value> actionGen( keyGen: Gen<Key>, valueGen: Gen<Value>): Gen<Action<Key, Value!>> = Gen.oneOf( Gen.bind(keyGen, valueGen) { key, value !-> Action.Save(key, value) }, keyGen.map { key !-> Action.Remove(key) }, keyGen.map { key !-> Action.Get(key) }, Gen.create { Action.GetAll() }, Gen.create { Action.Clear() }, Gen.create { Action.Size() } ) Generator
71. fun <Key, Value> actionGen( keyGen: Gen<Key>, valueGen: Gen<Value>): Gen<Action<Key, Value!>> = Gen.oneOf( Gen.bind(keyGen, valueGen) { key, value !-> Action.Save(key, value) }, keyGen.map { key !-> Action.Remove(key) }, keyGen.map { key !-> Action.Get(key) }, Gen.create { Action.GetAll() }, Gen.create { Action.Clear() }, Gen.create { Action.Size() } ) Generator
72. fun <Key, Value> actionGen( keyGen: Gen<Key>, valueGen: Gen<Value>): Gen<Action<Key, Value!>> = Gen.oneOf( Gen.bind(keyGen, valueGen) { key, value !-> Action.Save(key, value) }, keyGen.map { key !-> Action.Remove(key) }, keyGen.map { key !-> Action.Get(key) }, Gen.create { Action.GetAll() }, Gen.create { Action.Clear() }, Gen.create { Action.Size() } ) Generator
73. fun <Key, Value> actionGen( keyGen: Gen<Key>, valueGen: Gen<Value>): Gen<Action<Key, Value!>> = Gen.oneOf( Gen.bind(keyGen, valueGen) { key, value !-> Action.Save(key, value) }, keyGen.map { key !-> Action.Remove(key) }, keyGen.map { key !-> Action.Get(key) }, Gen.create { Action.GetAll() }, Gen.create { Action.Clear() }, Gen.create { Action.Size() } ) Generator
74. fun <Key, Value> actionGen( keyGen: Gen<Key>, valueGen: Gen<Value>): Gen<Action<Key, Value!>> = Gen.oneOf( Gen.bind(keyGen, valueGen) { key, value !-> Action.Save(key, value) }, keyGen.map { key !-> Action.Remove(key) }, keyGen.map { key !-> Action.Get(key) }, Gen.create { Action.GetAll() }, Gen.create { Action.Clear() }, Gen.create { Action.Size() } ) Generator
75. listOf( Action.Save(2, Order), Action.Clear(), Action.Get(2), Action.Remove(2), Action.Save(1, Order), Action.GetAll(), Action.Size() … ) Generator
76. private fun <Key, Value> runAction( model: MutableMap<Key, Value>, action: Action<Key, Value>) { when (action) { is Action.Save !-> model.put(action.key, action.value) is Action.Remove !-> model.remove(action.key) is Action.Get !-> model.get(action.key) is Action.GetAll !-> model.values is Action.Clear !-> model.clear() is Action.Size !-> model.size } } Runner - Model
77. private fun <Key, Value> runAction( model: MutableMap<Key, Value>, action: Action<Key, Value>) { when (action) { is Action.Save !-> model.put(action.key, action.value) is Action.Remove !-> model.remove(action.key) is Action.Get !-> model.get(action.key) is Action.GetAll !-> model.values is Action.Clear !-> model.clear() is Action.Size !-> model.size } } Runner - Model
78. private fun <Key, Value> runAction( db: DB, action: Action<Key, Value>) { when (action) { is Action.Save !-> db.save(action.key, action.value) is Action.Remove !-> db.remove(action.key) is Action.Get !-> db.get(action.key) is Action.GetAll !-> db.getAll() is Action.Clear !-> db.clear() is Action.Size !-> db.size() } } Runner - Database
79. private fun <Key, Value> runAction( db: DB, action: Action<Key, Value>) { when (action) { is Action.Save !-> db.save(action.key, action.value) is Action.Remove !-> db.remove(action.key) is Action.Get !-> db.get(action.key) is Action.GetAll !-> db.getAll() is Action.Clear !-> db.clear() is Action.Size !-> db.size() } } Runner - Database
80. describe(“saving orders in key value database") { val keyGen = Gen.choose(0, 50) val actionsGen = Gen.list(actionGen(keyGen, ordersGen())) it("should have same values as a map") { assertAll(actionsGen) { actions !-> val model = HashMap<Int, Order>() val db = DB() actions.forEach { action !-> runAction(model, action) runAction(db, action) } db.getAll().shouldContainExactlyInAnyOrder(model.values) } } } Test
81. describe(“saving orders in key value database") { val keyGen = Gen.choose(0, 50) val actionsGen = Gen.list(actionGen(keyGen, ordersGen())) it("should have same values as a map") { assertAll(actionsGen) { actions !-> val model = HashMap<Int, Order>() val db = DB() actions.forEach { action !-> runAction(model, action) runAction(db, action) } db.getAll().shouldContainExactlyInAnyOrder(model.values) } } } Test
82. describe(“saving orders in key value database") { val keyGen = Gen.choose(0, 50) val actionsGen = Gen.list(actionGen(keyGen, ordersGen())) it("should have same values as a map") { assertAll(actionsGen) { actions !-> val model = HashMap<Int, Order>() val db = DB() actions.forEach { action !-> runAction(model, action) runAction(db, action) } db.getAll().shouldContainExactlyInAnyOrder(model.values) } } } Test
83. Bug in the code
84. No Test Unhappy user found it “Doesn’t work. 1 star, 0 if I could” Check analytics or crash reports?
85. Example Test Assertion fails
 Collection should be exactly [Order(id=2, date=2019-06-24, products=[Product(name=, price=2)])] but was [Order(id=1, date=2019-06-24, products=[Product(name=, price=1)])] Debug and figure out why
86. Property Test Assertion fails
 Collection should be exactly [Order(id=2, date=2019-06-24, products=[Product(name=, price=2)])] but was [Order(id=1, date=2019-06-24, products=[Product(name=, price=1)])] See steps from shrinking
 Shrink result !=> [ Save(key=1, value=Order(id=1, date=2019-06-24, products=[Product(name="", price=1)], Remove(key=1), Save(key=2, value=Order(id=2, date=2019-06-24, products=[Product(name="", price=2)] ] fail
87. Perfect Bug Report Exact methods to run In this order With the simplest inputs Shrink result !=> [ Save(key=1, value=Order(id=1, date=2019-06-24, products=[Product(name="", price=1)], Remove(key=1), Save(key=2, value=Order(id=2, date=2019-06-24, products=[Product(name="", price=2)] ] fail
88. Why ever write example-based tests?
89. Example Tests Known edge cases Faster to run Not everything can be expressed as a property Easier to write
90. Success Stories
91. Clojure Bug in core found when converting between immutable and mutable data structures. Fixed in Clojure 1.6 Clojure spec provides out of the box support for property based testing Functional Lisp for the JVM
92. LevelDB Bug found using the model pattern within a few minutes Required 17 steps to reproduce Involved opening and closing the database, adding and deleting the same key Key-Value database from Google
93. js-yaml Over 1 million downloads per day Bug found using symmetry pattern in ~10 lines of code YAML parsing JavaScript library
94. Learn More Property-Based Testing with PropEr, Erlang, and Elixir
 https://propertesting.com/ Don’t Write Tests - John Hughes
 https://www.youtube.com/watch?v=hXnS_Xjwk2Y Generating test cases so you don’t have to
 https://labs.spotify.com/2015/06/25/rapid-check/