Property Based Testing

Property-based testing is an approach to testing that involves checking that a system meets certain expected properties. The approach is frequently promoted as a desired technique when adopting a functional style of programming. It typically involves guiding the generation of large data sets using a generator framework which can be much less work than coding large test suites by hand. This talk looks at the concepts behind this approach and some of the available libraries. The examples are mostly in Groovy but also in a few other JVM languages. The concepts apply across all languages. Property-based testing libraries are available for most languages.

Transcript

1. objectcomputing.com © 2020, Object Computing, Inc. (OCI). All rights reserved. No part of these notes may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior, written permission of Object Computing, Inc. (OCI) An introduction to Property-based Testing (with Groovy & other languages) Dr Paul King OCI Groovy Lead V.P. and PMC Chair Apache Groovy @paulk_asert © 2020 Object Computing, Inc. (OCI). All rights reserved. objectcomputing.com
2. Dr Paul King OCI Groovy Lead V.P. and PMC Chair Apache Groovy Slides: https://speakerdeck.com/paulk/property-based-testing Examples repo: https://github.com/paulk-asert/property-based-testing ReGinA author: https://www.manning.com/books/groovy-in-action-second-edition Twitter: @paulk_asert
3. Friends of Apache Groovy Open Collective
4. © paulk_asert 2006-2020 What is property-based testing • An approach to derive automatic test cases for some system • To reduce the effort required to create manual test cases • To gather additional cases that we might miss by hand • Importantly, with a view to confirming desirable properties about the system under test
5. © paulk_asert 2006-2020 Motivating example public class MathUtil { public static int sumBiggestPair(int a, int b, int c) { int op1 = a; int op2 = b; if (c > a) { op1 = c; } else if (c > b) { op2 = c; } return op1 + op2; } private MathUtil(){} }
6. © paulk_asert 2006-2020 Motivating example import org.junit.Test import static util.MathUtil.sumBiggestPair class MathUtilTest { @Test void examples() { assert sumBiggestPair(5, 4, 1) == 9 assert sumBiggestPair(4, 5, 1) == 9 assert sumBiggestPair(5, 9, 6) == 15 } }
7. © paulk_asert 2006-2020 Motivating example import org.junit.Test import static util.MathUtil.sumBiggestPair class MathUtilTest { @Test void examples() { assert sumBiggestPair(5, 4, 1) == 9 assert sumBiggestPair(4, 5, 1) == 9 assert sumBiggestPair(5, 9, 6) == 15 } } Tests pass but not 100% coverage, so let’s add some more test cases.
8. © paulk_asert 2006-2020 Motivating example import org.junit.Test import static util.MathUtil.sumBiggestPair class MathUtilTest { @Test void examples() { assert sumBiggestPair(5, 4, 1) == 9 assert sumBiggestPair(4, 5, 1) == 9 assert sumBiggestPair(5, 9, 6) == 15 assert sumBiggestPair(10, 2, 6) == 16 } }
9. © paulk_asert 2006-2020 Motivating example import org.junit.Test import static util.MathUtil.sumBiggestPair class MathUtilTest { @Test void examples() { assert sumBiggestPair(5, 4, 1) == 9 assert sumBiggestPair(4, 5, 1) == 9 assert sumBiggestPair(5, 9, 6) == 15 assert sumBiggestPair(10, 2, 6) == 16 } }
10. © paulk_asert 2006-2020 Motivating example import org.junit.Test import static util.MathUtil.sumBiggestPair class MathUtilTest { @Test void examples() { assert sumBiggestPair(5, 4, 1) == 9 assert sumBiggestPair(4, 5, 1) == 9 assert sumBiggestPair(5, 9, 6) == 15 assert sumBiggestPair(10, 2, 6) == 16 } } 100% coverage, so it must be bug free, right? Let’s ship it!
11. © paulk_asert 2006-2020 Motivating example import org.junit.Test import static util.MathUtil.sumBiggestPair class MathUtilTest { @Test void examples() { assert sumBiggestPair(5, 4, 1) == 9 assert sumBiggestPair(4, 5, 1) == 9 assert sumBiggestPair(5, 9, 6) == 15 assert sumBiggestPair(10, 2, 6) == 16 // 100% assert sumBiggestPair(2, 5, 6) == 11 assert sumBiggestPair(5, 2, 6) == 11 } } But we’ll just add a couple more tests.
12. © paulk_asert 2006-2020 Motivating example import org.junit.Test import static util.MathUtil.sumBiggestPair class MathUtilTest { @Test void examples() { assert sumBiggestPair(5, 4, 1) == 9 assert sumBiggestPair(4, 5, 1) == 9 assert sumBiggestPair(5, 9, 6) == 15 assert sumBiggestPair(10, 2, 6) == 16 // 100% assert sumBiggestPair(2, 5, 6) == 11 assert sumBiggestPair(5, 2, 6) == 11 } }
13. © paulk_asert 2006-2020 Motivating example public class MathUtil { public static int sumBiggestPair(int a, int b, int c) { int op1 = a; int op2 = b; if (c > a) { op1 = c; } else if (c > b) { op2 = c; } return op1 + op2; } private MathUtil(){} } Else branch logic is flawed for case where c is biggest and b is smallest.
14. © paulk_asert 2006-2020 Motivating example public class MathUtil { public static int sumBiggestPair(int a, int b, int c) { int op1 = a; int op2 = b; if (c > Math.min(a, b)) { op1 = c; op2 = Math.max(a, b); } return op1 + op2; } private MathUtil(){} } One way to correct the logic.
15. Motivating example Solution? • Write more tests? • Expensive and potentially hinders agile refactoring • How do I know when to stop? • Spend more time thinking about tests
16. Motivating example Solution? • Write more tests? • Expensive and potentially hinders agile refactoring • How do I know when to stop? • Spend more time thinking about tests Alternative solution? • Write less tests! • Spend more time thinking about system properties
17. © paulk_asert 2006-2020 Motivating example import org.junit.Test import static util.MathUtil.sumBiggestPair class MathUtilTest { @Test void examples() { assert sumBiggestPair(5, 4, 1) == 9 assert sumBiggestPair(4, 5, 1) == 9 assert sumBiggestPair(5, 9, 6) == 15 assert sumBiggestPair(10, 2, 6) == 16 assert sumBiggestPair(2, 5, 6) == 11 assert sumBiggestPair(5, 2, 6) == 11 } } Goal will be to replace input values with randomly generated values But how do we get the output value?
18. © paulk_asert 2006-2020 Motivating example import net.jqwik.api.ForAll import net.jqwik.api.Property import static util.MathUtil.sumBiggestPair class MathUtilTest { @Property void auto(@ForAll Short a, @ForAll Short b, @ForAll Short c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() } } Jqwik uses annotations to guide where randomly generated values will be used and what those values might be. For this case, we can find an alternative way to calculate the answer
19. © paulk_asert 2006-2020 Motivating example import net.jqwik.api.ForAll import net.jqwik.api.Property import static util.MathUtil.sumBiggestPair class MathUtilTest { @Property void auto(@ForAll Short a, @ForAll Short b, @ForAll Short c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() } } |--------------------jqwik-------------------- tries = 12 | # of calls to property checks = 12 | # of not rejected calls generation-mode = RANDOMIZED | parameters are randomly generated after-failure = PREVIOUS_SEED | use the previous seed seed = 6313771727151213536 | random seed to reproduce generated values sample = [-98, -99, 0] original-sample = [-111, -13803, -97] For faulty case
20. © paulk_asert 2006-2020 Motivating example import net.jqwik.api.ForAll import net.jqwik.api.Property import static util.MathUtil.sumBiggestPair class MathUtilTest { @Property void auto(@ForAll Short a, @ForAll Short b, @ForAll Short c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() } } |--------------------jqwik-------------------- tries = 1000 | # of calls to property checks = 1000 | # of not rejected calls generation-mode = RANDOMIZED | parameters are randomly generated after-failure = PREVIOUS_SEED | use the previous seed seed = 6313771727151213536 | random seed to reproduce generated values For fixed case
21. © paulk_asert 2006-2020 Motivating example import net.jqwik.api.ForAll import net.jqwik.api.Property import static util.MathUtil.sumBiggestPair class MathUtilTest { @Property void auto(@ForAll Short a, @ForAll Short b, @ForAll Short c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() } } So, it must be bug free now. It passes 1000 tests! Let’s ship it!
22. © paulk_asert 2006-2020 Motivating example import net.jqwik.api.ForAll import net.jqwik.api.Property import static util.MathUtil.sumBiggestPair class MathUtilTest { @Property void auto(@ForAll Integer a, @ForAll Integer b, @ForAll Integer c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() } } Replace Short with Integer
23. © paulk_asert 2006-2020 Motivating example import net.jqwik.api.ForAll import net.jqwik.api.Property import static util.MathUtil.sumBiggestPair class MathUtilTest { @Property void auto(@ForAll Integer a, @ForAll Integer b, @ForAll Integer c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() } } |--------------------jqwik-------------------- tries = 10 | # of calls to property checks = 10 | # of not rejected calls generation-mode = RANDOMIZED | parameters are randomly generated after-failure = PREVIOUS_SEED | use the previous seed seed = -1317849176950590163 | random seed to reproduce generated values sample = [200764882, 1946718766, 0] original-sample = [1942642352, 1946718766, -368]
24. © paulk_asert 2006-2020 Motivating example import net.jqwik.api.ForAll import net.jqwik.api.Property import static util.MathUtil.sumBiggestPair class MathUtilTest { @Property void auto(@ForAll Integer a, @ForAll Integer b, @ForAll Integer c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() } } |--------------------jqwik-------------------- tries = 10 | # of calls to property checks = 10 | # of not rejected calls generation-mode = RANDOMIZED | parameters are randomly generated after-failure = PREVIOUS_SEED | use the previous seed seed = -1317849176950590163 | random seed to reproduce generated values sample = [200764882, 1946718766, 0] original-sample = [1942642352, 1946718766, -368] Bug with code or property test?
25. © paulk_asert 2006-2020 Motivating example • No correct answer • Check with “customer” • Here we’ll decide that overflow is a “feature” • So, we’ll fix the test. We have several options.
26. © paulk_asert 2006-2020 Motivating example • We can limit the values fed into sumBiggestPair • We already saw this with using Short params import net.jqwik.api.ForAll import net.jqwik.api.Property import static util.MathUtil.sumBiggestPair class MathUtilTest { @Property void auto(@ForAll Short a, @ForAll Short b, @ForAll Short c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() } }
27. © paulk_asert 2006-2020 Motivating example • We can limit the values fed into sumBiggestPair • But we could also use an annotation like here: @Property void checkIntegerConstrained(@ForAll @IntRange(min = -1_000_000_000, max = 1_000_000_000) Integer a, @ForAll @IntRange(min = -1_000_000_000, max = 1_000_000_000) Integer b, @ForAll @IntRange(min = -1_000_000_000, max = 1_000_000_000) Integer c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() }
28. © paulk_asert 2006-2020 Motivating example • We can limit the values fed into sumBiggestPair • But we could also use a custom provider like here: @Property void checkIntegerConstrainedProvider(@ForAll('halfMax') Integer a, @ForAll('halfMax') Integer b, @ForAll('halfMax') Integer c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() } @Provide Arbitrary<Integer> halfMax() { int halfMax = Integer.MAX_VALUE >> 1 return Arbitraries.integers().between(-halfMax, halfMax) }
29. © paulk_asert 2006-2020 Motivating example • We can limit the values fed into sumBiggestPair • Or modify our test to have the same overflow characteristics as the method under test. We’ll do the calculations using long and truncate just at the end (which might overflow but identically to sumBiggestPair): @Property void checkIntegerWithLongCalculations(@ForAll Integer a, @ForAll Integer b, @ForAll Integer c) { def (al, bl, cl) = [a, b, c]*.toLong() assert sumBiggestPair(a, b, c) == [al+bl, al+cl, bl+cl].max().toInteger() } |--------------------jqwik-------------------- tries = 1000 | # of calls to property checks = 1000 | # of not rejected calls generation-mode = RANDOMIZED | parameters are randomly generated after-failure = PREVIOUS_SEED | use the previous seed seed = -1317849176950590163 | random seed to reproduce generated values
30. Temperature scales http://www.istockphoto.com/dk/vector/freezing-and-boiling-points-in-celsius-and-fahrenheit-gm533454646-94496887
31. Temperature scales http://imgur.com/gallery/KB3nyXX
32. © paulk_asert 2006-2020 Built-in assertions import static Converter.celsius assert 20 == celsius(68) assert 35 == celsius(95) assert -17 == celsius(0).toInteger() assert 0 == celsius(32) class Converter { static celsius (fahrenheit) { (fahrenheit - 32) * 5 / 9 } }
33. © paulk_asert 2006-2020 JUnit4 import org.junit.Test import static org.junit.Assert.assertEquals import static Converter.celsius class SimpleJUnit4Test { @Test void shouldConvert() { assert celsius(68) == 20 assert celsius(212) == 100, "Should convert boiling" assertEquals("Should convert freezing", 0.0, celsius(32.0)) assertEquals("Also for nearly freezing", 0.0, celsius(32.1), 0.1) } }
34. © paulk_asert 2006-2020 Parameterized import org.junit.Test import org.junit.runner.RunWith import org.junit.runners.Parameterized import org.junit.runners.Parameterized.Parameters import static Converter.celsius @RunWith(Parameterized) class DataDrivenJUnitTest { private c, f, scenario @Parameters static scenarios() {[ [0, 32, 'Freezing'], [20, 68, 'Garden party conditions'], [35, 95, 'Beach conditions'], [100, 212, 'Boiling'] ]*.toArray()} DataDrivenJUnitTest(c, f, scenario) this.c = c this.f = f this.scenario = scenario } @Test void convert() { def actual = celsius(f) def msg = "$scenario: ${f}°F should convert into ${c}°C" assert c == actual, msg } }
35. © paulk_asert 2006-2020 Spock @Grab('org.spockframework:spock-core:1.3-groovy-2.5') import spock.lang.* import static Converter.celsius class SpockDataDriven extends Specification { def "test temperature scenarios"() { expect: celsius(tempF) == tempC where: scenario | tempF || tempC 'Freezing' | 32 || 0 'Garden party conditions' | 68 || 20 'Beach conditions' | 95 || 35 'Boiling' | 212 || 100 } }
36. © paulk_asert 2006-2020 Spock - Celsius @Grab('org.spockframework:spock-core:1.3-groovy-2.5') import spock.lang.* import static Converter.celsius class SpockDataDriven extends Specification { @Unroll def "Scenario #scenario: #tempFºF should convert to #tempCºC"() { expect: celsius(tempF) == tempC where: scenario | tempF || tempC 'Freezing' | 32 || 0 'Garden party conditions' | 68 || 20 'Beach conditions' | 95 || 34 'Boiling' | 212 || 100 } }
37. © paulk_asert 2006-2020 Spock - Celsius @Grab('org.spockframework:spock-core:1.3-groovy-2.5') import spock.lang.* import static Converter.celsius class SpockDataDriven extends Specification { @Unroll def "Scenario #scenario: #tempFºF should convert to #tempCºC"() { expect: celsius(tempF) == tempC where: scenario | tempF || tempC 'Freezing' | 32 || 0 'Garden party conditions' | 68 || 20 'Beach conditions' | 95 || 34 'Boiling' | 212 || 100 } }
38. © paulk_asert 2006-2020 Property-based testing https://www.fantasticfridges.com/Content/CMS/Files/statesofmatter.jpg I can group the input values and output values into meaningful groups: 1. phase of water at given temp Other observed properties 2. Conversion preserves higher/lower ordering 3. Degrees Celsius < degrees Fahrenheit
39. © paulk_asert 2006-2020 Property-based testing @Grab('net.java.quickcheck:quickcheck:0.6') import static net.java.quickcheck.generator.PrimitiveGenerators.* import static java.lang.Math.round import static Converter.celsius def gen = integers(-40, 240) def liquidC = 0..100 def liquidF = 32..212 100.times { int f = gen.next() int c = round(celsius(f)) assert c <= f // Prop 3 assert c in liquidC == f in liquidF // Prop 1 }
40. © paulk_asert 2006-2020 Property-based testing @Grab('net.java.quickcheck:quickcheck:0.6') import static net.java.quickcheck.generator.PrimitiveGenerators.* import static java.lang.Math.round import static Converter.celsius def gen1 = integers(-40, 240) def gen2 = integers(-40, 240) 100.times { int f1 = gen1.next() int f2 = gen2.next() def c1 = celsius(f1) def c2 = celsius(f2) assert (c1 <=> c2) == (f1 <=> f2) // Prop 2 }
41. © paulk_asert 2006-2020 Property-based testing: spock genesis @Grab('com.nagternal:spock-genesis:0.6.0') @GrabExclude('org.codehaus.groovy:groovy-all') import spock.genesis.transform.Iterations import spock.lang.Specification import static Converter.celsius import static java.lang.Math.round import static spock.genesis.Gen.integer class ConverterSpec extends Specification { def liquidC = 0..100 def liquidF = 32..212 @Iterations(100) def "test phase maintained"() { given: int tempF = integer(-40..240).iterator().next() when: int tempC = round(celsius(tempF)) then: tempC <= tempF tempC in liquidC == tempF in liquidF } …
42. © paulk_asert 2006-2020 Property-based testing: spock genesis @Grab('com.nagternal:spock-genesis:0.6.0') @GrabExclude('org.codehaus.groovy:groovy-all') import spock.genesis.transform.Iterations import spock.lang.Specification import static Converter.celsius import static java.lang.Math.round import static spock.genesis.Gen.integer class ConverterSpec extends Specification { def liquidC = 0..100 def liquidF = 32..212 @Iterations(100) def "test phase maintained"() { given: int tempF = integer(-40..240).iterator().next() when: int tempC = round(celsius(tempF)) then: tempC <= tempF tempC in liquidC == tempF in liquidF } … … @Iterations(100) def "test order maintained"() { given: int tempF1 = integer(-273..999).iterator().next() int tempF2 = integer(-273..999).iterator().next() when: int tempC1 = round(celsius(tempF1)) int tempC2 = round(celsius(tempF2)) then: (tempF1 <=> tempF2) == (tempC1 <=> tempC2) } }
43. © paulk_asert 2006-2020 Property-based testing – recap & further theory Agile testing game (TDD) • Minimum test code to steer design of minimal production code with desired business functionality but 100% code coverage • “Grey box” testing • Rarely used with functional programming
44. © paulk_asert 2006-2020 Property-based testing Agile testing game (TDD) • Minimum test code to steer design of minimal production code with desired business functionality but 100% code coverage • “Grey box” testing • Rarely used with functional programming • Instead validate certain properties
45. © paulk_asert 2006-2020 Property-based testing • Fuzzing (coverage-guided) • Invariant conditions string.size() >= 0 • Idempotent list.sort() == list.sort().sort().sort() • Inverse/round-tripping list == list.reverse().reverse() • Read/write, encode/decode, setProp/getProp • Obvious Property (business domain) • Commutativity, Associativity, Identity, … • x + y == y + x, x + 0 == x, (x + y) + z == x + (y + z) • Induction • Blackbox testing hard to solve, easy to verify • Test oracle • Advanced cases: stateful, race conditions
46. © paulk_asert 2006-2020 Property-based testing • Fuzzing (coverage-guided) • Invariant conditions string.size() >= 0 • Idempotent list.sort() == list.sort().sort().sort() • Inverse/round-tripping list == list.reverse().reverse() • Read/write, encode/decode, setProp/getProp • Obvious Property (business domain) • Commutativity, Associativity, Identity, … • x + y == y + x, x + 0 == x, (x + y) + z == x + (y + z) • Induction Explore the examples in the subprojects shown above
47. © paulk_asert 2006-2020 Generators: Genesis More examples: https://github.com/Bijnagte/spock-genesis @Immutable class Person { int id String name Date birthDate String title char gender } def 'test person'() { expect: p.name instanceof String && p.name[0] in 'A'..'Z' && p.name[-1] in 'a'..'z' p.birthDate instanceof Date p.id instanceof Integer && p.id < 10_000 p.title?.size() in [null, 0, 2, 3] where: p << Gen.type(Person, id: Gen.integer(1..9999), name: Gen.string(~/[A-Z][a-z]+( [A-Z][a-z]+)?/), birthDate: Gen.date(Date.parse('MM/dd/yyyy', '01/01/1980'), new Date()), title: Gen.these('', null).then(Gen.any('Dr', 'Mr', 'Ms', 'Mrs', 'Mx')), gender: Gen.character('MFX')).take(5) }
48. © paulk_asert 2006-2020 Generators: QuickCheck @Test void testGenerators() { def pets = fixedValues(['Ant', 'Bee', 'Cat', 'Dog']) def nums = excludeValues(integers(100, 999, INVERTED_NORMAL), 500..599) def months = new MonthGenerator() def enums = new Generator() { String next() { anyEnumValue(Nums) } } def gen = new DefaultFrequencyGenerator(pets, 40) .add(nums, 30) .add(months, 20) .add(enums, 10) 50.times { def next = gen.next().toString() print "$next " assert next.size() == 3 if (it % 10 == 9) println() } println() } Bee Oct Dog Bee TEN Cat Ant Ant 182 837 Ant 231 TEN Bee Bee ONE 984 SIX Jul Bee 924 Ant Dec Dec Jun SIX Ant Dog Jul Dog Apr Feb TWO 146 Dog May Ant Ant 876 194 Dog Nov Jul Cat 836 Cat Ant TWO 781 Cat static enum Nums { ONE, TWO, SIX, TEN } static class MonthGenerator implements Generator<String> { Generator<Date> genDate = dates() def sdf = new SimpleDateFormat('MMM', Locale.ENGLISH) String next() { sdf.format(genDate.next()) } }
49. © paulk_asert 2006-2020 Property-based testing: Case Study https://github.com/paulk-asert/MakeTestingGroovy
50. © paulk_asert 2006-2020 Property-based testing: Case Study
51. © paulk_asert 2006-2020 Property-based testing: Summary • Write less tests • But don’t be afraid to also use example-based tests • Focus on properties of the system • Find ways to validate properties with customer