objectcomputing.com © 2020, Object Computing, Inc. (OCI). All rights reserved. No part of these notes may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior, written permission of Object Computing, Inc. (OCI) An introduction to Property-based Testing (with Groovy & other languages) Dr Paul King OCI Groovy Lead V.P. and PMC Chair Apache Groovy @paulk_asert © 2020 Object Computing, Inc. (OCI). All rights reserved. objectcomputing.com

Dr Paul King OCI Groovy Lead V.P. and PMC Chair Apache Groovy Slides: https://speakerdeck.com/paulk/property-based-testing Examples repo: https://github.com/paulk-asert/property-based-testing ReGinA author: https://www.manning.com/books/groovy-in-action-second-edition Twitter: @paulk_asert

Friends of Apache Groovy Open Collective

© paulk_asert 2006-2020 What is property-based testing • An approach to derive automatic test cases for some system • To reduce the effort required to create manual test cases • To gather additional cases that we might miss by hand • Importantly, with a view to confirming desirable properties about the system under test

© paulk_asert 2006-2020 Motivating example public class MathUtil { public static int sumBiggestPair(int a, int b, int c) { int op1 = a; int op2 = b; if (c > a) { op1 = c; } else if (c > b) { op2 = c; } return op1 + op2; } private MathUtil(){} }

© paulk_asert 2006-2020 Motivating example import org.junit.Test import static util.MathUtil.sumBiggestPair class MathUtilTest { @Test void examples() { assert sumBiggestPair(5, 4, 1) == 9 assert sumBiggestPair(4, 5, 1) == 9 assert sumBiggestPair(5, 9, 6) == 15 } }

© paulk_asert 2006-2020 Motivating example import org.junit.Test import static util.MathUtil.sumBiggestPair class MathUtilTest { @Test void examples() { assert sumBiggestPair(5, 4, 1) == 9 assert sumBiggestPair(4, 5, 1) == 9 assert sumBiggestPair(5, 9, 6) == 15 } } Tests pass but not 100% coverage, so let’s add some more test cases.

© paulk_asert 2006-2020 Motivating example import org.junit.Test import static util.MathUtil.sumBiggestPair class MathUtilTest { @Test void examples() { assert sumBiggestPair(5, 4, 1) == 9 assert sumBiggestPair(4, 5, 1) == 9 assert sumBiggestPair(5, 9, 6) == 15 assert sumBiggestPair(10, 2, 6) == 16 } }

© paulk_asert 2006-2020 Motivating example import org.junit.Test import static util.MathUtil.sumBiggestPair class MathUtilTest { @Test void examples() { assert sumBiggestPair(5, 4, 1) == 9 assert sumBiggestPair(4, 5, 1) == 9 assert sumBiggestPair(5, 9, 6) == 15 assert sumBiggestPair(10, 2, 6) == 16 } }

© paulk_asert 2006-2020 Motivating example import org.junit.Test import static util.MathUtil.sumBiggestPair class MathUtilTest { @Test void examples() { assert sumBiggestPair(5, 4, 1) == 9 assert sumBiggestPair(4, 5, 1) == 9 assert sumBiggestPair(5, 9, 6) == 15 assert sumBiggestPair(10, 2, 6) == 16 } } 100% coverage, so it must be bug free, right? Let’s ship it!

© paulk_asert 2006-2020 Motivating example import org.junit.Test import static util.MathUtil.sumBiggestPair class MathUtilTest { @Test void examples() { assert sumBiggestPair(5, 4, 1) == 9 assert sumBiggestPair(4, 5, 1) == 9 assert sumBiggestPair(5, 9, 6) == 15 assert sumBiggestPair(10, 2, 6) == 16 // 100% assert sumBiggestPair(2, 5, 6) == 11 assert sumBiggestPair(5, 2, 6) == 11 } } But we’ll just add a couple more tests.

© paulk_asert 2006-2020 Motivating example import org.junit.Test import static util.MathUtil.sumBiggestPair class MathUtilTest { @Test void examples() { assert sumBiggestPair(5, 4, 1) == 9 assert sumBiggestPair(4, 5, 1) == 9 assert sumBiggestPair(5, 9, 6) == 15 assert sumBiggestPair(10, 2, 6) == 16 // 100% assert sumBiggestPair(2, 5, 6) == 11 assert sumBiggestPair(5, 2, 6) == 11 } }

© paulk_asert 2006-2020 Motivating example public class MathUtil { public static int sumBiggestPair(int a, int b, int c) { int op1 = a; int op2 = b; if (c > a) { op1 = c; } else if (c > b) { op2 = c; } return op1 + op2; } private MathUtil(){} } Else branch logic is flawed for case where c is biggest and b is smallest.

© paulk_asert 2006-2020 Motivating example public class MathUtil { public static int sumBiggestPair(int a, int b, int c) { int op1 = a; int op2 = b; if (c > Math.min(a, b)) { op1 = c; op2 = Math.max(a, b); } return op1 + op2; } private MathUtil(){} } One way to correct the logic.

Motivating example Solution? • Write more tests? • Expensive and potentially hinders agile refactoring • How do I know when to stop? • Spend more time thinking about tests

Motivating example Solution? • Write more tests? • Expensive and potentially hinders agile refactoring • How do I know when to stop? • Spend more time thinking about tests Alternative solution? • Write less tests! • Spend more time thinking about system properties

© paulk_asert 2006-2020 Motivating example import org.junit.Test import static util.MathUtil.sumBiggestPair class MathUtilTest { @Test void examples() { assert sumBiggestPair(5, 4, 1) == 9 assert sumBiggestPair(4, 5, 1) == 9 assert sumBiggestPair(5, 9, 6) == 15 assert sumBiggestPair(10, 2, 6) == 16 assert sumBiggestPair(2, 5, 6) == 11 assert sumBiggestPair(5, 2, 6) == 11 } } Goal will be to replace input values with randomly generated values But how do we get the output value?

© paulk_asert 2006-2020 Motivating example import net.jqwik.api.ForAll import net.jqwik.api.Property import static util.MathUtil.sumBiggestPair class MathUtilTest { @Property void auto(@ForAll Short a, @ForAll Short b, @ForAll Short c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() } } Jqwik uses annotations to guide where randomly generated values will be used and what those values might be. For this case, we can find an alternative way to calculate the answer

© paulk_asert 2006-2020 Motivating example import net.jqwik.api.ForAll import net.jqwik.api.Property import static util.MathUtil.sumBiggestPair class MathUtilTest { @Property void auto(@ForAll Short a, @ForAll Short b, @ForAll Short c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() } } |--------------------jqwik-------------------- tries = 12 | # of calls to property checks = 12 | # of not rejected calls generation-mode = RANDOMIZED | parameters are randomly generated after-failure = PREVIOUS_SEED | use the previous seed seed = 6313771727151213536 | random seed to reproduce generated values sample = [-98, -99, 0] original-sample = [-111, -13803, -97] For faulty case

© paulk_asert 2006-2020 Motivating example import net.jqwik.api.ForAll import net.jqwik.api.Property import static util.MathUtil.sumBiggestPair class MathUtilTest { @Property void auto(@ForAll Short a, @ForAll Short b, @ForAll Short c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() } } |--------------------jqwik-------------------- tries = 1000 | # of calls to property checks = 1000 | # of not rejected calls generation-mode = RANDOMIZED | parameters are randomly generated after-failure = PREVIOUS_SEED | use the previous seed seed = 6313771727151213536 | random seed to reproduce generated values For fixed case

© paulk_asert 2006-2020 Motivating example import net.jqwik.api.ForAll import net.jqwik.api.Property import static util.MathUtil.sumBiggestPair class MathUtilTest { @Property void auto(@ForAll Short a, @ForAll Short b, @ForAll Short c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() } } So, it must be bug free now. It passes 1000 tests! Let’s ship it!

© paulk_asert 2006-2020 Motivating example import net.jqwik.api.ForAll import net.jqwik.api.Property import static util.MathUtil.sumBiggestPair class MathUtilTest { @Property void auto(@ForAll Integer a, @ForAll Integer b, @ForAll Integer c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() } } Replace Short with Integer

© paulk_asert 2006-2020 Motivating example import net.jqwik.api.ForAll import net.jqwik.api.Property import static util.MathUtil.sumBiggestPair class MathUtilTest { @Property void auto(@ForAll Integer a, @ForAll Integer b, @ForAll Integer c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() } } |--------------------jqwik-------------------- tries = 10 | # of calls to property checks = 10 | # of not rejected calls generation-mode = RANDOMIZED | parameters are randomly generated after-failure = PREVIOUS_SEED | use the previous seed seed = -1317849176950590163 | random seed to reproduce generated values sample = [200764882, 1946718766, 0] original-sample = [1942642352, 1946718766, -368]

© paulk_asert 2006-2020 Motivating example import net.jqwik.api.ForAll import net.jqwik.api.Property import static util.MathUtil.sumBiggestPair class MathUtilTest { @Property void auto(@ForAll Integer a, @ForAll Integer b, @ForAll Integer c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() } } |--------------------jqwik-------------------- tries = 10 | # of calls to property checks = 10 | # of not rejected calls generation-mode = RANDOMIZED | parameters are randomly generated after-failure = PREVIOUS_SEED | use the previous seed seed = -1317849176950590163 | random seed to reproduce generated values sample = [200764882, 1946718766, 0] original-sample = [1942642352, 1946718766, -368] Bug with code or property test?

© paulk_asert 2006-2020 Motivating example • No correct answer • Check with “customer” • Here we’ll decide that overflow is a “feature” • So, we’ll fix the test. We have several options.

© paulk_asert 2006-2020 Motivating example • We can limit the values fed into sumBiggestPair • We already saw this with using Short params import net.jqwik.api.ForAll import net.jqwik.api.Property import static util.MathUtil.sumBiggestPair class MathUtilTest { @Property void auto(@ForAll Short a, @ForAll Short b, @ForAll Short c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() } }

© paulk_asert 2006-2020 Motivating example • We can limit the values fed into sumBiggestPair • But we could also use an annotation like here: @Property void checkIntegerConstrained(@ForAll @IntRange(min = -1_000_000_000, max = 1_000_000_000) Integer a, @ForAll @IntRange(min = -1_000_000_000, max = 1_000_000_000) Integer b, @ForAll @IntRange(min = -1_000_000_000, max = 1_000_000_000) Integer c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() }

© paulk_asert 2006-2020 Motivating example • We can limit the values fed into sumBiggestPair • But we could also use a custom provider like here: @Property void checkIntegerConstrainedProvider(@ForAll('halfMax') Integer a, @ForAll('halfMax') Integer b, @ForAll('halfMax') Integer c) { assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() } @Provide Arbitrary halfMax() { int halfMax = Integer.MAX_VALUE >> 1 return Arbitraries.integers().between(-halfMax, halfMax) }

© paulk_asert 2006-2020 Motivating example • We can limit the values fed into sumBiggestPair • Or modify our test to have the same overflow characteristics as the method under test. We’ll do the calculations using long and truncate just at the end (which might overflow but identically to sumBiggestPair): @Property void checkIntegerWithLongCalculations(@ForAll Integer a, @ForAll Integer b, @ForAll Integer c) { def (al, bl, cl) = [a, b, c]*.toLong() assert sumBiggestPair(a, b, c) == [al+bl, al+cl, bl+cl].max().toInteger() } |--------------------jqwik-------------------- tries = 1000 | # of calls to property checks = 1000 | # of not rejected calls generation-mode = RANDOMIZED | parameters are randomly generated after-failure = PREVIOUS_SEED | use the previous seed seed = -1317849176950590163 | random seed to reproduce generated values

Temperature scales http://www.istockphoto.com/dk/vector/freezing-and-boiling-points-in-celsius-and-fahrenheit-gm533454646-94496887

Temperature scales http://imgur.com/gallery/KB3nyXX

© paulk_asert 2006-2020 Built-in assertions import static Converter.celsius assert 20 == celsius(68) assert 35 == celsius(95) assert -17 == celsius(0).toInteger() assert 0 == celsius(32) class Converter { static celsius (fahrenheit) { (fahrenheit - 32) * 5 / 9 } }

© paulk_asert 2006-2020 JUnit4 import org.junit.Test import static org.junit.Assert.assertEquals import static Converter.celsius class SimpleJUnit4Test { @Test void shouldConvert() { assert celsius(68) == 20 assert celsius(212) == 100, "Should convert boiling" assertEquals("Should convert freezing", 0.0, celsius(32.0)) assertEquals("Also for nearly freezing", 0.0, celsius(32.1), 0.1) } }

© paulk_asert 2006-2020 Parameterized import org.junit.Test import org.junit.runner.RunWith import org.junit.runners.Parameterized import org.junit.runners.Parameterized.Parameters import static Converter.celsius @RunWith(Parameterized) class DataDrivenJUnitTest { private c, f, scenario @Parameters static scenarios() {[ [0, 32, 'Freezing'], [20, 68, 'Garden party conditions'], [35, 95, 'Beach conditions'], [100, 212, 'Boiling'] ]*.toArray()} DataDrivenJUnitTest(c, f, scenario) this.c = c this.f = f this.scenario = scenario } @Test void convert() { def actual = celsius(f) def msg = "$scenario: ${f}°F should convert into ${c}°C" assert c == actual, msg } }

© paulk_asert 2006-2020 Spock @Grab('org.spockframework:spock-core:1.3-groovy-2.5') import spock.lang.* import static Converter.celsius class SpockDataDriven extends Specification { def "test temperature scenarios"() { expect: celsius(tempF) == tempC where: scenario | tempF || tempC 'Freezing' | 32 || 0 'Garden party conditions' | 68 || 20 'Beach conditions' | 95 || 35 'Boiling' | 212 || 100 } }

© paulk_asert 2006-2020 Spock - Celsius @Grab('org.spockframework:spock-core:1.3-groovy-2.5') import spock.lang.* import static Converter.celsius class SpockDataDriven extends Specification { @Unroll def "Scenario #scenario: #tempFºF should convert to #tempCºC"() { expect: celsius(tempF) == tempC where: scenario | tempF || tempC 'Freezing' | 32 || 0 'Garden party conditions' | 68 || 20 'Beach conditions' | 95 || 34 'Boiling' | 212 || 100 } }

© paulk_asert 2006-2020 Spock - Celsius @Grab('org.spockframework:spock-core:1.3-groovy-2.5') import spock.lang.* import static Converter.celsius class SpockDataDriven extends Specification { @Unroll def "Scenario #scenario: #tempFºF should convert to #tempCºC"() { expect: celsius(tempF) == tempC where: scenario | tempF || tempC 'Freezing' | 32 || 0 'Garden party conditions' | 68 || 20 'Beach conditions' | 95 || 34 'Boiling' | 212 || 100 } }

© paulk_asert 2006-2020 Property-based testing https://www.fantasticfridges.com/Content/CMS/Files/statesofmatter.jpg I can group the input values and output values into meaningful groups: 1. phase of water at given temp Other observed properties 2. Conversion preserves higher/lower ordering 3. Degrees Celsius < degrees Fahrenheit

© paulk_asert 2006-2020 Property-based testing @Grab('net.java.quickcheck:quickcheck:0.6') import static net.java.quickcheck.generator.PrimitiveGenerators.* import static java.lang.Math.round import static Converter.celsius def gen = integers(-40, 240) def liquidC = 0..100 def liquidF = 32..212 100.times { int f = gen.next() int c = round(celsius(f)) assert c <= f // Prop 3 assert c in liquidC == f in liquidF // Prop 1 }

© paulk_asert 2006-2020 Property-based testing @Grab('net.java.quickcheck:quickcheck:0.6') import static net.java.quickcheck.generator.PrimitiveGenerators.* import static java.lang.Math.round import static Converter.celsius def gen1 = integers(-40, 240) def gen2 = integers(-40, 240) 100.times { int f1 = gen1.next() int f2 = gen2.next() def c1 = celsius(f1) def c2 = celsius(f2) assert (c1 <=> c2) == (f1 <=> f2) // Prop 2 }

© paulk_asert 2006-2020 Property-based testing: spock genesis @Grab('com.nagternal:spock-genesis:0.6.0') @GrabExclude('org.codehaus.groovy:groovy-all') import spock.genesis.transform.Iterations import spock.lang.Specification import static Converter.celsius import static java.lang.Math.round import static spock.genesis.Gen.integer class ConverterSpec extends Specification { def liquidC = 0..100 def liquidF = 32..212 @Iterations(100) def "test phase maintained"() { given: int tempF = integer(-40..240).iterator().next() when: int tempC = round(celsius(tempF)) then: tempC <= tempF tempC in liquidC == tempF in liquidF } …

© paulk_asert 2006-2020 Property-based testing: spock genesis @Grab('com.nagternal:spock-genesis:0.6.0') @GrabExclude('org.codehaus.groovy:groovy-all') import spock.genesis.transform.Iterations import spock.lang.Specification import static Converter.celsius import static java.lang.Math.round import static spock.genesis.Gen.integer class ConverterSpec extends Specification { def liquidC = 0..100 def liquidF = 32..212 @Iterations(100) def "test phase maintained"() { given: int tempF = integer(-40..240).iterator().next() when: int tempC = round(celsius(tempF)) then: tempC <= tempF tempC in liquidC == tempF in liquidF } … … @Iterations(100) def "test order maintained"() { given: int tempF1 = integer(-273..999).iterator().next() int tempF2 = integer(-273..999).iterator().next() when: int tempC1 = round(celsius(tempF1)) int tempC2 = round(celsius(tempF2)) then: (tempF1 <=> tempF2) == (tempC1 <=> tempC2) } }

© paulk_asert 2006-2020 Property-based testing – recap & further theory Agile testing game (TDD) • Minimum test code to steer design of minimal production code with desired business functionality but 100% code coverage • “Grey box” testing • Rarely used with functional programming

© paulk_asert 2006-2020 Property-based testing Agile testing game (TDD) • Minimum test code to steer design of minimal production code with desired business functionality but 100% code coverage • “Grey box” testing • Rarely used with functional programming • Instead validate certain properties

© paulk_asert 2006-2020 Property-based testing • Fuzzing (coverage-guided) • Invariant conditions string.size() >= 0 • Idempotent list.sort() == list.sort().sort().sort() • Inverse/round-tripping list == list.reverse().reverse() • Read/write, encode/decode, setProp/getProp • Obvious Property (business domain) • Commutativity, Associativity, Identity, … • x + y == y + x, x + 0 == x, (x + y) + z == x + (y + z) • Induction • Blackbox testing hard to solve, easy to verify • Test oracle • Advanced cases: stateful, race conditions

© paulk_asert 2006-2020 Property-based testing • Fuzzing (coverage-guided) • Invariant conditions string.size() >= 0 • Idempotent list.sort() == list.sort().sort().sort() • Inverse/round-tripping list == list.reverse().reverse() • Read/write, encode/decode, setProp/getProp • Obvious Property (business domain) • Commutativity, Associativity, Identity, … • x + y == y + x, x + 0 == x, (x + y) + z == x + (y + z) • Induction Explore the examples in the subprojects shown above

© paulk_asert 2006-2020 Generators: Genesis More examples: https://github.com/Bijnagte/spock-genesis @Immutable class Person { int id String name Date birthDate String title char gender } def 'test person'() { expect: p.name instanceof String && p.name[0] in 'A'..'Z' && p.name[-1] in 'a'..'z' p.birthDate instanceof Date p.id instanceof Integer && p.id < 10_000 p.title?.size() in [null, 0, 2, 3] where: p << Gen.type(Person, id: Gen.integer(1..9999), name: Gen.string(~/[A-Z][a-z]+( [A-Z][a-z]+)?/), birthDate: Gen.date(Date.parse('MM/dd/yyyy', '01/01/1980'), new Date()), title: Gen.these('', null).then(Gen.any('Dr', 'Mr', 'Ms', 'Mrs', 'Mx')), gender: Gen.character('MFX')).take(5) }

© paulk_asert 2006-2020 Generators: QuickCheck @Test void testGenerators() { def pets = fixedValues(['Ant', 'Bee', 'Cat', 'Dog']) def nums = excludeValues(integers(100, 999, INVERTED_NORMAL), 500..599) def months = new MonthGenerator() def enums = new Generator() { String next() { anyEnumValue(Nums) } } def gen = new DefaultFrequencyGenerator(pets, 40) .add(nums, 30) .add(months, 20) .add(enums, 10) 50.times { def next = gen.next().toString() print "$next " assert next.size() == 3 if (it % 10 == 9) println() } println() } Bee Oct Dog Bee TEN Cat Ant Ant 182 837 Ant 231 TEN Bee Bee ONE 984 SIX Jul Bee 924 Ant Dec Dec Jun SIX Ant Dog Jul Dog Apr Feb TWO 146 Dog May Ant Ant 876 194 Dog Nov Jul Cat 836 Cat Ant TWO 781 Cat static enum Nums { ONE, TWO, SIX, TEN } static class MonthGenerator implements Generator { Generator genDate = dates() def sdf = new SimpleDateFormat('MMM', Locale.ENGLISH) String next() { sdf.format(genDate.next()) } }

© paulk_asert 2006-2020 Property-based testing: Case Study https://github.com/paulk-asert/MakeTestingGroovy

© paulk_asert 2006-2020 Property-based testing: Case Study

© paulk_asert 2006-2020 Property-based testing: Summary • Write less tests • But don’t be afraid to also use example-based tests • Focus on properties of the system • Find ways to validate properties with customer