A journey to Property-Based Testing

Explore what is Property-Based Testing and how you can use this approach in your daily dev life.
Support of a workshop explained here : https://yoan-thirion.gitbook.io/knowledge-base/software-craftsmanship/code-katas/improve-your-software-quality-with-property-based-testing/a-journey-to-property-based-testing

Transcript

1. @yot88 A Journey to Property-based Testing
2. Calculator In mob : • Open the repository • Write tests to check the behavior of the Calculator class https://github.com/ythirion/property-based-testing-kata object Calculator { def add(x: Int, y: Int): Int = x + y } Source code is available in scala (scalatest / scalacheck) / C# (xunit / FsCheck / Language-Ext / FluentAssertions) / java (Junit5 / quick-check / vavr / AssertJ)
3. Calculator - Debriefing • Which tests did you write ? • What edge cases did you find ? • What are the limits of this example-based testing approach ? Given (1, 3) When I add them Then I expect 4 Given (-1, 3) When I add them Then I expect 2 Given (0, 99) When I add them Then I expect 99 Could we test it differently to cover more inputs ? it should "return their correct sum when I add 2 random numbers" in { Range(0, 100) .foreach(_ => { val x = random.nextInt() val y = random.nextInt() assert(add(x, y) == x + y) }) } Implementation leak in tests What does correct mean in this context ?
4. Example-based testing Do we cover the full scope of possible inputs ? Does the developed feature is compliant with what is expected ? When we write Unit tests : • we often focus on examples • examples are good to understand requirements Given (x, y, ...) When I [call the subject under test] with (x, y, ...) Then I expect this (output) Limitation : we only focus on the input scope we identified
5. Property-based testing – at our rescue ? PBT
6. What is Property-based testing • A property is the combination of an invariant with an input values generator. For each generated value, the invariant is treated as a predicate and checked whether it yields true or false for that value. • As soon as there is one value which yields false, the property is said to be falsified, and checking is aborted. • If a property cannot be invalidated after a specific amount of sample data, the property is assumed to be satisfied.
7. In other words, a property is something like : for all (x, y, ...) such as precondition (x, y, ...) holds property (x, y, ...) is satisfied run it multiple times generate random inputs (based on specified generators) check input pre-conditions Failure: not an error -> invalid entry run the predicate Failure: a counter-example will try to shrink the entry • Describe the input • Describe the properties of the output • Have the computer try lots of random examples • Check if it fails
8. @yot88 Which properties do we know on Addition ? Compare addition with other Math operations • What about parameter order ? (compared to subtract) The parameter order does not matter • “add 1” twice is the same as doing “add 2” (compared to multiply) • What about adding 0 ? (compared to multiply) Adding 0 does nothing We are confident the implementation is correct if we test these properties It applies to all inputs
9. Let’s write some tests it should "when I add 2 numbers the result should not depend on parameter order" in { Range(0, times) .map(_ => random.nextInt()) .foreach(x => { val y = random.nextInt() assert(add(x, y) == add(y, x)) }) } it should "when I add 1 twice it's the same as adding 2 once" in { Range(0, times) .map(_ => random.nextInt()) .foreach(x => assert(add(add(x, 1), 1) == add(x, 2))) } it should "when I add 0 to a random number is the same than doing nothing on this number" in { Range(0, times) .map(_ => random.nextInt()) .foreach(x => assert(add(x, 0) == x)) } Commutativity Associativity Identity
10. ScalaCheck “ScalaCheck is a tool for testing Scala and Java programs, based on property specifications and automatic test data generation. The basic idea is that you define a property that specifies the behaviour of a method or some unit of code, and ScalaCheck checks that the property holds. All test data are generated automatically in a random fashion, so you don’t have to worry about any missed cases.” - ScalaCheck : The Definitive Guide libraryDependencies += "org.scalatest" %% "scalatest" % "3.2.9" % Test libraryDependencies += "org.scalactic" %% "scalactic" % "3.2.9" % Test libraryDependencies += "org.scalatestplus" %% "scalacheck-1-15" % "3.2.9.0" % "test" https://github.com/typelevel/scalacheck/blob/main/doc/UserGuide.md A way to generate (and shrink) random test data
11. Addition properties with ScalaCheck object CalculatorProperties extends Properties("add") { property("commutativity") = forAll { (x: Int, y: Int) => add(x, y) == add(y, x) } property("associativity") = forAll { (x: Int) => add(add(x, 1), 1) == add(x, 2) } property("identity") = forAll { (x: Int) => add(x, 0) == x } }
12. How to read it ? property("commutativity") = forAll { (x: Int, y: Int) => add(x, y) == add(y, x) } For all possible x, y run the block of code that follows -1, 1887999732 1887999732, -2147483648 -2147483648, 1 1, -738646202 -738646202, 454567881 454567881, -1 -1, 1321200746 1321200746, 2147483647 2147483647, -2147483648 -2147483648, 912114712 912114712, 703322924 703322924, -802049796 -802049796, 1 1, 2147483647 2147483647, -1707626195 -1707626195, -1 -1, -1626087642 -1626087642, 2147483647 2147483647, 546871561 546871561, -651551026 -651551026, 0 0, 1 1, 1 1, 958219829 958219829, 1772245900 … Can help us identify values outside from our business frame
13. Integration with ScalaTest class CalculatorPropertiesFlatSpec extends AnyFlatSpec with Checkers { "add" should "be commutative" in { check(forAll { (x: Int, y: Int) => add(x, y) == add(y, x) }) } "add" should "be associative" in { check(forAll { (x: Int) => add(add(x, 1), 1) == add(x, 2) }) } "0" should "be identity" in { check(forAll { (x: Int) => add(x, 0) == x }) } } https://www.scalatest.org/user_guide/writing_scalacheck_style_properties
14. @yot88 What happens when it fails ScalaCheck provides us the values for which the property is falsified
15. Concepts • By default, each function is tested 10 times 10 times if all tests succeed (differs from scalacheck-only behavior) less than that if a test is falsified • Tests are run with randomly-generated data • ScalaCheck has built-in generators We can also write our own generators
16. PBT in real life
17. Postal Parcel Open `PostalParcel` : • Identify properties – 5’ What are the invariants ? • Write them in `PostalParcelPropertiesFlatSpec` - 15’ • Collective debriefing – 5’ object PostalParcel { val maxWeight = 20.0 val maxDeliveryCosts = 4.99 val minDeliveryCosts = 1.99 def fromDouble(weight: Double): Option[PostalParcel] = if (weight > 0) Some(PostalParcel(weight)) else None def calculateDeliveryCosts( postalParcel: Option[PostalParcel] ): Option[Double] = { postalParcel.map(p => if (p.weight > maxWeight) maxDeliveryCosts else minDeliveryCosts ) } } scala C# java
18. Bank withdrawal • Open `AccountService` : • Identify properties – 5’ What should we do to check those properties ? object AccountService { def withdraw(account: Account, command: Withdraw): Either[String, Account] = { if (hasAlreadyBeenApplied(account, command)) Right(account) else applyWithdraw(account, command) } private def applyWithdraw( account: Account, command: Withdraw ): Either[String, Account] = { command.amount.value match { case amount if exceedMaxWithdrawal(account, amount) => Left(s"Amount exceeding your limit of ${account.maxWithdrawal}") case amount if exceedBalance(account, amount) => Left(s"Insufficient balance to withdraw : $amount") case _ => Right( account.copy( balance = account.balance - command.amount.value, withdraws = account.withdraws :+ command ) ) } } private def hasAlreadyBeenApplied(account: Account, command: Withdraw) = account.withdraws.contains(command) private def exceedMaxWithdrawal( account: Account, withdrawAmount: Double ): Boolean = withdrawAmount >= account.maxWithdrawal private def exceedBalance(account: Account, withdrawAmount: Double): Boolean = withdrawAmount > account.balance && !account.isOverdraftAuthorized }
19. Bank withdrawal - Properties • "account balance" should "be decremented at least from the withdraw amount” • "account balance" should "be decremented at least from the withdraw amount when insufficient balance but overdraft authorized” • "withdraw" should "not be allowed when withdraw amount >= maxWithdrawal” • "withdraw" should "not be allowed when insufficient balance and no overdraft” • "withdraw" should "be idempotent"
20. Bank withdrawal using properties and when (WithdrawProperties) "withdraw" should "not be allowed when insufficient balance and no overdraft" in { checkProperty( (account, command) => { withInsufficientBalance(account, command) && withoutOverdraftAuthorized(account) && withoutReachingMaxWithdrawal(account, command) }, (_, _, debitedAccount) => { debitedAccount.left.value .startsWith("Insufficient balance to withdraw") } ) } Need to discard a lot of generated inputs implicit override val generatorDrivenConfig: PropertyCheckConfiguration = PropertyCheckConfiguration(minSize = 10, maxDiscardedFactor = 30) Need to increase maxDiscardedFactor to check the properties… Not a valid way to check our business invariants implicit val accountGenerator: Arbitrary[Account] = Arbitrary { for { balance <- Arbitrary.arbitrary[Double] isOverdraftAuthorized <- Arbitrary.arbitrary[Boolean] maxWithdrawal <- Arbitrary.arbitrary[Double] } yield Account(balance, isOverdraftAuthorized, maxWithdrawal) } implicit val withdrawCommandGenerator: Arbitrary[Withdraw] = Arbitrary { for { clientId <- Arbitrary.arbitrary[UUID] amount <- posNum[Double] requestDate <- Arbitrary.arbitrary[LocalDate] } yield Withdraw(clientId, Amount.from(amount).get, requestDate) } scala
21. Bank withdrawal using properties and commandGenerator + builder "withdraw" should "not be allowed when insufficient balance and no overdraft" in { checkProperty( (account, command) => account .withInsufficientBalance(command) .withoutOverDraftAuthorized() .withoutReachingMaxWithdrawal(command), (_, _, debitedAccount) => debitedAccount.left.value .startsWith("Insufficient balance to withdraw") ) } private def checkProperty( accountConfiguration: (AccountBuilder, Withdraw) => AccountBuilder, property: (Account, Withdraw, Either[String, Account]) => Boolean ): Assertion = { check(forAll { (command: Withdraw) => val account = accountConfiguration(AccountBuilder(), command).build() property(account, command, AccountService.withdraw(account, command)) }) } final case class AccountBuilder( balance: Double = 0, isOverdraftAuthorized: Boolean = false, maxWithdrawal: Double = 0 ) { def withEnoughMoney(command: Withdraw): AccountBuilder = copy(balance = command.amount.value + arbitraryPositiveNumber) def withdrawAmountReachingMaxWithdrawal(command: Withdraw): AccountBuilder = copy(maxWithdrawal = command.amount.value - arbitraryPositiveNumber) … We now instantiate accounts based on random commands with a Builder Do not need to change ScalaCheck configuration anymore Builder is maybe leaking too much of our business… scala C# java
22. private val `withdraw examples` = Table( ( "balance", "isOverdraftAuthorized", "maxWithdrawal", "withdrawAmount", "expectedResult" ), (10_000, false, 1200, 1199.99, Right(8800.01)), (0, true, 500, 50d, Right(-50)), (1, false, 1200, 1199.99, Left("Insufficient balance to withdraw")), (0, false, 500, 50d, Left("Insufficient balance to withdraw")), (10_000, true, 1200, 1200.0001, Left("Amount exceeding your limit of")), (0, false, 500, 500d, Left("Amount exceeding your limit of")) ) Bank withdrawal – how we would test it in real life ? Use ParameterizedTests -> TableDrivenPropertyChecks Describe test cases Check implementation based on them We can use Properties to help us identify edge cases and add new examples in our table A better approach for this kind of use cases : • Example-based with Table to allow easy addition of new tests cases • Use PBT as a cases finders • Use PBT to check idem potence "withdraw" should "pass examples" in { forAll(`withdraw examples`) { ( balance, isOverdraftAuthorized, maxWithdrawal, withdrawAmount, expectedResult ) => val command = toWithdraw(withdrawAmount) val debitedAccount = AccountService.withdraw( Account(balance, isOverdraftAuthorized, maxWithdrawal), command ) expectedResult match { case Left(expectedErrorMessage) => debitedAccount.left.value should startWith(expectedErrorMessage) case Right(expectedBalance) => debitedAccount.value.balance should be(expectedBalance) debitedAccount.value.withdraws should contain(command) debitedAccount.value.withdraws should have length 1 } } } scala C# java
23. Rental Calculator PBT on legacy code • Open `RentalCalculator` : Imagine this code is running in production and business is happy with it Imagine you need to adapt it, but you need to be sure that you don’t introduce any regression class RentalCalculator(val rentals: List[Rental]) { private var _amount = .0 private var _calculated = false def amount(): Double = _amount def calculated(): Boolean = _calculated def calculateRental(): Either[String, String] = { if (rentals.isEmpty) Left("No rentals !!!") else { val result = new StringBuilder for (rental <- rentals) { if (!_calculated) this._amount += rental.amount result.append(formatLine(rental, _amount)) } result.append(f"Total amount | ${this._amount}%.2f") _calculated = true Right(result.toString) } } private def formatLine(rental: Rental, amount: Double) = f"${rental.date} : ${rental.label} | ${rental.amount}%.2f${lineSeparator()}" } How PBT can help us ?
24. Rental Calculator PBT on legacy code • Duplicate the production code in a new file – NewRentalCalculator for example • Write a property checking that – 5’ f(x) == new_f(x) Ensure that your test fails if you change the new implementation • Now you can refactor the new implementation code – 15’ Identify the code smells Refactor the code Run the property at each changes • You can now plug the caller on the new implementation Delete the legacy implementation Remove the property Seeing a test failing is as important as seeing it passing Can use this technique for any rewriting / optimizing refactoring scala C# java
25. Apply Property-based Testing • Check idempotence f(f(x)) == f(x) ex : UpperCase, Create / Delete • Check roundtripping from(to(x)) == x ex : Serialization, PUT / GET, Reverse, Negate • Check invariants invariant(f(x)) == invariant(x) ex : Reverse, Map • Check commutativity f(x, y) == f(y, x) ex : Addition, Min, Max • Check rewriting f(x) == new_f(x) when rewriting or optimizing implementations • “Upgrade” parameterized tests Replace hardcoded values or discover new test cases (edge cases) • … We can use it to test general properties of our functions
26. Anti-patterns • Re-run on failures It throws away the specific failure (new generated values) Instead : investigate failure and add a new example-based test • Re-implement production code Leak all your production code in your properties • Perform ad-hoc filtering Do not filter input values by ourselves Instead : use the framework support
27. QuickCheck in your favorite language source : https://hypothesis.works/articles/quickcheck-in-every-language/
28. Conclusion Think about your last development, how this approach could have helped you ?
29. Resources ScalaCheck documentation : https://github.com/typelevel/scalacheck/blob/main/doc/UserGuide.md Scala exercises on PBT : https://www.scala-exercises.org/scalacheck/generators "Functions for nothing, and your tests for free" Property-based testing and F# - George Pollard Property based testing - step by step : https://www.leadingagile.com/2018/04/step-by-step-toward-property-based-testing/