Slide 1

Slide 1 text

@yot88 A Journey to Property-based Testing

Slide 2

Slide 2 text

Calculator In mob : • Open the repository • Write tests to check the behavior of the Calculator class https://github.com/ythirion/property-based-testing-kata object Calculator { def add(x: Int, y: Int): Int = x + y } Source code is available in scala (scalatest / scalacheck) / C# (xunit / FsCheck / Language-Ext / FluentAssertions) / java (Junit5 / quick-check / vavr / AssertJ)

Slide 3

Slide 3 text

Calculator - Debriefing • Which tests did you write ? • What edge cases did you find ? • What are the limits of this example-based testing approach ? Given (1, 3) When I add them Then I expect 4 Given (-1, 3) When I add them Then I expect 2 Given (0, 99) When I add them Then I expect 99 Could we test it differently to cover more inputs ? it should "return their correct sum when I add 2 random numbers" in { Range(0, 100) .foreach(_ => { val x = random.nextInt() val y = random.nextInt() assert(add(x, y) == x + y) }) } Implementation leak in tests What does correct mean in this context ?

Slide 4

Slide 4 text

Example-based testing Do we cover the full scope of possible inputs ? Does the developed feature is compliant with what is expected ? When we write Unit tests : • we often focus on examples • examples are good to understand requirements Given (x, y, ...) When I [call the subject under test] with (x, y, ...) Then I expect this (output) Limitation : we only focus on the input scope we identified

Slide 5

Slide 5 text

Property-based testing – at our rescue ? PBT

Slide 6

Slide 6 text

What is Property-based testing • A property is the combination of an invariant with an input values generator. For each generated value, the invariant is treated as a predicate and checked whether it yields true or false for that value. • As soon as there is one value which yields false, the property is said to be falsified, and checking is aborted. • If a property cannot be invalidated after a specific amount of sample data, the property is assumed to be satisfied.

Slide 7

Slide 7 text

In other words, a property is something like : for all (x, y, ...) such as precondition (x, y, ...) holds property (x, y, ...) is satisfied run it multiple times generate random inputs (based on specified generators) check input pre-conditions Failure: not an error -> invalid entry run the predicate Failure: a counter-example will try to shrink the entry • Describe the input • Describe the properties of the output • Have the computer try lots of random examples • Check if it fails

Slide 8

Slide 8 text

@yot88 Which properties do we know on Addition ? Compare addition with other Math operations • What about parameter order ? (compared to subtract) The parameter order does not matter • “add 1” twice is the same as doing “add 2” (compared to multiply) • What about adding 0 ? (compared to multiply) Adding 0 does nothing We are confident the implementation is correct if we test these properties It applies to all inputs

Slide 9

Slide 9 text

Let’s write some tests it should "when I add 2 numbers the result should not depend on parameter order" in { Range(0, times) .map(_ => random.nextInt()) .foreach(x => { val y = random.nextInt() assert(add(x, y) == add(y, x)) }) } it should "when I add 1 twice it's the same as adding 2 once" in { Range(0, times) .map(_ => random.nextInt()) .foreach(x => assert(add(add(x, 1), 1) == add(x, 2))) } it should "when I add 0 to a random number is the same than doing nothing on this number" in { Range(0, times) .map(_ => random.nextInt()) .foreach(x => assert(add(x, 0) == x)) } Commutativity Associativity Identity

Slide 10

Slide 10 text

ScalaCheck “ScalaCheck is a tool for testing Scala and Java programs, based on property specifications and automatic test data generation. The basic idea is that you define a property that specifies the behaviour of a method or some unit of code, and ScalaCheck checks that the property holds. All test data are generated automatically in a random fashion, so you don’t have to worry about any missed cases.” - ScalaCheck : The Definitive Guide libraryDependencies += "org.scalatest" %% "scalatest" % "3.2.9" % Test libraryDependencies += "org.scalactic" %% "scalactic" % "3.2.9" % Test libraryDependencies += "org.scalatestplus" %% "scalacheck-1-15" % "3.2.9.0" % "test" https://github.com/typelevel/scalacheck/blob/main/doc/UserGuide.md A way to generate (and shrink) random test data

Slide 11

Slide 11 text

Addition properties with ScalaCheck object CalculatorProperties extends Properties("add") { property("commutativity") = forAll { (x: Int, y: Int) => add(x, y) == add(y, x) } property("associativity") = forAll { (x: Int) => add(add(x, 1), 1) == add(x, 2) } property("identity") = forAll { (x: Int) => add(x, 0) == x } }

Slide 12

Slide 12 text

How to read it ? property("commutativity") = forAll { (x: Int, y: Int) => add(x, y) == add(y, x) } For all possible x, y run the block of code that follows -1, 1887999732 1887999732, -2147483648 -2147483648, 1 1, -738646202 -738646202, 454567881 454567881, -1 -1, 1321200746 1321200746, 2147483647 2147483647, -2147483648 -2147483648, 912114712 912114712, 703322924 703322924, -802049796 -802049796, 1 1, 2147483647 2147483647, -1707626195 -1707626195, -1 -1, -1626087642 -1626087642, 2147483647 2147483647, 546871561 546871561, -651551026 -651551026, 0 0, 1 1, 1 1, 958219829 958219829, 1772245900 … Can help us identify values outside from our business frame

Slide 13

Slide 13 text

Integration with ScalaTest class CalculatorPropertiesFlatSpec extends AnyFlatSpec with Checkers { "add" should "be commutative" in { check(forAll { (x: Int, y: Int) => add(x, y) == add(y, x) }) } "add" should "be associative" in { check(forAll { (x: Int) => add(add(x, 1), 1) == add(x, 2) }) } "0" should "be identity" in { check(forAll { (x: Int) => add(x, 0) == x }) } } https://www.scalatest.org/user_guide/writing_scalacheck_style_properties

Slide 14

Slide 14 text

@yot88 What happens when it fails ScalaCheck provides us the values for which the property is falsified

Slide 15

Slide 15 text

Concepts • By default, each function is tested 10 times 10 times if all tests succeed (differs from scalacheck-only behavior) less than that if a test is falsified • Tests are run with randomly-generated data • ScalaCheck has built-in generators We can also write our own generators

Slide 16

Slide 16 text

PBT in real life

Slide 17

Slide 17 text

Postal Parcel Open `PostalParcel` : • Identify properties – 5’ What are the invariants ? • Write them in `PostalParcelPropertiesFlatSpec` - 15’ • Collective debriefing – 5’ object PostalParcel { val maxWeight = 20.0 val maxDeliveryCosts = 4.99 val minDeliveryCosts = 1.99 def fromDouble(weight: Double): Option[PostalParcel] = if (weight > 0) Some(PostalParcel(weight)) else None def calculateDeliveryCosts( postalParcel: Option[PostalParcel] ): Option[Double] = { postalParcel.map(p => if (p.weight > maxWeight) maxDeliveryCosts else minDeliveryCosts ) } } scala C# java

Slide 18

Slide 18 text

Bank withdrawal • Open `AccountService` : • Identify properties – 5’ What should we do to check those properties ? object AccountService { def withdraw(account: Account, command: Withdraw): Either[String, Account] = { if (hasAlreadyBeenApplied(account, command)) Right(account) else applyWithdraw(account, command) } private def applyWithdraw( account: Account, command: Withdraw ): Either[String, Account] = { command.amount.value match { case amount if exceedMaxWithdrawal(account, amount) => Left(s"Amount exceeding your limit of ${account.maxWithdrawal}") case amount if exceedBalance(account, amount) => Left(s"Insufficient balance to withdraw : $amount") case _ => Right( account.copy( balance = account.balance - command.amount.value, withdraws = account.withdraws :+ command ) ) } } private def hasAlreadyBeenApplied(account: Account, command: Withdraw) = account.withdraws.contains(command) private def exceedMaxWithdrawal( account: Account, withdrawAmount: Double ): Boolean = withdrawAmount >= account.maxWithdrawal private def exceedBalance(account: Account, withdrawAmount: Double): Boolean = withdrawAmount > account.balance && !account.isOverdraftAuthorized }

Slide 19

Slide 19 text

Bank withdrawal - Properties • "account balance" should "be decremented at least from the withdraw amount” • "account balance" should "be decremented at least from the withdraw amount when insufficient balance but overdraft authorized” • "withdraw" should "not be allowed when withdraw amount >= maxWithdrawal” • "withdraw" should "not be allowed when insufficient balance and no overdraft” • "withdraw" should "be idempotent"

Slide 20

Slide 20 text

Bank withdrawal using properties and when (WithdrawProperties) "withdraw" should "not be allowed when insufficient balance and no overdraft" in { checkProperty( (account, command) => { withInsufficientBalance(account, command) && withoutOverdraftAuthorized(account) && withoutReachingMaxWithdrawal(account, command) }, (_, _, debitedAccount) => { debitedAccount.left.value .startsWith("Insufficient balance to withdraw") } ) } Need to discard a lot of generated inputs implicit override val generatorDrivenConfig: PropertyCheckConfiguration = PropertyCheckConfiguration(minSize = 10, maxDiscardedFactor = 30) Need to increase maxDiscardedFactor to check the properties… Not a valid way to check our business invariants implicit val accountGenerator: Arbitrary[Account] = Arbitrary { for { balance <- Arbitrary.arbitrary[Double] isOverdraftAuthorized <- Arbitrary.arbitrary[Boolean] maxWithdrawal <- Arbitrary.arbitrary[Double] } yield Account(balance, isOverdraftAuthorized, maxWithdrawal) } implicit val withdrawCommandGenerator: Arbitrary[Withdraw] = Arbitrary { for { clientId <- Arbitrary.arbitrary[UUID] amount <- posNum[Double] requestDate <- Arbitrary.arbitrary[LocalDate] } yield Withdraw(clientId, Amount.from(amount).get, requestDate) } scala

Slide 21

Slide 21 text

Bank withdrawal using properties and commandGenerator + builder "withdraw" should "not be allowed when insufficient balance and no overdraft" in { checkProperty( (account, command) => account .withInsufficientBalance(command) .withoutOverDraftAuthorized() .withoutReachingMaxWithdrawal(command), (_, _, debitedAccount) => debitedAccount.left.value .startsWith("Insufficient balance to withdraw") ) } private def checkProperty( accountConfiguration: (AccountBuilder, Withdraw) => AccountBuilder, property: (Account, Withdraw, Either[String, Account]) => Boolean ): Assertion = { check(forAll { (command: Withdraw) => val account = accountConfiguration(AccountBuilder(), command).build() property(account, command, AccountService.withdraw(account, command)) }) } final case class AccountBuilder( balance: Double = 0, isOverdraftAuthorized: Boolean = false, maxWithdrawal: Double = 0 ) { def withEnoughMoney(command: Withdraw): AccountBuilder = copy(balance = command.amount.value + arbitraryPositiveNumber) def withdrawAmountReachingMaxWithdrawal(command: Withdraw): AccountBuilder = copy(maxWithdrawal = command.amount.value - arbitraryPositiveNumber) … We now instantiate accounts based on random commands with a Builder Do not need to change ScalaCheck configuration anymore Builder is maybe leaking too much of our business… scala C# java

Slide 22

Slide 22 text

private val `withdraw examples` = Table( ( "balance", "isOverdraftAuthorized", "maxWithdrawal", "withdrawAmount", "expectedResult" ), (10_000, false, 1200, 1199.99, Right(8800.01)), (0, true, 500, 50d, Right(-50)), (1, false, 1200, 1199.99, Left("Insufficient balance to withdraw")), (0, false, 500, 50d, Left("Insufficient balance to withdraw")), (10_000, true, 1200, 1200.0001, Left("Amount exceeding your limit of")), (0, false, 500, 500d, Left("Amount exceeding your limit of")) ) Bank withdrawal – how we would test it in real life ? Use ParameterizedTests -> TableDrivenPropertyChecks Describe test cases Check implementation based on them We can use Properties to help us identify edge cases and add new examples in our table A better approach for this kind of use cases : • Example-based with Table to allow easy addition of new tests cases • Use PBT as a cases finders • Use PBT to check idem potence "withdraw" should "pass examples" in { forAll(`withdraw examples`) { ( balance, isOverdraftAuthorized, maxWithdrawal, withdrawAmount, expectedResult ) => val command = toWithdraw(withdrawAmount) val debitedAccount = AccountService.withdraw( Account(balance, isOverdraftAuthorized, maxWithdrawal), command ) expectedResult match { case Left(expectedErrorMessage) => debitedAccount.left.value should startWith(expectedErrorMessage) case Right(expectedBalance) => debitedAccount.value.balance should be(expectedBalance) debitedAccount.value.withdraws should contain(command) debitedAccount.value.withdraws should have length 1 } } } scala C# java

Slide 23

Slide 23 text

Rental Calculator PBT on legacy code • Open `RentalCalculator` : Imagine this code is running in production and business is happy with it Imagine you need to adapt it, but you need to be sure that you don’t introduce any regression class RentalCalculator(val rentals: List[Rental]) { private var _amount = .0 private var _calculated = false def amount(): Double = _amount def calculated(): Boolean = _calculated def calculateRental(): Either[String, String] = { if (rentals.isEmpty) Left("No rentals !!!") else { val result = new StringBuilder for (rental <- rentals) { if (!_calculated) this._amount += rental.amount result.append(formatLine(rental, _amount)) } result.append(f"Total amount | ${this._amount}%.2f") _calculated = true Right(result.toString) } } private def formatLine(rental: Rental, amount: Double) = f"${rental.date} : ${rental.label} | ${rental.amount}%.2f${lineSeparator()}" } How PBT can help us ?

Slide 24

Slide 24 text

Rental Calculator PBT on legacy code • Duplicate the production code in a new file – NewRentalCalculator for example • Write a property checking that – 5’ f(x) == new_f(x) Ensure that your test fails if you change the new implementation • Now you can refactor the new implementation code – 15’ Identify the code smells Refactor the code Run the property at each changes • You can now plug the caller on the new implementation Delete the legacy implementation Remove the property Seeing a test failing is as important as seeing it passing Can use this technique for any rewriting / optimizing refactoring scala C# java

Slide 25

Slide 25 text

Apply Property-based Testing • Check idempotence f(f(x)) == f(x) ex : UpperCase, Create / Delete • Check roundtripping from(to(x)) == x ex : Serialization, PUT / GET, Reverse, Negate • Check invariants invariant(f(x)) == invariant(x) ex : Reverse, Map • Check commutativity f(x, y) == f(y, x) ex : Addition, Min, Max • Check rewriting f(x) == new_f(x) when rewriting or optimizing implementations • “Upgrade” parameterized tests Replace hardcoded values or discover new test cases (edge cases) • … We can use it to test general properties of our functions

Slide 26

Slide 26 text

Anti-patterns • Re-run on failures It throws away the specific failure (new generated values) Instead : investigate failure and add a new example-based test • Re-implement production code Leak all your production code in your properties • Perform ad-hoc filtering Do not filter input values by ourselves Instead : use the framework support

Slide 27

Slide 27 text

QuickCheck in your favorite language source : https://hypothesis.works/articles/quickcheck-in-every-language/

Slide 28

Slide 28 text

Conclusion Think about your last development, how this approach could have helped you ?

Slide 29

Slide 29 text

Resources ScalaCheck documentation : https://github.com/typelevel/scalacheck/blob/main/doc/UserGuide.md Scala exercises on PBT : https://www.scala-exercises.org/scalacheck/generators "Functions for nothing, and your tests for free" Property-based testing and F# - George Pollard Property based testing - step by step : https://www.leadingagile.com/2018/04/step-by-step-toward-property-based-testing/