A journey to Property-Based Testing

Explore what is Property-Based Testing and how you can use this approach in your daily dev life.
Support of a workshop explained here : https://yoan-thirion.gitbook.io/knowledge-base/software-craftsmanship/code-katas/improve-your-software-quality-with-property-based-testing/a-journey-to-property-based-testing

Transcript

1. @yot88
   A Journey
   to Property-based Testing
   

   View Slide

2. Calculator
   In mob :
   • Open the repository
   • Write tests to check the behavior of the Calculator class
   https://github.com/ythirion/property-based-testing-kata
   object Calculator {
   def add(x: Int, y: Int): Int = x + y
   }
   Source code is available in scala (scalatest / scalacheck) / C# (xunit / FsCheck / Language-Ext / FluentAssertions) / java (Junit5 / quick-check / vavr / AssertJ)
   

   View Slide

3. Calculator - Debriefing
   • Which tests did you write ?
   • What edge cases did you find ?
   • What are the limits of this example-based testing approach ?
   Given (1, 3)
   When I add them
   Then I expect 4
   Given (-1, 3)
   When I add them
   Then I expect 2
   Given (0, 99)
   When I add them
   Then I expect 99
   Could we test it differently to cover
   more inputs ?
   it should "return their correct sum when I add 2 random numbers" in {
   Range(0, 100)
   .foreach(_ => {
   val x = random.nextInt()
   val y = random.nextInt()
   assert(add(x, y) == x + y)
   })
   }
   Implementation leak in tests
   What does correct mean in this context ?
   

   View Slide

4. Example-based testing
   Do we cover the full scope of
   possible inputs ?
   Does the developed feature is compliant with what is expected ?
   When we write Unit tests :
   • we often focus on examples
   • examples are good to understand requirements
   Given (x, y, ...)
   When I [call the subject under test] with (x, y, ...)
   Then I expect this (output)
   Limitation : we only focus on the input scope we identified
   

   View Slide

5. Property-based testing – at our rescue ?
   PBT
   

   View Slide

6. What is Property-based testing
   • A property is the combination of an invariant with an input values generator.
   For each generated value, the invariant is treated as a predicate and checked whether it yields true or false for that value.
   • As soon as there is one value which yields false, the property is said to be falsified, and checking is aborted.
   • If a property cannot be invalidated after a specific amount of sample data, the property is assumed to be
   satisfied.
   

   View Slide

7. In other words, a property is something like :
   for all (x, y, ...)
   such as precondition (x, y, ...) holds
   property (x, y, ...) is satisfied
   run it multiple times generate random inputs
   (based on specified generators)
   check input pre-conditions
   Failure: not an error -> invalid entry
   run the predicate
   Failure: a counter-example
   will try to shrink the entry
   • Describe the input
   • Describe the properties of the output
   • Have the computer try lots of random examples
   • Check if it fails
   

   View Slide

8. @yot88
   Which properties do we know on Addition ?
   Compare addition with other Math operations
   • What about parameter order ? (compared to subtract)
   The parameter order does not matter
   • “add 1” twice is the same as doing “add 2” (compared to multiply)
   • What about adding 0 ? (compared to multiply)
   Adding 0 does nothing
   We are confident the implementation is correct if we test these properties
   It applies to all inputs
   

   View Slide

9. Let’s write some tests
   it should "when I add 2 numbers the result should not depend on parameter
   order" in {
   Range(0, times)
   .map(_ => random.nextInt())
   .foreach(x => {
   val y = random.nextInt()
   assert(add(x, y) == add(y, x))
   })
   }
   it should "when I add 1 twice it's the same as adding 2 once" in {
   Range(0, times)
   .map(_ => random.nextInt())
   .foreach(x => assert(add(add(x, 1), 1) == add(x, 2)))
   }
   it should "when I add 0 to a random number is the same than doing
   nothing on this number" in {
   Range(0, times)
   .map(_ => random.nextInt())
   .foreach(x => assert(add(x, 0) == x))
   }
   Commutativity
   Associativity
   Identity
   

   View Slide

10. ScalaCheck
    “ScalaCheck is a tool for testing Scala and Java programs, based on property specifications and
    automatic test data generation. The basic idea is that you define a property that specifies the
    behaviour of a method or some unit of code, and ScalaCheck checks that the property holds. All test
    data are generated automatically in a random fashion, so you don’t have to worry about any missed
    cases.” - ScalaCheck : The Definitive Guide
    libraryDependencies += "org.scalatest" %% "scalatest" % "3.2.9" % Test
    libraryDependencies += "org.scalactic" %% "scalactic" % "3.2.9" % Test
    libraryDependencies += "org.scalatestplus" %% "scalacheck-1-15" % "3.2.9.0" % "test"
    https://github.com/typelevel/scalacheck/blob/main/doc/UserGuide.md
    A way to generate (and
    shrink)
    random test data
    

    View Slide

11. Addition properties with ScalaCheck
    object CalculatorProperties extends Properties("add") {
    property("commutativity") = forAll { (x: Int, y: Int) =>
    add(x, y) == add(y, x)
    }
    property("associativity") = forAll { (x: Int) =>
    add(add(x, 1), 1) == add(x, 2)
    }
    property("identity") = forAll { (x: Int) =>
    add(x, 0) == x
    }
    }
    

    View Slide

12. How to read it ?
    property("commutativity") = forAll { (x: Int, y: Int) =>
    add(x, y) == add(y, x)
    }
    For all possible x, y run the block of code that follows
    -1, 1887999732
    1887999732, -2147483648
    -2147483648, 1
    1, -738646202
    -738646202, 454567881
    454567881, -1
    -1, 1321200746
    1321200746, 2147483647
    2147483647, -2147483648
    -2147483648, 912114712
    912114712, 703322924
    703322924, -802049796
    -802049796, 1
    1, 2147483647
    2147483647, -1707626195
    -1707626195, -1
    -1, -1626087642
    -1626087642, 2147483647
    2147483647, 546871561
    546871561, -651551026
    -651551026, 0
    0, 1
    1, 1
    1, 958219829
    958219829, 1772245900
    …
    Can help us identify values outside from our business frame
    

    View Slide

13. Integration with ScalaTest
    class CalculatorPropertiesFlatSpec extends AnyFlatSpec with Checkers {
    "add" should "be commutative" in {
    check(forAll { (x: Int, y: Int) =>
    add(x, y) == add(y, x)
    })
    }
    "add" should "be associative" in {
    check(forAll { (x: Int) =>
    add(add(x, 1), 1) == add(x, 2)
    })
    }
    "0" should "be identity" in {
    check(forAll { (x: Int) =>
    add(x, 0) == x
    })
    }
    }
    https://www.scalatest.org/user_guide/writing_scalacheck_style_properties
    

    View Slide

14. @yot88
    What happens when it fails
    ScalaCheck provides us the values for which the property is falsified
    

    View Slide

15. Concepts
    • By default, each function is tested 10 times
    10 times if all tests succeed (differs from scalacheck-only behavior)
    less than that if a test is falsified
    • Tests are run with randomly-generated data
    • ScalaCheck has built-in generators
    We can also write our own generators
    

    View Slide

16. PBT in real life
    

    View Slide

17. Postal Parcel
    Open `PostalParcel` :
    • Identify properties – 5’
    What are the invariants ?
    • Write them in `PostalParcelPropertiesFlatSpec` - 15’
    • Collective debriefing – 5’
    object PostalParcel {
    val maxWeight = 20.0
    val maxDeliveryCosts = 4.99
    val minDeliveryCosts = 1.99
    def fromDouble(weight: Double): Option[PostalParcel] =
    if (weight > 0) Some(PostalParcel(weight)) else None
    def calculateDeliveryCosts(
    postalParcel: Option[PostalParcel]
    ): Option[Double] = {
    postalParcel.map(p =>
    if (p.weight > maxWeight) maxDeliveryCosts else minDeliveryCosts
    )
    }
    }
    scala C# java
    

    View Slide

18. Bank withdrawal
    • Open `AccountService` :
    • Identify properties – 5’
    What should we do to check those properties ?
    object AccountService {
    def withdraw(account: Account, command: Withdraw): Either[String, Account] = {
    if (hasAlreadyBeenApplied(account, command))
    Right(account)
    else
    applyWithdraw(account, command)
    }
    private def applyWithdraw(
    account: Account,
    command: Withdraw
    ): Either[String, Account] = {
    command.amount.value match {
    case amount if exceedMaxWithdrawal(account, amount) =>
    Left(s"Amount exceeding your limit of ${account.maxWithdrawal}")
    case amount if exceedBalance(account, amount) =>
    Left(s"Insufficient balance to withdraw : $amount")
    case _ =>
    Right(
    account.copy(
    balance = account.balance - command.amount.value,
    withdraws = account.withdraws :+ command
    )
    )
    }
    }
    private def hasAlreadyBeenApplied(account: Account, command: Withdraw) =
    account.withdraws.contains(command)
    private def exceedMaxWithdrawal(
    account: Account,
    withdrawAmount: Double
    ): Boolean =
    withdrawAmount >= account.maxWithdrawal
    private def exceedBalance(account: Account, withdrawAmount: Double): Boolean =
    withdrawAmount > account.balance && !account.isOverdraftAuthorized
    }
    

    View Slide

19. Bank withdrawal - Properties
    • "account balance" should "be decremented at least from the withdraw amount”
    • "account balance" should "be decremented at least from the withdraw amount when insufficient balance but overdraft authorized”
    • "withdraw" should "not be allowed when withdraw amount >= maxWithdrawal”
    • "withdraw" should "not be allowed when insufficient balance and no overdraft”
    • "withdraw" should "be idempotent"
    

    View Slide

20. Bank withdrawal using properties and when (WithdrawProperties)
    "withdraw" should "not be allowed when insufficient balance and no overdraft" in {
    checkProperty(
    (account, command) => {
    withInsufficientBalance(account, command) &&
    withoutOverdraftAuthorized(account) &&
    withoutReachingMaxWithdrawal(account, command)
    },
    (_, _, debitedAccount) => {
    debitedAccount.left.value
    .startsWith("Insufficient balance to withdraw")
    }
    )
    } Need to discard a lot of generated inputs
    implicit override val generatorDrivenConfig: PropertyCheckConfiguration =
    PropertyCheckConfiguration(minSize = 10, maxDiscardedFactor = 30)
    Need to increase maxDiscardedFactor to check the properties…
    Not a valid way to check our business invariants
    implicit val accountGenerator: Arbitrary[Account] = Arbitrary {
    for {
    balance <- Arbitrary.arbitrary[Double]
    isOverdraftAuthorized <- Arbitrary.arbitrary[Boolean]
    maxWithdrawal <- Arbitrary.arbitrary[Double]
    } yield Account(balance, isOverdraftAuthorized, maxWithdrawal)
    }
    implicit val withdrawCommandGenerator: Arbitrary[Withdraw] = Arbitrary {
    for {
    clientId <- Arbitrary.arbitrary[UUID]
    amount <- posNum[Double]
    requestDate <- Arbitrary.arbitrary[LocalDate]
    } yield Withdraw(clientId, Amount.from(amount).get, requestDate)
    }
    scala
    

    View Slide

21. Bank withdrawal using properties and commandGenerator + builder
    "withdraw" should "not be allowed when insufficient balance and no overdraft" in {
    checkProperty(
    (account, command) =>
    account
    .withInsufficientBalance(command)
    .withoutOverDraftAuthorized()
    .withoutReachingMaxWithdrawal(command),
    (_, _, debitedAccount) =>
    debitedAccount.left.value
    .startsWith("Insufficient balance to withdraw")
    )
    }
    private def checkProperty(
    accountConfiguration: (AccountBuilder, Withdraw) => AccountBuilder,
    property: (Account, Withdraw, Either[String, Account]) => Boolean
    ): Assertion = {
    check(forAll { (command: Withdraw) =>
    val account = accountConfiguration(AccountBuilder(), command).build()
    property(account, command, AccountService.withdraw(account, command))
    })
    }
    final case class AccountBuilder(
    balance: Double = 0,
    isOverdraftAuthorized: Boolean = false,
    maxWithdrawal: Double = 0
    ) {
    def withEnoughMoney(command: Withdraw): AccountBuilder =
    copy(balance = command.amount.value + arbitraryPositiveNumber)
    def withdrawAmountReachingMaxWithdrawal(command: Withdraw): AccountBuilder =
    copy(maxWithdrawal = command.amount.value - arbitraryPositiveNumber)
    …
    We now instantiate accounts based on random
    commands with a Builder
    Do not need to change ScalaCheck
    configuration anymore
    Builder is maybe leaking too much of our business…
    scala C# java
    

    View Slide

22. private val `withdraw examples` =
    Table(
    (
    "balance",
    "isOverdraftAuthorized",
    "maxWithdrawal",
    "withdrawAmount",
    "expectedResult"
    ),
    (10_000, false, 1200, 1199.99, Right(8800.01)),
    (0, true, 500, 50d, Right(-50)),
    (1, false, 1200, 1199.99, Left("Insufficient balance to withdraw")),
    (0, false, 500, 50d, Left("Insufficient balance to withdraw")),
    (10_000, true, 1200, 1200.0001, Left("Amount exceeding your limit of")),
    (0, false, 500, 500d, Left("Amount exceeding your limit of"))
    )
    Bank withdrawal – how we would test it in real life ?
    Use ParameterizedTests -> TableDrivenPropertyChecks
    Describe test cases
    Check implementation based on them
    We can use Properties to help us identify edge cases
    and add new examples in our table
    A better approach for this kind of use cases :
    • Example-based with Table to allow easy addition of new tests cases
    • Use PBT as a cases finders
    • Use PBT to check idem potence
    "withdraw" should "pass examples" in {
    forAll(`withdraw examples`) {
    (
    balance,
    isOverdraftAuthorized,
    maxWithdrawal,
    withdrawAmount,
    expectedResult
    ) =>
    val command = toWithdraw(withdrawAmount)
    val debitedAccount =
    AccountService.withdraw(
    Account(balance, isOverdraftAuthorized, maxWithdrawal),
    command
    )
    expectedResult match {
    case Left(expectedErrorMessage) =>
    debitedAccount.left.value should startWith(expectedErrorMessage)
    case Right(expectedBalance) =>
    debitedAccount.value.balance should be(expectedBalance)
    debitedAccount.value.withdraws should contain(command)
    debitedAccount.value.withdraws should have length 1
    }
    }
    }
    scala C# java
    

    View Slide

23. Rental Calculator PBT on legacy code
    • Open `RentalCalculator` :
    Imagine this code is running in production and business is happy with it
    Imagine you need to adapt it, but you need to be sure that you don’t introduce any regression
    class RentalCalculator(val rentals: List[Rental]) {
    private var _amount = .0
    private var _calculated = false
    def amount(): Double = _amount
    def calculated(): Boolean = _calculated
    def calculateRental(): Either[String, String] = {
    if (rentals.isEmpty)
    Left("No rentals !!!")
    else {
    val result = new StringBuilder
    for (rental <- rentals) {
    if (!_calculated) this._amount += rental.amount
    result.append(formatLine(rental, _amount))
    }
    result.append(f"Total amount | ${this._amount}%.2f")
    _calculated = true
    Right(result.toString)
    }
    }
    private def formatLine(rental: Rental, amount: Double) =
    f"${rental.date} : ${rental.label} | ${rental.amount}%.2f${lineSeparator()}"
    }
    How PBT can help us ?
    

    View Slide

24. Rental Calculator PBT on legacy code
    • Duplicate the production code in a new file – NewRentalCalculator for example
    • Write a property checking that – 5’
    f(x) == new_f(x)
    Ensure that your test fails if you change the new implementation
    • Now you can refactor the new implementation code – 15’
    Identify the code smells
    Refactor the code
    Run the property at each changes
    • You can now plug the caller on the new implementation
    Delete the legacy implementation
    Remove the property
    Seeing a test failing is as important
    as seeing it passing
    Can use this technique for any rewriting / optimizing refactoring
    scala C# java
    

    View Slide

25. Apply Property-based Testing
    • Check idempotence
    f(f(x)) == f(x)
    ex : UpperCase, Create / Delete
    • Check roundtripping
    from(to(x)) == x
    ex : Serialization, PUT / GET, Reverse, Negate
    • Check invariants
    invariant(f(x)) == invariant(x)
    ex : Reverse, Map
    • Check commutativity
    f(x, y) == f(y, x)
    ex : Addition, Min, Max
    • Check rewriting
    f(x) == new_f(x)
    when rewriting or optimizing implementations
    • “Upgrade” parameterized tests
    Replace hardcoded values or discover new test cases (edge cases)
    • …
    We can use it to test general properties of our functions
    

    View Slide

26. Anti-patterns
    • Re-run on failures
    It throws away the specific failure (new generated values)
    Instead : investigate failure and add a new example-based test
    • Re-implement production code
    Leak all your production code in your properties
    • Perform ad-hoc filtering
    Do not filter input values by ourselves
    Instead : use the framework support
    

    View Slide

27. QuickCheck in your favorite language
    source : https://hypothesis.works/articles/quickcheck-in-every-language/
    

    View Slide

28. Conclusion
    Think about your last development, how this approach could have helped you ?
    

    View Slide

29. Resources
    ScalaCheck documentation : https://github.com/typelevel/scalacheck/blob/main/doc/UserGuide.md
    Scala exercises on PBT : https://www.scala-exercises.org/scalacheck/generators
    "Functions for nothing, and your tests for free" Property-based testing and F# - George Pollard
    Property based testing - step by step : https://www.leadingagile.com/2018/04/step-by-step-toward-property-based-testing/
    

    View Slide