Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A journey to Property-Based Testing

Yoan
January 10, 2022

A journey to Property-Based Testing

Explore what is Property-Based Testing and how you can use this approach in your daily dev life.
Support of a workshop explained here : https://yoan-thirion.gitbook.io/knowledge-base/software-craftsmanship/code-katas/improve-your-software-quality-with-property-based-testing/a-journey-to-property-based-testing

Yoan

January 10, 2022
Tweet

More Decks by Yoan

Other Decks in Programming

Transcript

  1. @yot88
    A Journey
    to Property-based Testing

    View Slide

  2. Calculator
    In mob :
    • Open the repository
    • Write tests to check the behavior of the Calculator class
    https://github.com/ythirion/property-based-testing-kata
    object Calculator {
    def add(x: Int, y: Int): Int = x + y
    }
    Source code is available in scala (scalatest / scalacheck) / C# (xunit / FsCheck / Language-Ext / FluentAssertions) / java (Junit5 / quick-check / vavr / AssertJ)

    View Slide

  3. Calculator - Debriefing
    • Which tests did you write ?
    • What edge cases did you find ?
    • What are the limits of this example-based testing approach ?
    Given (1, 3)
    When I add them
    Then I expect 4
    Given (-1, 3)
    When I add them
    Then I expect 2
    Given (0, 99)
    When I add them
    Then I expect 99
    Could we test it differently to cover
    more inputs ?
    it should "return their correct sum when I add 2 random numbers" in {
    Range(0, 100)
    .foreach(_ => {
    val x = random.nextInt()
    val y = random.nextInt()
    assert(add(x, y) == x + y)
    })
    }
    Implementation leak in tests
    What does correct mean in this context ?

    View Slide

  4. Example-based testing
    Do we cover the full scope of
    possible inputs ?
    Does the developed feature is compliant with what is expected ?
    When we write Unit tests :
    • we often focus on examples
    • examples are good to understand requirements
    Given (x, y, ...)
    When I [call the subject under test] with (x, y, ...)
    Then I expect this (output)
    Limitation : we only focus on the input scope we identified

    View Slide

  5. Property-based testing – at our rescue ?
    PBT

    View Slide

  6. What is Property-based testing
    • A property is the combination of an invariant with an input values generator.
    For each generated value, the invariant is treated as a predicate and checked whether it yields true or false for that value.
    • As soon as there is one value which yields false, the property is said to be falsified, and checking is aborted.
    • If a property cannot be invalidated after a specific amount of sample data, the property is assumed to be
    satisfied.

    View Slide

  7. In other words, a property is something like :
    for all (x, y, ...)
    such as precondition (x, y, ...) holds
    property (x, y, ...) is satisfied
    run it multiple times generate random inputs
    (based on specified generators)
    check input pre-conditions
    Failure: not an error -> invalid entry
    run the predicate
    Failure: a counter-example
    will try to shrink the entry
    • Describe the input
    • Describe the properties of the output
    • Have the computer try lots of random examples
    • Check if it fails

    View Slide

  8. @yot88
    Which properties do we know on Addition ?
    Compare addition with other Math operations
    • What about parameter order ? (compared to subtract)
    The parameter order does not matter
    • “add 1” twice is the same as doing “add 2” (compared to multiply)
    • What about adding 0 ? (compared to multiply)
    Adding 0 does nothing
    We are confident the implementation is correct if we test these properties
    It applies to all inputs

    View Slide

  9. Let’s write some tests
    it should "when I add 2 numbers the result should not depend on parameter
    order" in {
    Range(0, times)
    .map(_ => random.nextInt())
    .foreach(x => {
    val y = random.nextInt()
    assert(add(x, y) == add(y, x))
    })
    }
    it should "when I add 1 twice it's the same as adding 2 once" in {
    Range(0, times)
    .map(_ => random.nextInt())
    .foreach(x => assert(add(add(x, 1), 1) == add(x, 2)))
    }
    it should "when I add 0 to a random number is the same than doing
    nothing on this number" in {
    Range(0, times)
    .map(_ => random.nextInt())
    .foreach(x => assert(add(x, 0) == x))
    }
    Commutativity
    Associativity
    Identity

    View Slide

  10. ScalaCheck
    “ScalaCheck is a tool for testing Scala and Java programs, based on property specifications and
    automatic test data generation. The basic idea is that you define a property that specifies the
    behaviour of a method or some unit of code, and ScalaCheck checks that the property holds. All test
    data are generated automatically in a random fashion, so you don’t have to worry about any missed
    cases.” - ScalaCheck : The Definitive Guide
    libraryDependencies += "org.scalatest" %% "scalatest" % "3.2.9" % Test
    libraryDependencies += "org.scalactic" %% "scalactic" % "3.2.9" % Test
    libraryDependencies += "org.scalatestplus" %% "scalacheck-1-15" % "3.2.9.0" % "test"
    https://github.com/typelevel/scalacheck/blob/main/doc/UserGuide.md
    A way to generate (and
    shrink)
    random test data

    View Slide

  11. Addition properties with ScalaCheck
    object CalculatorProperties extends Properties("add") {
    property("commutativity") = forAll { (x: Int, y: Int) =>
    add(x, y) == add(y, x)
    }
    property("associativity") = forAll { (x: Int) =>
    add(add(x, 1), 1) == add(x, 2)
    }
    property("identity") = forAll { (x: Int) =>
    add(x, 0) == x
    }
    }

    View Slide

  12. How to read it ?
    property("commutativity") = forAll { (x: Int, y: Int) =>
    add(x, y) == add(y, x)
    }
    For all possible x, y run the block of code that follows
    -1, 1887999732
    1887999732, -2147483648
    -2147483648, 1
    1, -738646202
    -738646202, 454567881
    454567881, -1
    -1, 1321200746
    1321200746, 2147483647
    2147483647, -2147483648
    -2147483648, 912114712
    912114712, 703322924
    703322924, -802049796
    -802049796, 1
    1, 2147483647
    2147483647, -1707626195
    -1707626195, -1
    -1, -1626087642
    -1626087642, 2147483647
    2147483647, 546871561
    546871561, -651551026
    -651551026, 0
    0, 1
    1, 1
    1, 958219829
    958219829, 1772245900

    Can help us identify values outside from our business frame

    View Slide

  13. Integration with ScalaTest
    class CalculatorPropertiesFlatSpec extends AnyFlatSpec with Checkers {
    "add" should "be commutative" in {
    check(forAll { (x: Int, y: Int) =>
    add(x, y) == add(y, x)
    })
    }
    "add" should "be associative" in {
    check(forAll { (x: Int) =>
    add(add(x, 1), 1) == add(x, 2)
    })
    }
    "0" should "be identity" in {
    check(forAll { (x: Int) =>
    add(x, 0) == x
    })
    }
    }
    https://www.scalatest.org/user_guide/writing_scalacheck_style_properties

    View Slide

  14. @yot88
    What happens when it fails
    ScalaCheck provides us the values for which the property is falsified

    View Slide

  15. Concepts
    • By default, each function is tested 10 times
    10 times if all tests succeed (differs from scalacheck-only behavior)
    less than that if a test is falsified
    • Tests are run with randomly-generated data
    • ScalaCheck has built-in generators
    We can also write our own generators

    View Slide

  16. PBT in real life

    View Slide

  17. Postal Parcel
    Open `PostalParcel` :
    • Identify properties – 5’
    What are the invariants ?
    • Write them in `PostalParcelPropertiesFlatSpec` - 15’
    • Collective debriefing – 5’
    object PostalParcel {
    val maxWeight = 20.0
    val maxDeliveryCosts = 4.99
    val minDeliveryCosts = 1.99
    def fromDouble(weight: Double): Option[PostalParcel] =
    if (weight > 0) Some(PostalParcel(weight)) else None
    def calculateDeliveryCosts(
    postalParcel: Option[PostalParcel]
    ): Option[Double] = {
    postalParcel.map(p =>
    if (p.weight > maxWeight) maxDeliveryCosts else minDeliveryCosts
    )
    }
    }
    scala C# java

    View Slide

  18. Bank withdrawal
    • Open `AccountService` :
    • Identify properties – 5’
    What should we do to check those properties ?
    object AccountService {
    def withdraw(account: Account, command: Withdraw): Either[String, Account] = {
    if (hasAlreadyBeenApplied(account, command))
    Right(account)
    else
    applyWithdraw(account, command)
    }
    private def applyWithdraw(
    account: Account,
    command: Withdraw
    ): Either[String, Account] = {
    command.amount.value match {
    case amount if exceedMaxWithdrawal(account, amount) =>
    Left(s"Amount exceeding your limit of ${account.maxWithdrawal}")
    case amount if exceedBalance(account, amount) =>
    Left(s"Insufficient balance to withdraw : $amount")
    case _ =>
    Right(
    account.copy(
    balance = account.balance - command.amount.value,
    withdraws = account.withdraws :+ command
    )
    )
    }
    }
    private def hasAlreadyBeenApplied(account: Account, command: Withdraw) =
    account.withdraws.contains(command)
    private def exceedMaxWithdrawal(
    account: Account,
    withdrawAmount: Double
    ): Boolean =
    withdrawAmount >= account.maxWithdrawal
    private def exceedBalance(account: Account, withdrawAmount: Double): Boolean =
    withdrawAmount > account.balance && !account.isOverdraftAuthorized
    }

    View Slide

  19. Bank withdrawal - Properties
    • "account balance" should "be decremented at least from the withdraw amount”
    • "account balance" should "be decremented at least from the withdraw amount when insufficient balance but overdraft authorized”
    • "withdraw" should "not be allowed when withdraw amount >= maxWithdrawal”
    • "withdraw" should "not be allowed when insufficient balance and no overdraft”
    • "withdraw" should "be idempotent"

    View Slide

  20. Bank withdrawal using properties and when (WithdrawProperties)
    "withdraw" should "not be allowed when insufficient balance and no overdraft" in {
    checkProperty(
    (account, command) => {
    withInsufficientBalance(account, command) &&
    withoutOverdraftAuthorized(account) &&
    withoutReachingMaxWithdrawal(account, command)
    },
    (_, _, debitedAccount) => {
    debitedAccount.left.value
    .startsWith("Insufficient balance to withdraw")
    }
    )
    } Need to discard a lot of generated inputs
    implicit override val generatorDrivenConfig: PropertyCheckConfiguration =
    PropertyCheckConfiguration(minSize = 10, maxDiscardedFactor = 30)
    Need to increase maxDiscardedFactor to check the properties…
    Not a valid way to check our business invariants
    implicit val accountGenerator: Arbitrary[Account] = Arbitrary {
    for {
    balance <- Arbitrary.arbitrary[Double]
    isOverdraftAuthorized <- Arbitrary.arbitrary[Boolean]
    maxWithdrawal <- Arbitrary.arbitrary[Double]
    } yield Account(balance, isOverdraftAuthorized, maxWithdrawal)
    }
    implicit val withdrawCommandGenerator: Arbitrary[Withdraw] = Arbitrary {
    for {
    clientId <- Arbitrary.arbitrary[UUID]
    amount <- posNum[Double]
    requestDate <- Arbitrary.arbitrary[LocalDate]
    } yield Withdraw(clientId, Amount.from(amount).get, requestDate)
    }
    scala

    View Slide

  21. Bank withdrawal using properties and commandGenerator + builder
    "withdraw" should "not be allowed when insufficient balance and no overdraft" in {
    checkProperty(
    (account, command) =>
    account
    .withInsufficientBalance(command)
    .withoutOverDraftAuthorized()
    .withoutReachingMaxWithdrawal(command),
    (_, _, debitedAccount) =>
    debitedAccount.left.value
    .startsWith("Insufficient balance to withdraw")
    )
    }
    private def checkProperty(
    accountConfiguration: (AccountBuilder, Withdraw) => AccountBuilder,
    property: (Account, Withdraw, Either[String, Account]) => Boolean
    ): Assertion = {
    check(forAll { (command: Withdraw) =>
    val account = accountConfiguration(AccountBuilder(), command).build()
    property(account, command, AccountService.withdraw(account, command))
    })
    }
    final case class AccountBuilder(
    balance: Double = 0,
    isOverdraftAuthorized: Boolean = false,
    maxWithdrawal: Double = 0
    ) {
    def withEnoughMoney(command: Withdraw): AccountBuilder =
    copy(balance = command.amount.value + arbitraryPositiveNumber)
    def withdrawAmountReachingMaxWithdrawal(command: Withdraw): AccountBuilder =
    copy(maxWithdrawal = command.amount.value - arbitraryPositiveNumber)

    We now instantiate accounts based on random
    commands with a Builder
    Do not need to change ScalaCheck
    configuration anymore
    Builder is maybe leaking too much of our business…
    scala C# java

    View Slide

  22. private val `withdraw examples` =
    Table(
    (
    "balance",
    "isOverdraftAuthorized",
    "maxWithdrawal",
    "withdrawAmount",
    "expectedResult"
    ),
    (10_000, false, 1200, 1199.99, Right(8800.01)),
    (0, true, 500, 50d, Right(-50)),
    (1, false, 1200, 1199.99, Left("Insufficient balance to withdraw")),
    (0, false, 500, 50d, Left("Insufficient balance to withdraw")),
    (10_000, true, 1200, 1200.0001, Left("Amount exceeding your limit of")),
    (0, false, 500, 500d, Left("Amount exceeding your limit of"))
    )
    Bank withdrawal – how we would test it in real life ?
    Use ParameterizedTests -> TableDrivenPropertyChecks
    Describe test cases
    Check implementation based on them
    We can use Properties to help us identify edge cases
    and add new examples in our table
    A better approach for this kind of use cases :
    • Example-based with Table to allow easy addition of new tests cases
    • Use PBT as a cases finders
    • Use PBT to check idem potence
    "withdraw" should "pass examples" in {
    forAll(`withdraw examples`) {
    (
    balance,
    isOverdraftAuthorized,
    maxWithdrawal,
    withdrawAmount,
    expectedResult
    ) =>
    val command = toWithdraw(withdrawAmount)
    val debitedAccount =
    AccountService.withdraw(
    Account(balance, isOverdraftAuthorized, maxWithdrawal),
    command
    )
    expectedResult match {
    case Left(expectedErrorMessage) =>
    debitedAccount.left.value should startWith(expectedErrorMessage)
    case Right(expectedBalance) =>
    debitedAccount.value.balance should be(expectedBalance)
    debitedAccount.value.withdraws should contain(command)
    debitedAccount.value.withdraws should have length 1
    }
    }
    }
    scala C# java

    View Slide

  23. Rental Calculator PBT on legacy code
    • Open `RentalCalculator` :
    Imagine this code is running in production and business is happy with it
    Imagine you need to adapt it, but you need to be sure that you don’t introduce any regression
    class RentalCalculator(val rentals: List[Rental]) {
    private var _amount = .0
    private var _calculated = false
    def amount(): Double = _amount
    def calculated(): Boolean = _calculated
    def calculateRental(): Either[String, String] = {
    if (rentals.isEmpty)
    Left("No rentals !!!")
    else {
    val result = new StringBuilder
    for (rental <- rentals) {
    if (!_calculated) this._amount += rental.amount
    result.append(formatLine(rental, _amount))
    }
    result.append(f"Total amount | ${this._amount}%.2f")
    _calculated = true
    Right(result.toString)
    }
    }
    private def formatLine(rental: Rental, amount: Double) =
    f"${rental.date} : ${rental.label} | ${rental.amount}%.2f${lineSeparator()}"
    }
    How PBT can help us ?

    View Slide

  24. Rental Calculator PBT on legacy code
    • Duplicate the production code in a new file – NewRentalCalculator for example
    • Write a property checking that – 5’
    f(x) == new_f(x)
    Ensure that your test fails if you change the new implementation
    • Now you can refactor the new implementation code – 15’
    Identify the code smells
    Refactor the code
    Run the property at each changes
    • You can now plug the caller on the new implementation
    Delete the legacy implementation
    Remove the property
    Seeing a test failing is as important
    as seeing it passing
    Can use this technique for any rewriting / optimizing refactoring
    scala C# java

    View Slide

  25. Apply Property-based Testing
    • Check idempotence
    f(f(x)) == f(x)
    ex : UpperCase, Create / Delete
    • Check roundtripping
    from(to(x)) == x
    ex : Serialization, PUT / GET, Reverse, Negate
    • Check invariants
    invariant(f(x)) == invariant(x)
    ex : Reverse, Map
    • Check commutativity
    f(x, y) == f(y, x)
    ex : Addition, Min, Max
    • Check rewriting
    f(x) == new_f(x)
    when rewriting or optimizing implementations
    • “Upgrade” parameterized tests
    Replace hardcoded values or discover new test cases (edge cases)
    • …
    We can use it to test general properties of our functions

    View Slide

  26. Anti-patterns
    • Re-run on failures
    It throws away the specific failure (new generated values)
    Instead : investigate failure and add a new example-based test
    • Re-implement production code
    Leak all your production code in your properties
    • Perform ad-hoc filtering
    Do not filter input values by ourselves
    Instead : use the framework support

    View Slide

  27. QuickCheck in your favorite language
    source : https://hypothesis.works/articles/quickcheck-in-every-language/

    View Slide

  28. Conclusion
    Think about your last development, how this approach could have helped you ?

    View Slide

  29. Resources
    ScalaCheck documentation : https://github.com/typelevel/scalacheck/blob/main/doc/UserGuide.md
    Scala exercises on PBT : https://www.scala-exercises.org/scalacheck/generators
    "Functions for nothing, and your tests for free" Property-based testing and F# - George Pollard
    Property based testing - step by step : https://www.leadingagile.com/2018/04/step-by-step-toward-property-based-testing/

    View Slide