Yoan
January 10, 2022
780

# A journey to Property-Based Testing

Explore what is Property-Based Testing and how you can use this approach in your daily dev life.
Support of a workshop explained here : https://yoan-thirion.gitbook.io/knowledge-base/software-craftsmanship/code-katas/improve-your-software-quality-with-property-based-testing/a-journey-to-property-based-testing

January 10, 2022

## Transcript

1. @yot88
A Journey
to Property-based Testing

2. Calculator
In mob :
• Open the repository
• Write tests to check the behavior of the Calculator class
https://github.com/ythirion/property-based-testing-kata
object Calculator {
def add(x: Int, y: Int): Int = x + y
}
Source code is available in scala (scalatest / scalacheck) / C# (xunit / FsCheck / Language-Ext / FluentAssertions) / java (Junit5 / quick-check / vavr / AssertJ)

3. Calculator - Debriefing
• Which tests did you write ?
• What edge cases did you find ?
• What are the limits of this example-based testing approach ?
Given (1, 3)
Then I expect 4
Given (-1, 3)
Then I expect 2
Given (0, 99)
Then I expect 99
Could we test it differently to cover
more inputs ?
it should "return their correct sum when I add 2 random numbers" in {
Range(0, 100)
.foreach(_ => {
val x = random.nextInt()
val y = random.nextInt()
assert(add(x, y) == x + y)
})
}
Implementation leak in tests
What does correct mean in this context ?

4. Example-based testing
Do we cover the full scope of
possible inputs ?
Does the developed feature is compliant with what is expected ?
When we write Unit tests :
• we often focus on examples
• examples are good to understand requirements
Given (x, y, ...)
When I [call the subject under test] with (x, y, ...)
Then I expect this (output)
Limitation : we only focus on the input scope we identified

5. Property-based testing – at our rescue ?
PBT

6. What is Property-based testing
• A property is the combination of an invariant with an input values generator.
For each generated value, the invariant is treated as a predicate and checked whether it yields true or false for that value.
• As soon as there is one value which yields false, the property is said to be falsified, and checking is aborted.
• If a property cannot be invalidated after a specific amount of sample data, the property is assumed to be
satisfied.

7. In other words, a property is something like :
for all (x, y, ...)
such as precondition (x, y, ...) holds
property (x, y, ...) is satisfied
run it multiple times generate random inputs
(based on specified generators)
check input pre-conditions
Failure: not an error -> invalid entry
run the predicate
Failure: a counter-example
will try to shrink the entry
• Describe the input
• Describe the properties of the output
• Have the computer try lots of random examples
• Check if it fails

8. @yot88
Which properties do we know on Addition ?
Compare addition with other Math operations
• What about parameter order ? (compared to subtract)
The parameter order does not matter
• “add 1” twice is the same as doing “add 2” (compared to multiply)
We are confident the implementation is correct if we test these properties
It applies to all inputs

9. Let’s write some tests
it should "when I add 2 numbers the result should not depend on parameter
order" in {
Range(0, times)
.map(_ => random.nextInt())
.foreach(x => {
val y = random.nextInt()
})
}
it should "when I add 1 twice it's the same as adding 2 once" in {
Range(0, times)
.map(_ => random.nextInt())
}
it should "when I add 0 to a random number is the same than doing
nothing on this number" in {
Range(0, times)
.map(_ => random.nextInt())
.foreach(x => assert(add(x, 0) == x))
}
Commutativity
Associativity
Identity

10. ScalaCheck
“ScalaCheck is a tool for testing Scala and Java programs, based on property specifications and
automatic test data generation. The basic idea is that you define a property that specifies the
behaviour of a method or some unit of code, and ScalaCheck checks that the property holds. All test
data are generated automatically in a random fashion, so you don’t have to worry about any missed
cases.” - ScalaCheck : The Definitive Guide
libraryDependencies += "org.scalatest" %% "scalatest" % "3.2.9" % Test
libraryDependencies += "org.scalactic" %% "scalactic" % "3.2.9" % Test
libraryDependencies += "org.scalatestplus" %% "scalacheck-1-15" % "3.2.9.0" % "test"
https://github.com/typelevel/scalacheck/blob/main/doc/UserGuide.md
A way to generate (and
shrink)
random test data

property("commutativity") = forAll { (x: Int, y: Int) =>
}
property("associativity") = forAll { (x: Int) =>
}
property("identity") = forAll { (x: Int) =>
}
}

12. How to read it ?
property("commutativity") = forAll { (x: Int, y: Int) =>
}
For all possible x, y run the block of code that follows
-1, 1887999732
1887999732, -2147483648
-2147483648, 1
1, -738646202
-738646202, 454567881
454567881, -1
-1, 1321200746
1321200746, 2147483647
2147483647, -2147483648
-2147483648, 912114712
912114712, 703322924
703322924, -802049796
-802049796, 1
1, 2147483647
2147483647, -1707626195
-1707626195, -1
-1, -1626087642
-1626087642, 2147483647
2147483647, 546871561
546871561, -651551026
-651551026, 0
0, 1
1, 1
1, 958219829
958219829, 1772245900

Can help us identify values outside from our business frame

13. Integration with ScalaTest
class CalculatorPropertiesFlatSpec extends AnyFlatSpec with Checkers {
"add" should "be commutative" in {
check(forAll { (x: Int, y: Int) =>
})
}
"add" should "be associative" in {
check(forAll { (x: Int) =>
})
}
"0" should "be identity" in {
check(forAll { (x: Int) =>
})
}
}
https://www.scalatest.org/user_guide/writing_scalacheck_style_properties

14. @yot88
What happens when it fails
ScalaCheck provides us the values for which the property is falsified

15. Concepts
• By default, each function is tested 10 times
10 times if all tests succeed (differs from scalacheck-only behavior)
less than that if a test is falsified
• Tests are run with randomly-generated data
• ScalaCheck has built-in generators
We can also write our own generators

16. PBT in real life

17. Postal Parcel
Open `PostalParcel` :
• Identify properties – 5’
What are the invariants ?
• Write them in `PostalParcelPropertiesFlatSpec` - 15’
• Collective debriefing – 5’
object PostalParcel {
val maxWeight = 20.0
val maxDeliveryCosts = 4.99
val minDeliveryCosts = 1.99
def fromDouble(weight: Double): Option[PostalParcel] =
if (weight > 0) Some(PostalParcel(weight)) else None
def calculateDeliveryCosts(
postalParcel: Option[PostalParcel]
): Option[Double] = {
postalParcel.map(p =>
if (p.weight > maxWeight) maxDeliveryCosts else minDeliveryCosts
)
}
}
scala C# java

18. Bank withdrawal
• Open `AccountService` :
• Identify properties – 5’
What should we do to check those properties ?
object AccountService {
def withdraw(account: Account, command: Withdraw): Either[String, Account] = {
Right(account)
else
applyWithdraw(account, command)
}
private def applyWithdraw(
account: Account,
command: Withdraw
): Either[String, Account] = {
command.amount.value match {
case amount if exceedMaxWithdrawal(account, amount) =>
Left(s"Amount exceeding your limit of \${account.maxWithdrawal}")
case amount if exceedBalance(account, amount) =>
Left(s"Insufficient balance to withdraw : \$amount")
case _ =>
Right(
account.copy(
balance = account.balance - command.amount.value,
withdraws = account.withdraws :+ command
)
)
}
}
private def hasAlreadyBeenApplied(account: Account, command: Withdraw) =
account.withdraws.contains(command)
private def exceedMaxWithdrawal(
account: Account,
withdrawAmount: Double
): Boolean =
withdrawAmount >= account.maxWithdrawal
private def exceedBalance(account: Account, withdrawAmount: Double): Boolean =
withdrawAmount > account.balance && !account.isOverdraftAuthorized
}

19. Bank withdrawal - Properties
• "account balance" should "be decremented at least from the withdraw amount”
• "account balance" should "be decremented at least from the withdraw amount when insufficient balance but overdraft authorized”
• "withdraw" should "not be allowed when withdraw amount >= maxWithdrawal”
• "withdraw" should "not be allowed when insufficient balance and no overdraft”
• "withdraw" should "be idempotent"

20. Bank withdrawal using properties and when (WithdrawProperties)
"withdraw" should "not be allowed when insufficient balance and no overdraft" in {
checkProperty(
(account, command) => {
withInsufficientBalance(account, command) &&
withoutOverdraftAuthorized(account) &&
withoutReachingMaxWithdrawal(account, command)
},
(_, _, debitedAccount) => {
debitedAccount.left.value
.startsWith("Insufficient balance to withdraw")
}
)
} Need to discard a lot of generated inputs
implicit override val generatorDrivenConfig: PropertyCheckConfiguration =
PropertyCheckConfiguration(minSize = 10, maxDiscardedFactor = 30)
Need to increase maxDiscardedFactor to check the properties…
Not a valid way to check our business invariants
implicit val accountGenerator: Arbitrary[Account] = Arbitrary {
for {
balance <- Arbitrary.arbitrary[Double]
isOverdraftAuthorized <- Arbitrary.arbitrary[Boolean]
maxWithdrawal <- Arbitrary.arbitrary[Double]
} yield Account(balance, isOverdraftAuthorized, maxWithdrawal)
}
implicit val withdrawCommandGenerator: Arbitrary[Withdraw] = Arbitrary {
for {
clientId <- Arbitrary.arbitrary[UUID]
amount <- posNum[Double]
requestDate <- Arbitrary.arbitrary[LocalDate]
} yield Withdraw(clientId, Amount.from(amount).get, requestDate)
}
scala

21. Bank withdrawal using properties and commandGenerator + builder
"withdraw" should "not be allowed when insufficient balance and no overdraft" in {
checkProperty(
(account, command) =>
account
.withInsufficientBalance(command)
.withoutOverDraftAuthorized()
.withoutReachingMaxWithdrawal(command),
(_, _, debitedAccount) =>
debitedAccount.left.value
.startsWith("Insufficient balance to withdraw")
)
}
private def checkProperty(
accountConfiguration: (AccountBuilder, Withdraw) => AccountBuilder,
property: (Account, Withdraw, Either[String, Account]) => Boolean
): Assertion = {
check(forAll { (command: Withdraw) =>
val account = accountConfiguration(AccountBuilder(), command).build()
property(account, command, AccountService.withdraw(account, command))
})
}
final case class AccountBuilder(
balance: Double = 0,
isOverdraftAuthorized: Boolean = false,
maxWithdrawal: Double = 0
) {
def withEnoughMoney(command: Withdraw): AccountBuilder =
copy(balance = command.amount.value + arbitraryPositiveNumber)
def withdrawAmountReachingMaxWithdrawal(command: Withdraw): AccountBuilder =
copy(maxWithdrawal = command.amount.value - arbitraryPositiveNumber)

We now instantiate accounts based on random
commands with a Builder
Do not need to change ScalaCheck
configuration anymore
Builder is maybe leaking too much of our business…
scala C# java

22. private val `withdraw examples` =
Table(
(
"balance",
"isOverdraftAuthorized",
"maxWithdrawal",
"withdrawAmount",
"expectedResult"
),
(10_000, false, 1200, 1199.99, Right(8800.01)),
(0, true, 500, 50d, Right(-50)),
(1, false, 1200, 1199.99, Left("Insufficient balance to withdraw")),
(0, false, 500, 50d, Left("Insufficient balance to withdraw")),
(10_000, true, 1200, 1200.0001, Left("Amount exceeding your limit of")),
(0, false, 500, 500d, Left("Amount exceeding your limit of"))
)
Bank withdrawal – how we would test it in real life ?
Use ParameterizedTests -> TableDrivenPropertyChecks
Describe test cases
Check implementation based on them
We can use Properties to help us identify edge cases
and add new examples in our table
A better approach for this kind of use cases :
• Example-based with Table to allow easy addition of new tests cases
• Use PBT as a cases finders
• Use PBT to check idem potence
"withdraw" should "pass examples" in {
forAll(`withdraw examples`) {
(
balance,
isOverdraftAuthorized,
maxWithdrawal,
withdrawAmount,
expectedResult
) =>
val command = toWithdraw(withdrawAmount)
val debitedAccount =
AccountService.withdraw(
Account(balance, isOverdraftAuthorized, maxWithdrawal),
command
)
expectedResult match {
case Left(expectedErrorMessage) =>
debitedAccount.left.value should startWith(expectedErrorMessage)
case Right(expectedBalance) =>
debitedAccount.value.balance should be(expectedBalance)
debitedAccount.value.withdraws should contain(command)
debitedAccount.value.withdraws should have length 1
}
}
}
scala C# java

23. Rental Calculator PBT on legacy code
• Open `RentalCalculator` :
Imagine this code is running in production and business is happy with it
Imagine you need to adapt it, but you need to be sure that you don’t introduce any regression
class RentalCalculator(val rentals: List[Rental]) {
private var _amount = .0
private var _calculated = false
def amount(): Double = _amount
def calculated(): Boolean = _calculated
def calculateRental(): Either[String, String] = {
if (rentals.isEmpty)
Left("No rentals !!!")
else {
val result = new StringBuilder
for (rental <- rentals) {
if (!_calculated) this._amount += rental.amount
result.append(formatLine(rental, _amount))
}
result.append(f"Total amount | \${this._amount}%.2f")
_calculated = true
Right(result.toString)
}
}
private def formatLine(rental: Rental, amount: Double) =
f"\${rental.date} : \${rental.label} | \${rental.amount}%.2f\${lineSeparator()}"
}
How PBT can help us ?

24. Rental Calculator PBT on legacy code
• Duplicate the production code in a new file – NewRentalCalculator for example
• Write a property checking that – 5’
f(x) == new_f(x)
Ensure that your test fails if you change the new implementation
• Now you can refactor the new implementation code – 15’
Identify the code smells
Refactor the code
Run the property at each changes
• You can now plug the caller on the new implementation
Delete the legacy implementation
Remove the property
Seeing a test failing is as important
as seeing it passing
Can use this technique for any rewriting / optimizing refactoring
scala C# java

25. Apply Property-based Testing
• Check idempotence
f(f(x)) == f(x)
ex : UpperCase, Create / Delete
• Check roundtripping
from(to(x)) == x
ex : Serialization, PUT / GET, Reverse, Negate
• Check invariants
invariant(f(x)) == invariant(x)
ex : Reverse, Map
• Check commutativity
f(x, y) == f(y, x)
• Check rewriting
f(x) == new_f(x)
when rewriting or optimizing implementations
Replace hardcoded values or discover new test cases (edge cases)
• …
We can use it to test general properties of our functions

26. Anti-patterns
• Re-run on failures
It throws away the specific failure (new generated values)
• Re-implement production code
Do not filter input values by ourselves
Instead : use the framework support

27. QuickCheck in your favorite language
source : https://hypothesis.works/articles/quickcheck-in-every-language/

28. Conclusion
Think about your last development, how this approach could have helped you ?

29. Resources
ScalaCheck documentation : https://github.com/typelevel/scalacheck/blob/main/doc/UserGuide.md
Scala exercises on PBT : https://www.scala-exercises.org/scalacheck/generators