Writing bulletproof code with property-based testing

Slides of my "Writing bulletproof code with property-based testing" tech talk at Droidcon Lisbon & Droidcon Berlin
-------------------------------------------------------------------------------
Unit tests are an important part of the development process. Among other things, they help us verify the correctness of our code.

Thus, when writing tests, we usually pick one or two relevant examples, let the tests run green and call it a day. This is called example-based testing.

But, is that enough? Have you missed any significant case? Have you considered all edge-cases? What if any of those non-tested examples would make the app crash?

Fear not, we can fill that gap with property-based tests. Let me show to you how, step by step.

In this tech-talk you will learn:
- some easy tips to improve your unit tests
- why example-based unit tests are not always enough
- how property-based testing can make your tests more robust
- property-based testing in very detail:
* what it is
* its pros and cons
* which tools/frameworks are available for implementing them
* how to make the most out of them

And all that by using as examples some features you would find in most Android apps, like "create account" or "text editing".
----------------------------------------------------------------
Repo with code examples:
https://github.com/sergio-sastre/Multiplying_the_quality_of_unit_tests

Transcript

1. PROPERTY - BASED
   BY SERGIO SASTRE
   WRITING BULLETPROOF CODE WITH
   TESTING
   

   View Slide

2. Example-based testing
   of your software
   writing examples of it
   tests specify the behaviour
   where
   by
   @Gio_Sastre
   

   View Slide

3. Example-based testing
   @Gio_Sastre
   @Test

   fun is_string_palindrome() {
   
   
   val inputString = “1221”

   
   
   val revertedInput = inputString.reversed(
   )
   
   assertTrue(inputString == revertedInput) 

   }
   

   View Slide

4. Example-based testing
   @Gio_Sastre
   @Test

   fun is_string_palindrome() {
   
   
   val inputString = “1221”

   
   
   val revertedInput = inputString.reversed(
   )
   
   assertTrue(inputString == revertedInput) 

   }
   

   View Slide

5. Example-based testing
   @Gio_Sastre
   @Test

   fun is_string_palindrome() {
   
   
   val inputString = “12a21”

   
   
   val revertedInput = inputString.reversed(
   )
   
   assertTrue(inputString == revertedInput) 

   }
   

   View Slide

6. Example-based testing
   @Gio_Sastre
   @Test

   fun is_string_palindrome() {
   
   
   val inputString = “”

   
   
   val revertedInput = inputString.reversed(
   )
   
   assertTrue(inputString == revertedInput) 

   }
   

   View Slide

7. Text editor
   @Gio_Sastre
   Strong password validation
   

   View Slide

8. @Gio_Sastre
   Text editor
   Strong password validation
   

   View Slide

9. @Gio_Sastre
   Strong password validation Text editor
   

   View Slide

10. @Gio_Sastre
    fun clickSignUpButton() {
    
    
    mutablePasswordError.value =
    
    
    passwordValidator.validate(passwordInput)

    }
    ViewModel
    

    View Slide

11. @Gio_Sastre
    fun clickSignUpButton() {
    
    
    mutablePasswordError.value =
    
    
    passwordValidator.validate(passwordInput)

    }
    ViewModel
    

    View Slide

12. @Gio_Sastre
    The password contains
    at least 1 digit
    
    
    Show “must contain digits" otherwise
    1
    at least 1 latin char in upper case.
    
    
    Show “must contain upper case letters” otherwise
    2
    at least 6 chars.
    
    
    Show “must contain at least 6 chars” otherwise
    3
    no blanks.
    
    
    Show “must not contain blanks” otherwise
    4
    at least 1 latin char in lower case.
    
    
    Show “must contain lower case letters” otherwise
    5
    @Test

    fun passwordValidatorDigitsTest() {
    assertThat
    (
    
    passwordValidator.validate(“ABCDEF”)
    ).contains(“must contain digits”)

    }
    

    View Slide

13. @Gio_Sastre
    The password contains
    at least 1 digit
    
    
    Show “must contain digits" otherwise
    1
    at least 1 latin char in upper case.
    
    
    Show “must contain upper case letters” otherwise
    2
    at least 6 chars.
    
    
    Show “must contain at least 6 chars” otherwise
    3
    no blanks.
    
    
    Show “must not contain blanks” otherwise
    4
    at least 1 latin char in lower case.
    
    
    Show “must contain lower case letters” otherwise
    5
    @Test

    fun passwordValidatorDigitsTest() {
    assertThat
    (
    
    passwordValidator.validate(“ABCDEF”)
    ).contains(“must contain digits”)

    assertThat
    (
    
    passwordValidator.validate(“123456”)
    ).contains(“must contain upper case letters”)
    }
    @Test

    fun passwordValidatorUpperCaseTest() {
    }
    

    View Slide

14. @Gio_Sastre
    The password contains
    at least 1 digit
    
    
    Show “must contain digits" otherwise
    1
    at least 1 latin char in upper case.
    
    
    Show “must contain upper case letters” otherwise
    2
    at least 6 chars.
    
    
    Show “must contain at least 6 chars” otherwise
    3
    no blanks.
    
    
    Show “must not contain blanks” otherwise
    4
    at least 1 latin char in lower case.
    
    
    Show “must contain lower case letters” otherwise
    5
    @ParameterizedTest(

    name = “when password \”{0}\”, error contains \”{1}\””
    )
    
    @CsvSource
    (
    
    “ABCDEF, must contain digits”,
    “123456, must contain upper case letters”
    ,
    
    “12345, must contain at least 6 chars”,

    “12 3 456, must not contain blanks”
    ,
    
    “HELLO, must contain lower case letters”,
    )

    fun passwordValidatorTest
    (
    
    password: String?, expectedError: String
    ?
    
    ) {
    assertThat
    (
    
    passwordValidator.validate(password)
    ).contains(expectedError)
    }
    

    View Slide

15. @Gio_Sastre
    The password contains
    at least 1 digit
    
    
    Show “must contain digits" otherwise
    1
    at least 1 latin char in upper case.
    
    
    Show “must contain upper case letters” otherwise
    2
    at least 6 chars.
    
    
    Show “must contain at least 6 chars” otherwise
    3
    no blanks.
    
    
    Show “must not contain blanks” otherwise
    4
    at least 1 latin char in lower case.
    
    
    Show “must contain lower case letters” otherwise
    5
    @ParameterizedTest(

    name = “when password \”{0}\”, error contains \”{1}\””
    )
    
    @CsvSource
    (
    
    “ABCDEF, must contain digits”,
    “123456, must contain upper case letters”
    ,
    
    “12345, must contain at least 6 chars”,

    “12 3 456, must not contain blanks”
    ,
    
    “HELLO, must contain lower case letters”,
    )

    fun passwordValidatorTest
    (
    
    password: String?, expectedError: String
    ?
    
    ) {
    assertThat
    (
    
    passwordValidator.validate(password)
    ).contains(expectedError)
    }
    

    View Slide

16. @Gio_Sastre
    The password contains
    at least 1 digit
    
    
    Show “must contain digits" otherwise
    1
    at least 1 latin char in upper case.
    
    
    Show “must contain upper case letters” otherwise
    2
    at least 6 chars.
    
    
    Show “must contain at least 6 chars” otherwise
    3
    no blanks.
    
    
    Show “must not contain blanks” otherwise
    4
    at least 1 latin char in lower case.
    
    
    Show “must contain lower case letters” otherwise
    5
    @ParameterizedTest(

    name = “when password \”{0}\”, error contains \”{1}\””
    )
    
    @CsvSource
    (
    
    “ABCDEF, must contain digits”,
    “123456, must contain upper case letters”
    ,
    
    “12345, must contain at least 6 chars”,

    “12 3 456, must not contain blanks”
    ,
    
    “HELLO, must contain lower case letters”,
    )

    fun passwordValidatorTest
    (
    
    password: String?, expectedError: String
    ?
    
    ) {
    assertThat
    (
    
    passwordValidator.validate(password)
    ).contains(expectedError)
    }
    

    View Slide

17. @Gio_Sastre
    The password contains
    at least 1 digit
    
    
    Show “must contain digits" otherwise
    1
    at least 1 latin char in upper case.
    
    
    Show “must contain upper case letters” otherwise
    2
    at least 6 chars.
    
    
    Show “must contain at least 6 chars” otherwise
    3
    no blanks.
    
    
    Show “must not contain blanks” otherwise
    4
    at least 1 latin char in lower case.
    
    
    Show “must contain lower case letters” otherwise
    5
    @ParameterizedTest(

    name = “when password \”{0}\”, error contains \”{1}\””
    )
    
    @CsvSource
    (
    
    “ABCDEF, must contain digits”,
    “123456, must contain upper case letters”
    ,
    
    “12345, must contain at least 6 chars”,

    “12 3 456, must not contain blanks”
    ,
    
    “HELLO, must contain lower case letters”,
    )

    fun passwordValidatorTest
    (
    
    password: String?, expectedError: String
    ?
    
    ) {
    assertThat
    (
    
    passwordValidator.validate(password)
    ).contains(expectedError)
    }
    

    View Slide

18. Are
    enough?
    Example-based tests
    @Gio_Sastre
    

    View Slide

19. class PasswordValidator(…) {
    
    
    
    
    override fun isValid(password: String?): Boolean
    {
    
    …
    
    // validate contains upper case letter
    s
    
    if (!password?.contains(“[A-Z]”.toRegex()))
    {
    
    …
    
    
    }
    
    …
    
    }
    
    }
    @Gio_Sastre
    

    View Slide

20. class PasswordValidator(…) {
    
    
    
    
    override fun isValid(password: String?): Boolean
    {
    
    …
    
    // validate contains upper case letter
    s
    
    if (!password?.contains(“[a-z]”.toRegex()))
    {
    
    …
    
    
    }
    
    …
    
    }
    
    }
    @Gio_Sastre
    

    View Slide

21. @Gio_Sastre
    

    View Slide

22. 🧐
    Upper case: “[a-z]”.toRegex()
    @Gio_Sastre
    

    View Slide

23. 🧐
    @Gio_Sastre
    Upper case: “[a-z]”.toRegex()
    show “must contain upper case letters
    ”
    
    if “123456” has no
    

    View Slide

24. 🧐
    @Gio_Sastre
    Lower case: “[a-z]”.toRegex()
    show “must contain lower case letters
    ”
    
    🧐
    if “123456” has no
    

    View Slide

25. 🧐
    @Gio_Sastre
    Upper case: “[a-z]”.toRegex()
    if “123456” has no
    show “must contain upper case letters
    ”
    
    🤦
    

    View Slide

26. @Gio_Sastre
    Upper case: “[a-z]”.toRegex()
    if “123456” has no
    show “must contain upper case letters
    ”
    
    🤦
    🧐
    

    View Slide

27. @Gio_Sastre
    🙃
    Upper case: “[a-z]”.toRegex()
    if “abcdef” has no
    show “must contain upper case letters
    ”
    
    

    View Slide

28. 1
    example-based tests
    Problems with
    2
    Finding good examples
    @Gio_Sastre
    Covering all relevant cases
    

    View Slide

29. Can
    those issues?
    we mitigate
    @Gio_Sastre
    

    View Slide

30. Example-based testing
    of your software
    writing examples of it
    tests specify the behaviour
    where
    by
    @Gio_Sastre
    

    View Slide

31. Property-based testing
    generating thousands of
    tests specify the behaviour
    where
    by
    @Gio_Sastre
    semi-random examples of it
    of your software
    

    View Slide

32. Example-based testing
    @Gio_Sastre
    @Test

    fun is_string_palindrome() {
    
    
    val inputString = “1221”

    
    
    val revertedInput = inputString.reversed(
    )
    
    assertTrue(inputString == revertedInput) 

    }
    

    View Slide

33. fun is_string_palindrome() {
    
    
    val inputString = SemiRandomString(seed)

    
    
    val revertedInput = inputString.reversed(
    )
    
    assertTrue(inputString == revertedInput) 

    }
    @Gio_Sastre
    Property-based testing
    @Test(tries = 1_000)
    

    View Slide

34. fun is_string_palindrome() {
    
    
    val inputString = SemiRandomString(seed)

    
    
    val revertedInput = inputString.reversed(
    )
    
    assertTrue(inputString == revertedInput) 

    }
    @Gio_Sastre
    Property-based testing
    @Test(tries = 1_000)
    

    View Slide

35. @Gio_Sastre
    Kotlin support
    Nullables for non-top-level types
    
    
    requires @WithNull
    Missing features & configurability
    Previous failing seed not saved, etc.
    Kotlin multiplatform support
    Requires IntelliJ plugin
    to run from IDE
    Smooth integration
    Full featured & Configurable
    via Gradle
    Stateful testing, statistics, etc.
    Kotlin support
    Kotlin Native too
    Kotlin multiplatform support
    Jqwik otest
    

    View Slide

36. @Gio_Sastre
    Smooth integration
    Full featured & Configurable
    via Gradle
    Kotlin support Kotlin support
    Stateful testing, statistics, etc.
    Nullables for non-top-level types
    
    
    requires @WithNull
    Missing features & configurability
    Previous failing seed not saved, etc.
    Kotlin multiplatform support
    Kotlin Native too
    Kotlin multiplatform support
    Requires IntelliJ plugin
    to run from IDE
    Jqwik otest
    

    View Slide

37. @Gio_Sastre
    Smooth integration
    Full featured & Configurable
    via Gradle
    Kotlin support
    Stateful testing, statistics, etc.
    Nullables for non-top-level types
    
    
    requires @WithNull
    Missing features & configurability
    Previous failing seed not saved, etc.
    Kotlin multiplatform support
    Requires IntelliJ plugin
    to run from IDE
    Kotlin support
    Kotlin Native too
    Kotlin multiplatform support
    Jqwik otest
    

    View Slide

38. @Gio_Sastre
    Smooth integration
    Full featured & Configurable
    via Gradle
    Kotlin support
    Stateful testing, statistics, etc.
    Nullables for non-top-level types
    
    
    requires @WithNull
    Missing features & configurability
    Previous failing seed not saved, etc.
    Kotlin multiplatform support
    Kotlin support
    Kotlin Native too
    Kotlin multiplatform support
    Requires IntelliJ plugin
    to run from IDE
    Jqwik otest
    

    View Slide

39. @Gio_Sastre
    Kotlin support
    Nullables for non-top-level types
    
    
    requires @WithNull
    Missing features & configurability
    Previous failing seed not saved, etc.
    String?
    Smooth integration
    Full featured & Configurable
    via Gradle
    Stateful testing, statistics, etc.
    Kotlin support
    Kotlin Native too
    Kotlin multiplatform support
    Requires IntelliJ plugin
    to run from IDE
    Jqwik otest
    

    View Slide

40. @Gio_Sastre
    Kotlin support
    Nullables for non-top-level types
    
    
    requires @WithNull
    Missing features & configurability
    Previous failing seed not saved, etc.
    List<@WithNull String>
    Smooth integration
    Full featured & Configurable
    via Gradle
    Stateful testing, statistics, etc.
    Kotlin support
    Kotlin Native too
    Kotlin multiplatform support
    Requires IntelliJ plugin
    to run from IDE
    Jqwik otest
    

    View Slide

41. @Gio_Sastre
    Kotlin support
    Nullables for non-top-level types
    
    
    requires @WithNull
    Missing features & configurability
    Previous failing seed not saved, etc.
    Kotlin multiplatform support
    Smooth integration
    Full featured & Configurable
    via Gradle
    Stateful testing, statistics, etc.
    Kotlin support
    Kotlin Native too
    Kotlin multiplatform support
    Requires IntelliJ plugin
    to run from IDE
    Jqwik otest
    

    View Slide

42. How
    @Gio_Sastre
    to write
    property-based tests?
    

    View Slide

43. @Gio_Sastre
    Strong password validation
    upper case
    

    View Slide

44. @Gio_Sastre
    1 Define the property
    If password without upper case
    
    
    Show “must contain upper case letters”
    3 Test the property
    Default is 1.000 times
    2 Create a generator
    Password without upper case chars
    
    
    String without upper case chars
    

    View Slide

45. @Gio_Sastre
    1 Define the property
    If password without upper case
    
    
    Show “must contain upper case letters”
    fun noUpperCase() =
    @Provide
    Arbitraries.strings().ascii(
    )
    
    .filter { it.matches(“[^A-Z]”.toRegex()) }
    3 Test the property
    Default is 1.000 times
    2 Create a generator
    Password without upper case chars
    
    
    String without upper case chars
    

    View Slide

46. @Gio_Sastre
    1 Define the property
    If password without upper case
    
    
    Show “must contain upper case letters”
    2 Create a generator
    Password without upper case chars
    
    
    String without upper case chars
    fun noUpperCase() =
    @Provide
    Arbitraries.strings().ascii(
    )
    
    .filter { it.matches(“[^A-Z]”.toRegex()) }
    3 Test the property
    Default is 1.000 times
    fun testPasswordValidator
    (
    
    @ForAll(“noUpperCase”) password: String
    ?
    
    ) {
    @Property
    val error = passwordValidator.validate(password
    )
    
    assertThat(error
    )
    
    .contains(“must contain upper case letters”)
    }
    

    View Slide

47. @Gio_Sastre
    1 Define the property
    If password without upper case
    
    
    Show “must contain upper case letters”
    3 Test the property
    Default is 1.000 times
    fun testPasswordValidator
    (
    
    @ForAll(“noUpperCase”) password: String
    ?
    
    ) {
    @Property(tries = 100)
    val error = passwordValidator.validate(password
    )
    
    assertThat(error
    )
    
    .contains(“must contain upper case letters”)
    }
    2 Create a generator
    Password without upper case chars
    
    
    String without upper case chars
    fun noUpperCase() =
    @Provide
    Arbitraries.strings().ascii(
    )
    
    .filter { it.matches(“[^A-Z]”.toRegex()) }
    

    View Slide

48. @Gio_Sastre
    1 Define the property
    If password without upper case
    
    
    Show “must contain upper case letters”
    3 Test the property
    Default is 1.000 times
    fun testPasswordValidator
    (
    
    @ForAll(“noUpperCase”) password: String
    ?
    
    ) {
    @Property(tries = 100)
    val error = passwordValidator.validate(password
    )
    
    assertThat(error
    )
    
    .contains(“must contain upper case letters”)
    }
    @Label(“if password without upper case, error ” +
    “shows ‘must contain upper case letters’”)
    2 Create a generator
    Password without upper case chars
    
    
    String without upper case chars
    fun noUpperCase() =
    @Provide
    Arbitraries.strings().ascii(
    )
    
    .filter { it.matches(“[^A-Z]”.toRegex()) }
    

    View Slide

49. @Gio_Sastre
    1 Define the property
    If password without upper case
    
    
    Show “must contain upper case letters”
    3 Test the property
    Default is 1.000 times
    2 Create a generator
    Password without upper case chars
    
    
    String without upper case chars fun testPasswordValidator
    (
    
    @ForAll(“noUpperCase”) password: String
    ?
    
    ) {
    @Property(tries = 100)
    val error = passwordValidator.validate(password
    )
    
    assertThat(error
    )
    
    .contains(“must contain upper case letters”)
    }
    @Label(“if password without upper case, error ” +
    “shows ‘must contain upper case letters’”)
    fun noUpperCase() =
    @Provide
    Arbitraries.strings().ascii(
    )
    
    .filter { it.matches(“[^A-Z]”.toRegex()) }
    

    View Slide

50. @Gio_Sastre
    

    View Slide

51. @Gio_Sastre
    

    View Slide

52. @Gio_Sastre
    

    View Slide

53. @Gio_Sastre
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    null
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    

    View Slide

54. @Gio_Sastre
    Shrinking
    Trying to find the simplest example that fails
    
    
    for the same reason
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    null
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    

    View Slide

55. @Gio_Sastre
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    null
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    

    View Slide

56. @Gio_Sastre
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    null
    “abc_1234567890“
    

    View Slide

57. @Gio_Sastre
    

    View Slide

58. @Gio_Sastre
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    null
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    null
    

    View Slide

59. @Gio_Sastre
    null
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    null
    

    View Slide

60. @Gio_Sastre
    null
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    fun testPasswordValidator
    (
    
    @ForAll(“noUpperCase”) password: String
    ?
    
    ) {
    @Property(tries = 100)
    val error = passwordValidator.validate(password
    )
    
    assertThat(error
    )
    
    .contains(“must contain upper case letters”)
    }
    @Label(“if password without upper case, error ”
    +
    
    “shows ‘must contain upper case letters’”)
    null
    

    View Slide

61. @Gio_Sastre
    null
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    fun testPasswordValidator
    (
    
    @ForAll(“noUpperCase”) password: String
    ?
    
    ) {
    @Property(tries = 100)
    val error = passwordValidator.validate(password
    )
    
    assertThat(error
    )
    
    .contains(“must contain upper case letters”)
    }
    @Label(“if password without upper case, error ” +
    “shows ‘must contain upper case letters’”)
    null
    

    View Slide

62. @Gio_Sastre
    null
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    fun testPasswordValidator
    (
    
    @ForAll(“noUpperCase”) password: String
    ?
    
    ) {
    @Property(tries = 100, seed = -3508921172508693187)
    val error = passwordValidator.validate(password
    )
    
    assertThat(error
    )
    
    .contains(“must contain upper case letters”)
    }
    @Label(“if password without upper case, error ” +
    “shows ‘must contain upper case letters’”)
    null
    

    View Slide

63. @Gio_Sastre
    null
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    No flaky tests
    Use the seed to reliably reproduce the results
    
    
    of a failing test
    null
    

    View Slide

64. @Gio_Sastre
    null
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    null
    

    View Slide

65. @Gio_Sastre
    null
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    Total edge cases
    fun noUpperCase() =
    @Provide
    Arbitraries.strings().ascii(
    )
    
    .filter { it.matches(“[^A-Z]”.toRegex()) }
    null
    (jqwik only)
    

    View Slide

66. @Gio_Sastre
    null
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    “漢!字)
    𡨸
    $&指>事<字9§漢0=漢*字```+#会意@字<指?%”
    𡨸
    o字9=“
    fun noUpperCase() =
    @Provide
    Arbitraries.strings().ascii(
    )
    
    .filter { it.matches(“[^A-Z]”.toRegex()) }
    noUpperCase().edgeCases()
    null
    (Java.lang.Character.MIN_VALUE)
    “/u0000”
    “” (Empty char)
    “ ” (Space char)
    Total edge cases
    (jqwik only)
    

    View Slide

67. 1
    Property-based testing
    main features
    2
    Shrinking (minimum failing sample)
    Reproduce failing tests (seeds)
    3 Tests for edge cases
    @Gio_Sastre
    

    View Slide

68. What else
    @Gio_Sastre
    can
    Jqwik do?
    

    View Slide

69. @Gio_Sastre
    Stateful testing
    Text editor
    

    View Slide

70. @Gio_Sastre
    Model
    val currentText: String?
    val undoActions: List
    val redoActions: List
    data class TextEditor (
    
    
    )
    
    
    

    View Slide

71. @Gio_Sastre
    Actions on model
    1 fun changeTextTo(newText: String?) { … }
    2 fun undo() { … }
    3 fun redo() { … }
    

    View Slide

72. (if something to redo)
    (if something to undo)
    @Gio_Sastre
    1 On “text change”
    one more “undo” action possible, max 20
    no “redo” actions possible
    2 On “undo”
    one more “redo” action possible, max 20
    next “redo” brings the previous text back
    3 On “redo”
    one more “undo” action possible, max 20
    next “undo” brings the previous text back
    Requirements
    

    View Slide

73. (if something to redo)
    (if something to undo)
    @Gio_Sastre
    1 On “text change”
    one more “undo” action possible, max 20
    no “redo” actions possible
    2 On “undo”
    one more “redo” action possible, max 20
    next “redo” brings the previous text back
    3 On “redo”
    one more “undo” action possible, max 20
    next “undo” brings the previous text back
    Requirements
    

    View Slide

74. (if something to redo)
    @Gio_Sastre
    1 On “text change”
    one more “undo” action possible, max 20
    no “redo” actions possible
    2 On “undo”
    one more “redo” action possible, max 20
    next “redo” brings the previous text back
    3 On “redo”
    one more “undo” action possible, max 20
    next “undo” brings the previous text back
    Requirements
    (if something to undo)
    

    View Slide

75. (if something to undo)
    @Gio_Sastre
    1 On “text change”
    one more “undo” action possible, max 20
    no “redo” actions possible
    2 On “undo”
    one more “redo” action possible, max 20
    next “redo” brings the previous text back
    3 On “redo”
    one more “undo” action possible, max 20
    next “undo” brings the previous text back
    Requirements
    (if something to redo)
    

    View Slide

76. @Gio_Sastre
    1 On “text change”
    one more “undo” action possible, max 20
    no “redo” actions possible
    2 On “undo”
    one more “redo” action possible, max 20
    next “redo” brings the previous text back
    3 On “redo”
    one more “undo” action possible, max 20
    next “undo” brings the previous text back
    Requirements
    (if something to undo)
    (if something to redo)
    

    View Slide

77. @Gio_Sastre
    1 On “text change”
    one more “undo” action possible, max 20
    no “redo” actions possible
    2 On “undo”
    one more “redo” action possible, max 20
    next “redo” brings the previous text back
    3 On “redo”
    one more “undo” action possible, max 20
    next “undo” brings the previous text back
    Requirements
    (if something to undo)
    (if something to redo)
    

    View Slide

78. @Gio_Sastre
    1 On “text change”
    one more “undo” action possible, max 20
    no “redo” actions possible
    fun onTextChange_max_undo_actions_is_20() {
    @Test
    val editor = TextEditor(
    )
    
    repeat(21){
    
    
    editor.textChange(“abc”)
    
    
    }
    assertThat(editor.undoActions).size(20)
    }
    Requirements
    

    View Slide

79. @Gio_Sastre
    1 On “text change”
    one more “undo” action possible, max 20
    no “redo” actions possible
    fun onTextChange_max_undo_actions_is_20() {
    @Test
    val editor = TextEditor(
    )
    
    repeat(21){
    
    
    editor.textChange(“abc”)
    
    
    }
    assertThat(editor.undoActions).size(20)
    }
    Requirements
    

    View Slide

80. @Gio_Sastre
    1 On “text change”
    one more “undo” action possible, max 20
    no “redo” actions possible
    

    View Slide

81. @Gio_Sastre
    1 On “text change”
    one more “undo” action possible, max 20
    no “redo” actions possible
    class TextChange(val text: String?): Action {
    override fun run(editor: TextEditor)
    {
    
    val previousUndoCount = editor.undoActionsCount(
    )
    
    
    
    editor.changeTextTo(text
    )
    
    assertThat(editor
    )
    
    .undoActionsEquals(min(previousUndoCount + 1, 20)
    )
    
    .redoActionsEquals(0
    )
    
    }
    
    }
    

    View Slide

82. @Gio_Sastre
    1 On “text change”
    one more “undo” action possible, max 20
    no “redo” actions possible
    class TextChange(val text: String?): Action {
    override fun run(editor: TextEditor)
    {
    
    val previousUndoCount = editor.undoActionsCount(
    )
    
    
    
    editor.changeTextTo(text
    )
    
    assertThat(editor
    )
    
    .undoActionsEquals(min(previousUndoCount + 1, 20)
    )
    
    .redoActionsEquals(0
    )
    
    }
    
    }
    

    View Slide

83. class TextChange(val text: String?): Action {
    override fun run(editor: TextEditor)
    {
    
    val previousUndoCount = editor.undoActionsCount(
    )
    
    
    
    editor.changeTextTo(text
    )
    
    assertThat(editor
    )
    
    .undoActionsEquals(min(previousUndoCount + 1, 20)
    )
    
    .redoActionsEquals(0
    )
    
    }
    
    }
    @Gio_Sastre
    1 On “text change”
    one more “undo” action possible, max 20
    no “redo” actions possible
    

    View Slide

84. @Gio_Sastre
    class TextChange(val text: String?): Action {
    override fun run(editor: TextEditor)
    {
    
    val previousUndoCount = editor.undoActionsCount(
    )
    
    
    
    editor.changeTextTo(text
    )
    
    assertThat(editor
    )
    
    .undoActionsEquals(min(previousUndoCount + 1, 20)
    )
    
    .redoActionsEquals(0
    )
    
    }
    
    }
    1 On “text change”
    one more “undo” action possible, max 20
    no “redo” actions possible
    

    View Slide

85. @Gio_Sastre
    class TextChange(val text: String?): Action {
    override fun run(editor: TextEditor)
    {
    
    val previousUndoCount = editor.undoActionsCount(
    )
    
    
    
    editor.changeTextTo(text
    )
    
    assertThat(editor
    )
    
    .undoActionsEquals(min(previousUndoCount + 1, 20)
    )
    
    .redoActionsEquals(0
    )
    
    }
    
    }
    1 On “text change”
    one more “undo” action possible, max 20
    no “redo” actions possible
    

    View Slide

86. @Gio_Sastre
    2 On “undo”
    one more “redo” action possible, max 20
    next “redo” brings the previous text back
    (if something to undo)
    

    View Slide

87. @Gio_Sastre
    2 On “undo”
    one more “redo” action possible, max 20
    next “redo” brings the previous text back
    (if something to undo)
    class Undo(): Action {
    override fun run(editor: TextEditor)
    {
    
    val preEditor = editor.copy(
    )
    
    
    
    editor.undo(
    )
    
    assertThat(editor
    )
    
    .redoActionsEquals(min(preEditor.redoActions() + 1, 20)
    )
    
    .nextRedoTextEquals(preEditor.text
    )
    
    }
    
    }
    override fun precondition(editor: TextEditor) =
    editor.undoActionsCount() > 0
    

    View Slide

88. @Gio_Sastre
    2 On “undo”
    one more “redo” action possible, max 20
    next “redo” brings the previous text back
    (if something to undo)
    class Undo(): Action {
    override fun run(editor: TextEditor)
    {
    
    val preEditor = editor.copy(
    )
    
    
    
    editor.undo(
    )
    
    assertThat(editor
    )
    
    .redoActionsEquals(min(preEditor.redoActions() + 1, 20)
    )
    
    .nextRedoTextEquals(preEditor.text
    )
    
    }
    
    }
    override fun precondition(editor: TextEditor) =
    editor.undoActionsCount() > 0
    

    View Slide

89. @Gio_Sastre
    2 On “undo”
    one more “redo” action possible, max 20
    next “redo” brings the previous text back
    (if something to undo)
    class Undo(): Action {
    override fun run(editor: TextEditor)
    {
    
    val preEditor = editor.copy(
    )
    
    
    
    editor.undo(
    )
    
    assertThat(editor
    )
    
    .redoActionsEquals(min(preEditor.redoActions() + 1, 20)
    )
    
    .nextRedoTextEquals(preEditor.text
    )
    
    }
    
    }
    override fun precondition(editor: TextEditor) =
    editor.undoActionsCount() > 0
    

    View Slide

90. fun randomTextChange() = Arbitraries.strings().map { TextChange(it)
    }
    
    fun undo() = Arbitraries.just(Undo())
    fun redo() = Arbitraries.just(Redo())
    @Gio_Sastre
    Semi-random action generator
    fun textEditorActionGenerator() =
    @Provide
    Arbitraries.sequences(randomActions())
    fun randomActions() = Arbitraries.oneOf(randomTextChange(), undo(), redo())
    

    View Slide

91. @Gio_Sastre
    Semi-random action generator
    fun textEditorActionGenerator() =
    @Provide
    Arbitraries.sequences(randomActions())
    fun randomActions() = Arbitraries.oneOf(randomTextChange(), undo(), redo())
    fun randomTextChange() = Arbitraries.strings().map { TextChange(it)
    }
    
    fun undo() = Arbitraries.just(Undo())
    fun redo() = Arbitraries.just(Redo())
    

    View Slide

92. @Gio_Sastre
    Semi-random action generator
    fun textEditorActionGenerator() =
    @Provide
    Arbitraries.sequences(randomActions())
    fun randomActions() = Arbitraries.oneOf(randomTextChange(), undo(), redo())
    fun randomTextChange() = Arbitraries.strings().map { TextChange(it)
    }
    
    fun undo() = Arbitraries.just(Undo())
    fun redo() = Arbitraries.just(Redo())
    

    View Slide

93. @Gio_Sastre
    Semi-random action generator
    fun textEditorActionGenerator() =
    @Provide
    Arbitraries.sequences(randomActions())
    fun randomActions() = Arbitraries.oneOf(randomTextChange(), undo(), redo())
    fun randomTextChange() = Arbitraries.strings().map { TextChange(it)
    }
    
    fun undo() = Arbitraries.just(Undo())
    fun redo() = Arbitraries.just(Redo())
    

    View Slide

94. @Gio_Sastre
    Semi-random action generator
    fun textEditorActionGenerator() =
    @Provide
    Arbitraries.sequences(randomActions())
    fun randomActions() = Arbitraries.oneOf(randomTextChange(), undo(), redo())
    fun randomTextChange() = Arbitraries.strings().map { TextChange(it)
    }
    
    fun undo() = Arbitraries.just(Undo())
    fun redo() = Arbitraries.just(Redo())
    

    View Slide

95. @Gio_Sastre
    fun randomActions() = Arbitraries.oneOf(randomTextChange(), undo(), redo())
    fun textEditorActionGenerator() =
    @Provide
    Arbitraries.sequences(randomActions())
    Property test
    fun statefulTextEditorTest
    (
    
    @ForAll(“textEditorActionGenerator”) actions: ActionSequence>
    
    ) = actions.run(TextEditor())
    @Property
    fun randomTextChange() = Arbitraries.strings().map { TextChange(it)
    }
    
    fun undo() = Arbitraries.just(Undo())
    fun redo() = Arbitraries.just(Redo())
    class TextChange(val text: String?): Action { … }
    class Undo(): Action { … }
    class Redo(): Action { … }
    

    View Slide

96. @Gio_Sastre
    class TextChange(val text: String?): Action { … }
    class Undo(): Action { … }
    class Redo(): Action { … }
    fun statefulTextEditorTest
    (
    
    @ForAll(“textEditorActionGenerator”) actions: ActionSequence>
    
    ) = actions.run(TextEditor())
    @Property
    fun textEditorActionGenerator() =
    @Provide
    Arbitraries.sequences(randomActions())
    Property test
    

    View Slide

97. 1
    Property-based stateful testing
    2
    Identify actions
    Preconditions in actions
    3 Assertions in actions
    @Gio_Sastre
    with Jqwik
    

    View Slide

98. @Gio_Sastre
    pros & cons
    Property-based testing
    of
    

    View Slide

99. Cons
    @Gio_Sastre
    Pros
    @Gio_Sastre
    

    View Slide

100. Cons
     @Gio_Sastre
     Pros
     @Gio_Sastre
     More robust tests
     Cover edge cases we might forget
     Random but deterministic tests
     

     View Slide

101. Cons
     Hard to find properties
     Properties are not always sufficient
     @Gio_Sastre
     More robust tests
     Cover edge cases we might forget
     Random but deterministic tests
     Pros
     @Gio_Sastre
     Run more tests, so run more slowly
     

     View Slide

102. Cons
     Hard to find properties
     @Gio_Sastre
     @Gio_Sastre
     fun String?.reversed() : String? { … }
     Create method that reverses a given string
     Properties are not always sufficient
     Run more tests, so run more slowly
     

     View Slide

103. Cons
     Hard to find properties
     @Gio_Sastre
     @Gio_Sastre
     Patterns to find properties
     Business rule as property
     If password contains blanks show error message…
     fun String?.reversed() : String? { … }
     Create method that reverses a given string
     Properties are not always sufficient
     Run more tests, so run more slowly
     

     View Slide

104. Cons
     @Gio_Sastre
     @Gio_Sastre
     Patterns to find properties
     Test Oracle
     list.bubbleSort() == list.moreEfficientSort()
     fun String?.reversed() : String? { … }
     Create method that reverses a given string
     Hard to find properties
     Properties are not always sufficient
     Run more tests, so run more slowly
     

     View Slide

105. Cons
     @Gio_Sastre
     @Gio_Sastre
     Invariant functions
     Patterns to find properties
     string.reverse()?.length == string?.length
     fun String?.reversed() : String? { … }
     Create method that reverses a given string
     Hard to find properties
     Properties are not always sufficient
     Run more tests, so run more slowly
     

     View Slide

106. Cons
     @Gio_Sastre
     @Gio_Sastre
     fun String?.reversed() : String? { … }
     @Propert
     y
     
     fun reversedStringKeepsLength
     (
     
     @ForAll string: String
     ?
     
     )
     {
     
     assertTrue
     (
     
     string.reversed()?.length == string?.lengt
     h
     
     )
     
     }
     Create method that reverses a given string
     Hard to find properties
     Properties are not always sufficient
     Run more tests, so run more slowly
     

     View Slide

107. Cons
     @Gio_Sastre
     @Gio_Sastre
     Inverse functions
     Patterns to find properties
     string.reversed().reversed() == string
     @Propert
     y
     
     fun reversedStringKeepsLength
     (
     
     @ForAll string: String
     ?
     
     )
     {
     
     assertTrue
     (
     
     string.reversed()?.length == string?.lengt
     h
     
     )
     
     }
     fun String?.reversed() : String? { … }
     Create method that reverses a given string
     Hard to find properties
     Properties are not always sufficient
     Run more tests, so run more slowly
     

     View Slide

108. Cons
     @Gio_Sastre
     @Gio_Sastre
     @Propert
     y
     
     fun reversedStringKeepsLength
     (
     
     @ForAll string: String
     ?
     
     )
     {
     
     assertTrue
     (
     
     string.reversed()?.length == string?.lengt
     h
     
     )
     
     }
     @Propert
     y
     
     fun reversedStringTwiceReturnsOriginal
     (
     
     @ForAll string: String
     ?
     
     )
     {
     
     assertTrue
     (
     
     string.reversed().reversed() == string
     
     
     )
     
     }
     fun String?.reversed() : String? { … }
     Create method that reverses a given string
     Hard to find properties
     Properties are not always sufficient
     Run more tests, so run more slowly
     

     View Slide

109. Cons
     @Gio_Sastre
     @Gio_Sastre
     @Propert
     y
     
     fun reversedStringKeepsLength
     (
     
     @ForAll string: String
     ?
     
     )
     {
     
     assertTrue
     (
     
     string.reversed()?.length == string?.lengt
     h
     
     )
     
     }
     @Propert
     y
     
     fun reversedStringTwiceReturnsOriginal
     (
     
     @ForAll string: String
     ?
     
     )
     {
     
     assertTrue
     (
     
     string.reversed().reversed() == string
     
     
     )
     
     }
     fun String?.reversed() : String? { … }
     Create method that reverses a given string
     Hard to find properties
     Properties are not always sufficient
     Run more tests, so run more slowly
     

     View Slide

110. Cons
     @Gio_Sastre
     @Gio_Sastre
     @Propert
     y
     
     fun reversedStringKeepsLength
     (
     
     @ForAll string: String
     ?
     
     )
     {
     
     assertTrue
     (
     
     string.reversed()?.length == string?.lengt
     h
     
     )
     
     }
     @Propert
     y
     
     fun reversedStringTwiceReturnsOriginal
     (
     
     @ForAll string: String
     ?
     
     )
     {
     
     assertTrue
     (
     
     string.reversed().reversed() == string
     
     
     )
     
     }
     fun String?.reversed() : String? = this
     Create method that reverses a given string
     Hard to find properties
     Properties are not always sufficient
     Run more tests, so run more slowly
     

     View Slide

111. Cons
     @Gio_Sastre
     @Gio_Sastre
     @Propert
     y
     
     fun reversedStringKeepsLength
     (
     
     @ForAll string: String
     ?
     
     )
     {
     
     assertTrue
     (
     
     string.reversed()?.length == string?.lengt
     h
     
     )
     
     }
     @Propert
     y
     
     fun reversedStringTwiceReturnsOriginal
     (
     
     @ForAll string: String
     ?
     
     )
     {
     
     assertTrue
     (
     
     string.reversed().reversed() == string
     
     
     )
     
     }
     fun String?.reversed() : String? = this
     this this
     Create method that reverses a given string
     Hard to find properties
     Properties are not always sufficient
     Run more tests, so run more slowly
     

     View Slide

112. Cons
     @Gio_Sastre
     @Gio_Sastre
     @Propert
     y
     
     fun reversedStringKeepsLength
     (
     
     @ForAll string: String
     ?
     
     )
     {
     
     assertTrue
     (
     
     string.reversed()?.length == string?.lengt
     h
     
     )
     
     }
     @Propert
     y
     
     fun reversedStringTwiceReturnsOriginal
     (
     
     @ForAll string: String
     ?
     
     )
     {
     
     assertTrue
     (
     
     string.reversed().reversed() == string
     
     
     )
     
     }
     fun String?.reversed() : String? = this
     this this
     Create method that reverses a given string
     Hard to find properties
     Properties are not always sufficient
     Run more tests, so run more slowly
     

     View Slide

113. Cons
     @Gio_Sastre
     @Gio_Sastre
     Create method that reverses a given string
     @Propert
     y
     
     fun reversedStringKeepsLength
     (
     
     @ForAll string: String
     ?
     
     )
     {
     
     assertTrue
     (
     
     string.reversed()?.length == string?.lengt
     h
     
     )
     
     }
     @Propert
     y
     
     fun reversedStringTwiceReturnsOriginal
     (
     
     @ForAll string: String
     ?
     
     )
     {
     
     assertTrue
     (
     
     string.reversed().reversed() == string
     
     
     )
     
     }
     fun String?.reversed() : String? = this
     this this
     Hard to find properties
     Properties are not always sufficient
     Run more tests, so run more slowly
     

     View Slide

114. Cons
     @Gio_Sastre
     @Gio_Sastre
     @Propert
     y
     
     fun reversedStringKeepsLength
     (
     
     @ForAll string: String
     ?
     
     )
     {
     
     assertTrue
     (
     
     string.reversed()?.length == string?.lengt
     h
     
     )
     
     }
     @Propert
     y
     
     fun reversedStringTwiceReturnsOriginal
     (
     
     @ForAll string: String
     ?
     
     )
     {
     
     assertTrue
     (
     
     string.reversed().reversed() == string
     
     
     )
     
     }
     fun String?.reversed() : String? { … }
     Create method that reverses a given string
     Hard to find properties
     Properties are not always sufficient
     Run more tests, so run more slowly
     

     View Slide

115. Cons
     @Gio_Sastre
     @Gio_Sastre
     @Property(times = 100
     )
     
     fun reversedStringKeepsLength
     (
     
     @ForAll string: String
     ?
     
     )
     {
     
     assertTrue
     (
     
     string.reversed()?.length == string?.lengt
     h
     
     )
     
     }
     @Property(times = 100
     )
     
     fun reversedStringTwiceReturnsOriginal
     (
     
     @ForAll string: String
     ?
     
     )
     {
     
     assertTrue
     (
     
     string.reversed().reversed() == string
     
     
     )
     
     }
     fun String?.reversed() : String? { … }
     Create method that reverses a given string
     Hard to find properties
     Properties are not always sufficient
     Run more tests, so run more slowly
     

     View Slide

116. 1
     Property-based testing (PBT)
     opinionated advice
     2
     Example + property tests
     PBT for widely used code
     3
     @Gio_Sastre
     PBT for obvious properties
     

     View Slide

117. attention
     for your
     THANKS
     @Gio_Sastre
     

     View Slide

118. @Gio_Sastre
     code
     @sergio-sastre
     Sergio Sastre Flórez
     Lead & Senior
     Android developer appdev.de
     blogs
     @SergioSastre
     sergiosastre.hashnode.dev
     

     View Slide