$30 off During Our Annual Pro Sale. View Details »

Property-based test beginning with SwiftCheck

Property-based test beginning with SwiftCheck

Yusuke Hosonuma

March 22, 2019
Tweet

More Decks by Yusuke Hosonuma

Other Decks in Programming

Transcript

  1. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Property-based test
    beginning with SwiftCheck
    Yusuke Hosonuma
    @DeNA
    try! Swift 2019 TOKYO / Day 2

    View Slide

  2. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Yusuke Hosonuma
    @DeNA
    Favorite
    Language

    Swift / Go / Haskell
    Twitter: @tobi462

    View Slide

  3. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    SoftWare Engineer in Test
    Test Automation, CI/CD and more…

    View Slide

  4. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Organizer of the Events
    iOSɹɹAndroidɹɹCI/CD
    https://testnight.connpass.com/

    View Slide

  5. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Private Developer Community
    Pengin-mura

    View Slide

  6. Agenda
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.

    View Slide

  7. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Agenda
    • What’s Property-based test ?
    • What’s SwiftCheck ?
    • Use cases in Action
    • Conclusion

    View Slide

  8. 2VJDL
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    What’s

    Property-based test

    View Slide

  9. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Property-based test
    • Describe tests as property

    • Randomly generated input values

    View Slide

  10. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Property-based test
    • Describe tests as property

    • Randomly generated input values

    View Slide

  11. 2VJDL
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Example-based test

    View Slide

  12. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Example-based test
    • Give input and expected values

    • Pass the input values to the function

    • Check that the output result matches
    the expected value

    View Slide

  13. 2VJDL
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Array#reversed

    View Slide

  14. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Array#reversed
    1 2 3 4 5
    5 4 3 2 1

    View Slide

  15. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Array#reversed
    func testReversed() {
    let xs = [1, 2, 3, 4, 5]
    XCTAssertEqual([5, 4, 3, 2, 1], xs.reversed())
    }

    View Slide

  16. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Array#reversed
    func testReversed() {
    let xs = [1, 2, 3, 4, 5]
    XCTAssertEqual([5, 4, 3, 2, 1], xs.reversed())
    }

    View Slide

  17. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Array#reversed
    func testReversed() {
    let xs = [1, 2, 3, 4, 5]
    XCTAssertEqual([5, 4, 3, 2, 1], xs.reversed())
    }

    View Slide

  18. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Array#reversed
    func testReversed() {
    let xs = [1, 2, 3, 4, 5]
    XCTAssertEqual(xs.reversed(), [5, 4, 3, 2, 1])
    }

    How many patterns are necessary?

    View Slide

  19. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Array#reversed
    func testReversed() {
    let empty = Array()
    XCTAssertEqual(empty.reversed(), empty)
    XCTAssertEqual( [1].reversed(), [1])
    XCTAssertEqual( [1, 2].reversed(), [2, 1])
    XCTAssertEqual([1, 2, 3, 4, 5].reversed(), [5, 4, 3, 2, 1])
    ...
    }

    View Slide

  20. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Array#reversed
    func testReversed() {
    let empty = Array()
    XCTAssertEqual(empty.reversed(), empty)
    XCTAssertEqual( [1].reversed(), [1])
    XCTAssertEqual( [1, 2].reversed(), [2, 1])
    XCTAssertEqual([1, 2, 3, 4, 5].reversed(), [5, 4, 3, 2, 1])
    ...
    }

    There are infinite possible input values!

    View Slide

  21. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    How do we choose input values?
    • A value that is likely to cause a bug

    ⁃ Boundary value

    ⁃ Equivalence partition

    View Slide

  22. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    How do we choose input values?
    • A value that is likely to cause a bug

    ⁃ Boundary value

    ⁃ Equivalence partition

    Detect bugs when missed patterns

    View Slide

  23. 2VJDL
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Property-based test

    View Slide

  24. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Think property
    • Specific input values are not considered.

    • What is the relationship between the
    input value and the output value?

    • What is property of function?

    View Slide

  25. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    What’s property
    1 2 3 4 5
    5 4 3 2 1

    View Slide

  26. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    The length does not change
    1 2 3
    1 1 2 … 9
    3 2 1
    1 9 … 2 1

    View Slide

  27. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    reverse x2 = original
    1 2 3 4 5
    1 2 3
    5 4 3 2 1
    3 2 1
    1 2 3 4 5
    1 2 3

    View Slide

  28. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Property of #reversed
    • The length does not change

    • When it is executed twice, it returns to
    the original

    View Slide

  29. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Property of #reversed
    • The length does not change

    • When it is executed twice, it returns to
    the original
    This is a common property
    to any input values

    View Slide

  30. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Property-based test by self
    for xs in randomArrays() {
    // The length does not change
    XCTAssertEqual(xs.reversed().count, xs.count)
    // When it is executed twice, it returns to the original
    XCTAssertEqual(xs.reversed().reversed(), xs)
    }
    func randomArrays() -> [[Int]] {
    // generate random arrays
    }

    View Slide

  31. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Property-based test by self
    for xs in randomArrays() {
    // The length does not change
    XCTAssertEqual(xs.reversed().count, xs.count)
    // When it is executed twice, it returns to the original
    XCTAssertEqual(xs.reversed().reversed(), xs)
    }
    func randomArrays() -> [[Int]] {
    // generate random arrays
    }

    View Slide

  32. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Property-based test by self
    for xs in randomArrays() {
    // The length does not change
    XCTAssertEqual(xs.reversed().count, xs.count)
    // When it is executed twice, it returns to the original
    XCTAssertEqual(xs.reversed().reversed(), xs)
    }
    func randomArrays() -> [[Int]] {
    // generate random arrays
    }

    View Slide

  33. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Property-based test by self
    for xs in randomArrays() {
    // The length does not change
    XCTAssertEqual(xs.reversed().count, xs.count)
    // When it is executed twice, it returns to the original
    XCTAssertEqual(xs.reversed().reversed(), xs)
    }
    func randomArrays() -> [[Int]] {
    // generate random arrays
    }

    View Slide

  34. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Property-based test by self
    for xs in randomArrays() {
    // The length does not change
    XCTAssertEqual(xs.reversed().count, xs.count)
    // When it is executed twice, it returns to the original
    XCTAssertEqual(xs.reversed().reversed(), xs)
    }
    func randomArrays() -> [[Int]] {
    // generate random arrays
    }
    Let’s try SwiftCheck !

    View Slide

  35. 2VJDL
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    What’s SwiftCheck ?

    View Slide

  36. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    SwiftCheck
    • Property-based test in Swift (OSS)

    • Inspired by QuickCheck (in Haskell)

    • with XCTest
    https://github.com/typelift/SwiftCheck

    View Slide

  37. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Swift is inspired by Haskell
    • Functionality

    • Enum

    • Pattern match

    View Slide

  38. 2VJDL
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Write the 1st SwiftCheck

    View Slide

  39. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    The length does not change
    func testReversed() {
    property("length not changed”) <- forAll { (xs: [Int]) in
    return xs.reversed().count == xs.count
    }
    }

    View Slide

  40. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    The length does not change
    func testReversed() {
    property("length not changed”) <- forAll { (xs: [Int]) in
    return xs.reversed().count == xs.count
    }
    }
    XCTest method

    View Slide

  41. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    The length does not change
    func testReversed() {
    property("length not changed”) <- forAll { (xs: [Int]) in
    return xs.reversed().count == xs.count
    }
    }
    Describe summary

    View Slide

  42. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    The length does not change
    func testReversed() {
    property("length not changed”) <- forAll { (xs: [Int]) in
    return xs.reversed().count == xs.count
    }
    }
    Random values

    View Slide

  43. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    The length does not change
    func testReversed() {
    property("length not changed”) <- forAll { (xs: [Int]) in
    return xs.reversed().count == xs.count
    }
    }
    Describe property

    View Slide

  44. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    The length does not change
    func testReversed() {
    property("length not changed”) <- forAll { (xs: [Int]) in
    return xs.reversed().count == xs.count
    }
    }

    View Slide

  45. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    The length does not change
    func testReversed() {
    property("length not changed”) <- forAll { (xs: [Int]) in
    return xs.reversed().count == xs.count
    }
    }

    What input values are generated?

    View Slide

  46. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Print generated values
    []
    []
    [-2, -2]
    [-2]
    [2, -1, 4]
    [1, -4, -4]
    [-5, -4]
    ...
    *** Passed 100 tests

    View Slide

  47. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Print generated values
    []
    []
    [-2, -2]
    [-2]
    [2, -1, 4]
    [1, -4, -4]
    [-5, -4]
    ...
    *** Passed 100 tests
    Every time random

    View Slide

  48. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Print generated values
    []
    []
    [-2, -2]
    [-2]
    [2, -1, 4]
    [1, -4, -4]
    [-5, -4]
    ...
    *** Passed 100 tests
    Test is succeeded

    View Slide

  49. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    When failed
    *** Failed! Proposition: length not changed
    Falsifiable (after 1 test):
    []
    *** Passed 0 tests

    View Slide

  50. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    When failed
    *** Failed! Proposition: length not changed
    Falsifiable (after 1 test):
    []
    *** Passed 0 tests
    Summary

    View Slide

  51. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    When failed
    *** Failed! Proposition: length not changed
    Falsifiable (after 1 test):
    []
    *** Passed 0 tests
    Failed value

    View Slide

  52. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    2 words
    • Arbitrary

    • Shrinking

    View Slide

  53. 2VJDL
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Arbitrary

    View Slide

  54. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Arbitrary
    • Protocol to generate random values

    • Adopt it can be used as a random value

    • Swift standard types are adopted

    View Slide

  55. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Arbitrary Protocol
    public protocol Arbitrary {
    static var arbitrary : Gen { get }
    static func shrink(_ : Self) -> [Self]
    }

    View Slide

  56. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Arbitrary Protocol
    public protocol Arbitrary {
    static var arbitrary : Gen { get }
    static func shrink(_ : Self) -> [Self]
    }

    View Slide

  57. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Custom type
    struct Point {
    var x: Int
    var y: Int
    }
    extension Point: Arbitrary {
    static var arbitrary: Gen {
    return Gen<(Int, Int)>
    .zip(Int.arbitrary, Int.arbitrary)
    .map(Point.init)
    }
    }

    View Slide

  58. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Custom type
    struct Point {
    var x: Int
    var y: Int
    }
    extension Point: Arbitrary {
    static var arbitrary: Gen {
    return Gen<(Int, Int)>
    .zip(Int.arbitrary, Int.arbitrary)
    .map(Point.init)
    }
    }
    Has pair of Int

    View Slide

  59. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Custom type
    struct Point {
    var x: Int
    var y: Int
    }
    extension Point: Arbitrary {
    static var arbitrary: Gen {
    return Gen<(Int, Int)>
    .zip(Int.arbitrary, Int.arbitrary)
    .map(Point.init)
    }
    }
    Return generater of Point

    View Slide

  60. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Custom type
    for _ in 0...4 {
    print(Point.arbitrary.generate)
    }
    // Point(x: -2, y: -26)
    // Point(x: -15, y: 2)
    // Point(x: 20, y: -21)
    // Point(x: -30, y: -15)
    // Point(x: -30, y: 6)
    Generated random Points

    View Slide

  61. 2VJDL
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Shrinking

    View Slide

  62. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Shrinking
    • Get more small values

    • Reported a smaller failure case

    View Slide

  63. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Arbitrary Protocol
    public protocol Arbitrary {
    static var arbitrary : Gen { get }
    static func shrink(_ : Self) -> [Self]
    }

    View Slide

  64. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Arbitrary Protocol
    public protocol Arbitrary {
    static var arbitrary : Gen { get }
    static func shrink(_ : Self) -> [Self]
    }

    View Slide

  65. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Shrinking
    10
    10 is failed

    View Slide

  66. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Shrinking
    10
    shrink
    5

    View Slide

  67. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Shrinking
    10
    still failed
    5

    View Slide

  68. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Shrinking
    10 5 2
    still failed

    View Slide

  69. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Shrinking
    10 5 2 1
    still failed

    View Slide

  70. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Shrinking
    10
    passed !
    5 2 1 0

    View Slide

  71. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Shrinking
    10 5 2 1 0
    reported !

    View Slide

  72. 2VJDL
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Use cases in Action

    View Slide

  73. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Use cases
    • Randomized algorithm

    • Symmetric algorithm

    • Fast vs Slow

    View Slide

  74. 2VJDL
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Randomized algorithm

    View Slide

  75. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Example: Maze generation
    • Example-based Test is difficult

    • Random number seeds can given as
    input values, but the algorithm can not
    be improved

    View Slide

  76. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Example: Maze generation
    • Property

    ⁃ Maze must always be solved

    View Slide

  77. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Example: Maze generation
    func canSolve(_ maze: Maze) -> Bool {
    // judge solve or not
    }
    property("must solve") <- forAll { (seed: UInt64) in
    let maze = Maze.generate(seed: seed)
    return canSolve(maze)
    }

    View Slide

  78. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Example: Maze generation
    func canSolve(_ maze: Maze) -> Bool {
    // judge solve or not
    }
    property("must solve") <- forAll { (seed: UInt64) in
    let maze = Maze.generate(seed: seed)
    return canSolve(maze)
    }
    generate seed

    View Slide

  79. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Example: Maze generation
    func canSolve(_ maze: Maze) -> Bool {
    // judge solve or not
    }
    property("must solve") <- forAll { (seed: UInt64) in
    let maze = Maze.generate(seed: seed)
    return canSolve(maze)
    }
    always solve

    View Slide

  80. 2VJDL
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Symmetric algorithm

    View Slide

  81. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Example: encode / decode
    • Encoded and decoded matches the
    original value

    • The reverse is also the same

    View Slide

  82. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Example: encode / decode
    property("encode and decode are synmetry”)
    <- forAll { (bird: Bird) in
    let bytes = bird.encodeToBytes()
    return Bird.decode(bytes: bytes) == bird
    }

    View Slide

  83. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Example: encode / decode
    property("encode and decode are synmetry”)
    <- forAll { (bird: Bird) in
    let bytes = bird.encodeToBytes()
    return Bird.decode(bytes: bytes) == bird
    }

    View Slide

  84. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Example: encode / decode
    property("encode and decode are synmetry”)
    <- forAll { (bird: Bird) in
    let bytes = bird.encodeToBytes()
    return Bird.decode(bytes: bytes) == bird
    }

    View Slide

  85. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Example: encode / decode
    property("encode and decode are synmetry”)
    <- forAll { (bird: Bird) in
    let bytes = bird.encodeToBytes()
    return Bird.decode(bytes: bytes) == bird
    }

    View Slide

  86. 2VJDL
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Fast vs Slow

    View Slide

  87. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Fast vs Slow
    • Fast algorithm is complicated to implement

    • The obvious algorithm is slow

    View Slide

  88. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Fast vs Slow
    • Both should be the same result

    • Use explicit (but slow) algorithms for fast
    algorithm (but complex) verification

    View Slide

  89. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Fast vs Slow
    property("same result") <- forAll { (xs: [Int]) in
    return slowSort(xs) == fastSort(xs)
    }

    View Slide

  90. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Fast vs Slow
    property("same result") <- forAll { (xs: [Int]) in
    return slowSort(xs) == fastSort(xs)
    }

    View Slide

  91. 2VJDL
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Conclusion

    View Slide

  92. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Property-based Test
    • Describe tests as property

    • Randomly generated input values
    • SwiftCheck is QuickCheck for Swift

    View Slide

  93. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Is the Example-based Test is Dead?
    • Does not replace the Example-based Test

    • Reduce anxiety about the miss of test
    patterns.

    • Combining them makes it a powerful card

    View Slide

  94. 2VJDL
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Let’s try Property-based test
    with SwiftCheck

    View Slide

  95. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Appendix
    • https://en.wikipedia.org/wiki/Property_testing

    • https://github.com/typelift/SwiftCheck

    • https://www.objc.io/books/functional-swift/

    View Slide