Save 37% off PRO during our Black Friday Sale! »

Property-based test beginning with SwiftCheck

Property-based test beginning with SwiftCheck

70d9714ea13fc1133803d61fb16e4160?s=128

Yusuke Hosonuma

March 22, 2019
Tweet

Transcript

  1. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test beginning

    with SwiftCheck Yusuke Hosonuma @DeNA try! Swift 2019 TOKYO / Day 2
  2. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Yusuke Hosonuma @DeNA

    Favorite Language Swift / Go / Haskell Twitter: @tobi462
  3. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. SoftWare Engineer in

    Test Test Automation, CI/CD and more…
  4. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Organizer of the

    Events iOSɹɹAndroidɹɹCI/CD https://testnight.connpass.com/
  5. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Private Developer Community

    Pengin-mura
  6. Agenda Copyright (C) DeNA Co.,Ltd. All Rights Reserved.

  7. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Agenda • What’s

    Property-based test ? • What’s SwiftCheck ? • Use cases in Action • Conclusion
  8. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. What’s Property-based

    test
  9. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test •

    Describe tests as property • Randomly generated input values
  10. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test •

    Describe tests as property • Randomly generated input values
  11. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example-based test

  12. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example-based test •

    Give input and expected values • Pass the input values to the function • Check that the output result matches the expected value
  13. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed

  14. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed 1 2

    3 4 5 5 4 3 2 1
  15. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed()

    { let xs = [1, 2, 3, 4, 5] XCTAssertEqual([5, 4, 3, 2, 1], xs.reversed()) }
  16. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed()

    { let xs = [1, 2, 3, 4, 5] XCTAssertEqual([5, 4, 3, 2, 1], xs.reversed()) }
  17. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed()

    { let xs = [1, 2, 3, 4, 5] XCTAssertEqual([5, 4, 3, 2, 1], xs.reversed()) }
  18. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed()

    { let xs = [1, 2, 3, 4, 5] XCTAssertEqual(xs.reversed(), [5, 4, 3, 2, 1]) } How many patterns are necessary?
  19. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed()

    { let empty = Array<Int>() XCTAssertEqual(empty.reversed(), empty) XCTAssertEqual( [1].reversed(), [1]) XCTAssertEqual( [1, 2].reversed(), [2, 1]) XCTAssertEqual([1, 2, 3, 4, 5].reversed(), [5, 4, 3, 2, 1]) ... }
  20. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed()

    { let empty = Array<Int>() XCTAssertEqual(empty.reversed(), empty) XCTAssertEqual( [1].reversed(), [1]) XCTAssertEqual( [1, 2].reversed(), [2, 1]) XCTAssertEqual([1, 2, 3, 4, 5].reversed(), [5, 4, 3, 2, 1]) ... } There are infinite possible input values!
  21. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. How do we

    choose input values? • A value that is likely to cause a bug ⁃ Boundary value ⁃ Equivalence partition
  22. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. How do we

    choose input values? • A value that is likely to cause a bug ⁃ Boundary value ⁃ Equivalence partition Detect bugs when missed patterns
  23. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test

  24. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Think property •

    Specific input values are not considered. • What is the relationship between the input value and the output value? • What is property of function?
  25. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. What’s property 1

    2 3 4 5 5 4 3 2 1
  26. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change 1 2 3 1 1 2 … 9 3 2 1 1 9 … 2 1
  27. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. reverse x2 =

    original 1 2 3 4 5 1 2 3 5 4 3 2 1 3 2 1 1 2 3 4 5 1 2 3
  28. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property of #reversed

    • The length does not change • When it is executed twice, it returns to the original
  29. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property of #reversed

    • The length does not change • When it is executed twice, it returns to the original This is a common property to any input values
  30. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by

    self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays }
  31. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by

    self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays }
  32. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by

    self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays }
  33. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by

    self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays }
  34. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by

    self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays } Let’s try SwiftCheck !
  35. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. What’s SwiftCheck

    ?
  36. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. SwiftCheck • Property-based

    test in Swift (OSS) • Inspired by QuickCheck (in Haskell) • with XCTest https://github.com/typelift/SwiftCheck
  37. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Swift is inspired

    by Haskell • Functionality • Enum • Pattern match
  38. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Write the

    1st SwiftCheck
  39. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } }
  40. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } XCTest method
  41. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } Describe summary
  42. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } Random values
  43. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } Describe property
  44. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } }
  45. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } What input values are generated?
  46. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Print generated values

    [] [] [-2, -2] [-2] [2, -1, 4] [1, -4, -4] [-5, -4] ... *** Passed 100 tests
  47. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Print generated values

    [] [] [-2, -2] [-2] [2, -1, 4] [1, -4, -4] [-5, -4] ... *** Passed 100 tests Every time random
  48. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Print generated values

    [] [] [-2, -2] [-2] [2, -1, 4] [1, -4, -4] [-5, -4] ... *** Passed 100 tests Test is succeeded
  49. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. When failed ***

    Failed! Proposition: length not changed Falsifiable (after 1 test): [] *** Passed 0 tests
  50. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. When failed ***

    Failed! Proposition: length not changed Falsifiable (after 1 test): [] *** Passed 0 tests Summary
  51. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. When failed ***

    Failed! Proposition: length not changed Falsifiable (after 1 test): [] *** Passed 0 tests Failed value
  52. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2 words •

    Arbitrary • Shrinking
  53. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary

  54. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary • Protocol

    to generate random values • Adopt it can be used as a random value • Swift standard types are adopted
  55. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary Protocol public

    protocol Arbitrary { static var arbitrary : Gen<Self> { get } static func shrink(_ : Self) -> [Self] }
  56. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary Protocol public

    protocol Arbitrary { static var arbitrary : Gen<Self> { get } static func shrink(_ : Self) -> [Self] }
  57. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Custom type struct

    Point { var x: Int var y: Int } extension Point: Arbitrary { static var arbitrary: Gen<Point> { return Gen<(Int, Int)> .zip(Int.arbitrary, Int.arbitrary) .map(Point.init) } }
  58. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Custom type struct

    Point { var x: Int var y: Int } extension Point: Arbitrary { static var arbitrary: Gen<Point> { return Gen<(Int, Int)> .zip(Int.arbitrary, Int.arbitrary) .map(Point.init) } } Has pair of Int
  59. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Custom type struct

    Point { var x: Int var y: Int } extension Point: Arbitrary { static var arbitrary: Gen<Point> { return Gen<(Int, Int)> .zip(Int.arbitrary, Int.arbitrary) .map(Point.init) } } Return generater of Point
  60. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Custom type for

    _ in 0...4 { print(Point.arbitrary.generate) } // Point(x: -2, y: -26) // Point(x: -15, y: 2) // Point(x: 20, y: -21) // Point(x: -30, y: -15) // Point(x: -30, y: 6) Generated random Points
  61. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking

  62. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking • Get

    more small values • Reported a smaller failure case
  63. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary Protocol public

    protocol Arbitrary { static var arbitrary : Gen<Self> { get } static func shrink(_ : Self) -> [Self] }
  64. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary Protocol public

    protocol Arbitrary { static var arbitrary : Gen<Self> { get } static func shrink(_ : Self) -> [Self] }
  65. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking 10 10

    is failed
  66. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking 10 shrink

    5
  67. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking 10 still

    failed 5
  68. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking 10 5

    2 still failed
  69. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking 10 5

    2 1 still failed
  70. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking 10 passed

    ! 5 2 1 0
  71. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking 10 5

    2 1 0 reported !
  72. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Use cases

    in Action
  73. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Use cases •

    Randomized algorithm • Symmetric algorithm • Fast vs Slow
  74. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Randomized algorithm

  75. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: Maze generation

    • Example-based Test is difficult • Random number seeds can given as input values, but the algorithm can not be improved
  76. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: Maze generation

    • Property ⁃ Maze must always be solved
  77. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: Maze generation

    func canSolve(_ maze: Maze) -> Bool { // judge solve or not } property("must solve") <- forAll { (seed: UInt64) in let maze = Maze.generate(seed: seed) return canSolve(maze) }
  78. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: Maze generation

    func canSolve(_ maze: Maze) -> Bool { // judge solve or not } property("must solve") <- forAll { (seed: UInt64) in let maze = Maze.generate(seed: seed) return canSolve(maze) } generate seed
  79. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: Maze generation

    func canSolve(_ maze: Maze) -> Bool { // judge solve or not } property("must solve") <- forAll { (seed: UInt64) in let maze = Maze.generate(seed: seed) return canSolve(maze) } always solve
  80. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Symmetric algorithm

  81. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode /

    decode • Encoded and decoded matches the original value • The reverse is also the same
  82. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode /

    decode property("encode and decode are synmetry”) <- forAll { (bird: Bird) in let bytes = bird.encodeToBytes() return Bird.decode(bytes: bytes) == bird }
  83. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode /

    decode property("encode and decode are synmetry”) <- forAll { (bird: Bird) in let bytes = bird.encodeToBytes() return Bird.decode(bytes: bytes) == bird }
  84. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode /

    decode property("encode and decode are synmetry”) <- forAll { (bird: Bird) in let bytes = bird.encodeToBytes() return Bird.decode(bytes: bytes) == bird }
  85. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode /

    decode property("encode and decode are synmetry”) <- forAll { (bird: Bird) in let bytes = bird.encodeToBytes() return Bird.decode(bytes: bytes) == bird }
  86. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Fast vs

    Slow
  87. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Fast vs Slow

    • Fast algorithm is complicated to implement • The obvious algorithm is slow
  88. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Fast vs Slow

    • Both should be the same result • Use explicit (but slow) algorithms for fast algorithm (but complex) verification
  89. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Fast vs Slow

    property("same result") <- forAll { (xs: [Int]) in return slowSort(xs) == fastSort(xs) }
  90. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Fast vs Slow

    property("same result") <- forAll { (xs: [Int]) in return slowSort(xs) == fastSort(xs) }
  91. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Conclusion

  92. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based Test •

    Describe tests as property • Randomly generated input values • SwiftCheck is QuickCheck for Swift
  93. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Is the Example-based

    Test is Dead? • Does not replace the Example-based Test • Reduce anxiety about the miss of test patterns. • Combining them makes it a powerful card
  94. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Let’s try

    Property-based test with SwiftCheck
  95. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Appendix • https://en.wikipedia.org/wiki/Property_testing

    • https://github.com/typelift/SwiftCheck • https://www.objc.io/books/functional-swift/