Property-based test beginning with SwiftCheck

Property-based test beginning with SwiftCheck

70d9714ea13fc1133803d61fb16e4160?s=128

Yusuke Hosonuma

March 22, 2019
Tweet

Transcript

  1. 1.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test beginning

    with SwiftCheck Yusuke Hosonuma @DeNA try! Swift 2019 TOKYO / Day 2
  2. 2.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Yusuke Hosonuma @DeNA

    Favorite Language Swift / Go / Haskell Twitter: @tobi462
  3. 4.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Organizer of the

    Events iOSɹɹAndroidɹɹCI/CD https://testnight.connpass.com/
  4. 7.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Agenda • What’s

    Property-based test ? • What’s SwiftCheck ? • Use cases in Action • Conclusion
  5. 9.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test •

    Describe tests as property • Randomly generated input values
  6. 10.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test •

    Describe tests as property • Randomly generated input values
  7. 12.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example-based test •

    Give input and expected values • Pass the input values to the function • Check that the output result matches the expected value
  8. 15.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed()

    { let xs = [1, 2, 3, 4, 5] XCTAssertEqual([5, 4, 3, 2, 1], xs.reversed()) }
  9. 16.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed()

    { let xs = [1, 2, 3, 4, 5] XCTAssertEqual([5, 4, 3, 2, 1], xs.reversed()) }
  10. 17.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed()

    { let xs = [1, 2, 3, 4, 5] XCTAssertEqual([5, 4, 3, 2, 1], xs.reversed()) }
  11. 18.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed()

    { let xs = [1, 2, 3, 4, 5] XCTAssertEqual(xs.reversed(), [5, 4, 3, 2, 1]) } How many patterns are necessary?
  12. 19.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed()

    { let empty = Array<Int>() XCTAssertEqual(empty.reversed(), empty) XCTAssertEqual( [1].reversed(), [1]) XCTAssertEqual( [1, 2].reversed(), [2, 1]) XCTAssertEqual([1, 2, 3, 4, 5].reversed(), [5, 4, 3, 2, 1]) ... }
  13. 20.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed()

    { let empty = Array<Int>() XCTAssertEqual(empty.reversed(), empty) XCTAssertEqual( [1].reversed(), [1]) XCTAssertEqual( [1, 2].reversed(), [2, 1]) XCTAssertEqual([1, 2, 3, 4, 5].reversed(), [5, 4, 3, 2, 1]) ... } There are infinite possible input values!
  14. 21.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. How do we

    choose input values? • A value that is likely to cause a bug ⁃ Boundary value ⁃ Equivalence partition
  15. 22.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. How do we

    choose input values? • A value that is likely to cause a bug ⁃ Boundary value ⁃ Equivalence partition Detect bugs when missed patterns
  16. 24.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Think property •

    Specific input values are not considered. • What is the relationship between the input value and the output value? • What is property of function?
  17. 26.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change 1 2 3 1 1 2 … 9 3 2 1 1 9 … 2 1
  18. 27.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. reverse x2 =

    original 1 2 3 4 5 1 2 3 5 4 3 2 1 3 2 1 1 2 3 4 5 1 2 3
  19. 28.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property of #reversed

    • The length does not change • When it is executed twice, it returns to the original
  20. 29.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property of #reversed

    • The length does not change • When it is executed twice, it returns to the original This is a common property to any input values
  21. 30.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by

    self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays }
  22. 31.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by

    self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays }
  23. 32.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by

    self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays }
  24. 33.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by

    self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays }
  25. 34.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by

    self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays } Let’s try SwiftCheck !
  26. 36.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. SwiftCheck • Property-based

    test in Swift (OSS) • Inspired by QuickCheck (in Haskell) • with XCTest https://github.com/typelift/SwiftCheck
  27. 37.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Swift is inspired

    by Haskell • Functionality • Enum • Pattern match
  28. 39.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } }
  29. 40.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } XCTest method
  30. 41.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } Describe summary
  31. 42.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } Random values
  32. 43.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } Describe property
  33. 44.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } }
  34. 45.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } What input values are generated?
  35. 46.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Print generated values

    [] [] [-2, -2] [-2] [2, -1, 4] [1, -4, -4] [-5, -4] ... *** Passed 100 tests
  36. 47.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Print generated values

    [] [] [-2, -2] [-2] [2, -1, 4] [1, -4, -4] [-5, -4] ... *** Passed 100 tests Every time random
  37. 48.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Print generated values

    [] [] [-2, -2] [-2] [2, -1, 4] [1, -4, -4] [-5, -4] ... *** Passed 100 tests Test is succeeded
  38. 49.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. When failed ***

    Failed! Proposition: length not changed Falsifiable (after 1 test): [] *** Passed 0 tests
  39. 50.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. When failed ***

    Failed! Proposition: length not changed Falsifiable (after 1 test): [] *** Passed 0 tests Summary
  40. 51.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. When failed ***

    Failed! Proposition: length not changed Falsifiable (after 1 test): [] *** Passed 0 tests Failed value
  41. 54.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary • Protocol

    to generate random values • Adopt it can be used as a random value • Swift standard types are adopted
  42. 55.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary Protocol public

    protocol Arbitrary { static var arbitrary : Gen<Self> { get } static func shrink(_ : Self) -> [Self] }
  43. 56.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary Protocol public

    protocol Arbitrary { static var arbitrary : Gen<Self> { get } static func shrink(_ : Self) -> [Self] }
  44. 57.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Custom type struct

    Point { var x: Int var y: Int } extension Point: Arbitrary { static var arbitrary: Gen<Point> { return Gen<(Int, Int)> .zip(Int.arbitrary, Int.arbitrary) .map(Point.init) } }
  45. 58.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Custom type struct

    Point { var x: Int var y: Int } extension Point: Arbitrary { static var arbitrary: Gen<Point> { return Gen<(Int, Int)> .zip(Int.arbitrary, Int.arbitrary) .map(Point.init) } } Has pair of Int
  46. 59.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Custom type struct

    Point { var x: Int var y: Int } extension Point: Arbitrary { static var arbitrary: Gen<Point> { return Gen<(Int, Int)> .zip(Int.arbitrary, Int.arbitrary) .map(Point.init) } } Return generater of Point
  47. 60.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Custom type for

    _ in 0...4 { print(Point.arbitrary.generate) } // Point(x: -2, y: -26) // Point(x: -15, y: 2) // Point(x: 20, y: -21) // Point(x: -30, y: -15) // Point(x: -30, y: 6) Generated random Points
  48. 62.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking • Get

    more small values • Reported a smaller failure case
  49. 63.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary Protocol public

    protocol Arbitrary { static var arbitrary : Gen<Self> { get } static func shrink(_ : Self) -> [Self] }
  50. 64.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary Protocol public

    protocol Arbitrary { static var arbitrary : Gen<Self> { get } static func shrink(_ : Self) -> [Self] }
  51. 73.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Use cases •

    Randomized algorithm • Symmetric algorithm • Fast vs Slow
  52. 75.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: Maze generation

    • Example-based Test is difficult • Random number seeds can given as input values, but the algorithm can not be improved
  53. 76.
  54. 77.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: Maze generation

    func canSolve(_ maze: Maze) -> Bool { // judge solve or not } property("must solve") <- forAll { (seed: UInt64) in let maze = Maze.generate(seed: seed) return canSolve(maze) }
  55. 78.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: Maze generation

    func canSolve(_ maze: Maze) -> Bool { // judge solve or not } property("must solve") <- forAll { (seed: UInt64) in let maze = Maze.generate(seed: seed) return canSolve(maze) } generate seed
  56. 79.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: Maze generation

    func canSolve(_ maze: Maze) -> Bool { // judge solve or not } property("must solve") <- forAll { (seed: UInt64) in let maze = Maze.generate(seed: seed) return canSolve(maze) } always solve
  57. 81.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode /

    decode • Encoded and decoded matches the original value • The reverse is also the same
  58. 82.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode /

    decode property("encode and decode are synmetry”) <- forAll { (bird: Bird) in let bytes = bird.encodeToBytes() return Bird.decode(bytes: bytes) == bird }
  59. 83.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode /

    decode property("encode and decode are synmetry”) <- forAll { (bird: Bird) in let bytes = bird.encodeToBytes() return Bird.decode(bytes: bytes) == bird }
  60. 84.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode /

    decode property("encode and decode are synmetry”) <- forAll { (bird: Bird) in let bytes = bird.encodeToBytes() return Bird.decode(bytes: bytes) == bird }
  61. 85.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode /

    decode property("encode and decode are synmetry”) <- forAll { (bird: Bird) in let bytes = bird.encodeToBytes() return Bird.decode(bytes: bytes) == bird }
  62. 87.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Fast vs Slow

    • Fast algorithm is complicated to implement • The obvious algorithm is slow
  63. 88.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Fast vs Slow

    • Both should be the same result • Use explicit (but slow) algorithms for fast algorithm (but complex) verification
  64. 89.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Fast vs Slow

    property("same result") <- forAll { (xs: [Int]) in return slowSort(xs) == fastSort(xs) }
  65. 90.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Fast vs Slow

    property("same result") <- forAll { (xs: [Int]) in return slowSort(xs) == fastSort(xs) }
  66. 92.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based Test •

    Describe tests as property • Randomly generated input values • SwiftCheck is QuickCheck for Swift
  67. 93.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Is the Example-based

    Test is Dead? • Does not replace the Example-based Test • Reduce anxiety about the miss of test patterns. • Combining them makes it a powerful card
  68. 95.

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Appendix • https://en.wikipedia.org/wiki/Property_testing

    • https://github.com/typelift/SwiftCheck • https://www.objc.io/books/functional-swift/