Slide 1
Slide 1 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Property-based test
beginning with SwiftCheck
Yusuke Hosonuma
@DeNA
try! Swift 2019 TOKYO / Day 2
Slide 2
Slide 2 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Yusuke Hosonuma
@DeNA
Favorite
Language
Swift / Go / Haskell
Twitter: @tobi462
Slide 3
Slide 3 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
SoftWare Engineer in Test
Test Automation, CI/CD and more…
Slide 4
Slide 4 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Organizer of the Events
iOSɹɹAndroidɹɹCI/CD
https://testnight.connpass.com/
Slide 5
Slide 5 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Private Developer Community
Pengin-mura
Slide 6
Slide 6 text
Agenda
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Slide 7
Slide 7 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Agenda
• What’s Property-based test ?
• What’s SwiftCheck ?
• Use cases in Action
• Conclusion
Slide 8
Slide 8 text
2VJDL
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
What’s
Property-based test
Slide 9
Slide 9 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Property-based test
• Describe tests as property
• Randomly generated input values
Slide 10
Slide 10 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Property-based test
• Describe tests as property
• Randomly generated input values
Slide 11
Slide 11 text
2VJDL
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Example-based test
Slide 12
Slide 12 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Example-based test
• Give input and expected values
• Pass the input values to the function
• Check that the output result matches
the expected value
Slide 13
Slide 13 text
2VJDL
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Array#reversed
Slide 14
Slide 14 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Array#reversed
1 2 3 4 5
5 4 3 2 1
Slide 15
Slide 15 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Array#reversed
func testReversed() {
let xs = [1, 2, 3, 4, 5]
XCTAssertEqual([5, 4, 3, 2, 1], xs.reversed())
}
Slide 16
Slide 16 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Array#reversed
func testReversed() {
let xs = [1, 2, 3, 4, 5]
XCTAssertEqual([5, 4, 3, 2, 1], xs.reversed())
}
Slide 17
Slide 17 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Array#reversed
func testReversed() {
let xs = [1, 2, 3, 4, 5]
XCTAssertEqual([5, 4, 3, 2, 1], xs.reversed())
}
Slide 18
Slide 18 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Array#reversed
func testReversed() {
let xs = [1, 2, 3, 4, 5]
XCTAssertEqual(xs.reversed(), [5, 4, 3, 2, 1])
}
How many patterns are necessary?
Slide 19
Slide 19 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Array#reversed
func testReversed() {
let empty = Array()
XCTAssertEqual(empty.reversed(), empty)
XCTAssertEqual( [1].reversed(), [1])
XCTAssertEqual( [1, 2].reversed(), [2, 1])
XCTAssertEqual([1, 2, 3, 4, 5].reversed(), [5, 4, 3, 2, 1])
...
}
Slide 20
Slide 20 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Array#reversed
func testReversed() {
let empty = Array()
XCTAssertEqual(empty.reversed(), empty)
XCTAssertEqual( [1].reversed(), [1])
XCTAssertEqual( [1, 2].reversed(), [2, 1])
XCTAssertEqual([1, 2, 3, 4, 5].reversed(), [5, 4, 3, 2, 1])
...
}
There are infinite possible input values!
Slide 21
Slide 21 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
How do we choose input values?
• A value that is likely to cause a bug
⁃ Boundary value
⁃ Equivalence partition
Slide 22
Slide 22 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
How do we choose input values?
• A value that is likely to cause a bug
⁃ Boundary value
⁃ Equivalence partition
Detect bugs when missed patterns
Slide 23
Slide 23 text
2VJDL
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Property-based test
Slide 24
Slide 24 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Think property
• Specific input values are not considered.
• What is the relationship between the
input value and the output value?
• What is property of function?
Slide 25
Slide 25 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
What’s property
1 2 3 4 5
5 4 3 2 1
Slide 26
Slide 26 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
The length does not change
1 2 3
1 1 2 … 9
3 2 1
1 9 … 2 1
Slide 27
Slide 27 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
reverse x2 = original
1 2 3 4 5
1 2 3
5 4 3 2 1
3 2 1
1 2 3 4 5
1 2 3
Slide 28
Slide 28 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Property of #reversed
• The length does not change
• When it is executed twice, it returns to
the original
Slide 29
Slide 29 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Property of #reversed
• The length does not change
• When it is executed twice, it returns to
the original
This is a common property
to any input values
Slide 30
Slide 30 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Property-based test by self
for xs in randomArrays() {
// The length does not change
XCTAssertEqual(xs.reversed().count, xs.count)
// When it is executed twice, it returns to the original
XCTAssertEqual(xs.reversed().reversed(), xs)
}
func randomArrays() -> [[Int]] {
// generate random arrays
}
Slide 31
Slide 31 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Property-based test by self
for xs in randomArrays() {
// The length does not change
XCTAssertEqual(xs.reversed().count, xs.count)
// When it is executed twice, it returns to the original
XCTAssertEqual(xs.reversed().reversed(), xs)
}
func randomArrays() -> [[Int]] {
// generate random arrays
}
Slide 32
Slide 32 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Property-based test by self
for xs in randomArrays() {
// The length does not change
XCTAssertEqual(xs.reversed().count, xs.count)
// When it is executed twice, it returns to the original
XCTAssertEqual(xs.reversed().reversed(), xs)
}
func randomArrays() -> [[Int]] {
// generate random arrays
}
Slide 33
Slide 33 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Property-based test by self
for xs in randomArrays() {
// The length does not change
XCTAssertEqual(xs.reversed().count, xs.count)
// When it is executed twice, it returns to the original
XCTAssertEqual(xs.reversed().reversed(), xs)
}
func randomArrays() -> [[Int]] {
// generate random arrays
}
Slide 34
Slide 34 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Property-based test by self
for xs in randomArrays() {
// The length does not change
XCTAssertEqual(xs.reversed().count, xs.count)
// When it is executed twice, it returns to the original
XCTAssertEqual(xs.reversed().reversed(), xs)
}
func randomArrays() -> [[Int]] {
// generate random arrays
}
Let’s try SwiftCheck !
Slide 35
Slide 35 text
2VJDL
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
What’s SwiftCheck ?
Slide 36
Slide 36 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
SwiftCheck
• Property-based test in Swift (OSS)
• Inspired by QuickCheck (in Haskell)
• with XCTest
https://github.com/typelift/SwiftCheck
Slide 37
Slide 37 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Swift is inspired by Haskell
• Functionality
• Enum
• Pattern match
Slide 38
Slide 38 text
2VJDL
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Write the 1st SwiftCheck
Slide 39
Slide 39 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
The length does not change
func testReversed() {
property("length not changed”) <- forAll { (xs: [Int]) in
return xs.reversed().count == xs.count
}
}
Slide 40
Slide 40 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
The length does not change
func testReversed() {
property("length not changed”) <- forAll { (xs: [Int]) in
return xs.reversed().count == xs.count
}
}
XCTest method
Slide 41
Slide 41 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
The length does not change
func testReversed() {
property("length not changed”) <- forAll { (xs: [Int]) in
return xs.reversed().count == xs.count
}
}
Describe summary
Slide 42
Slide 42 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
The length does not change
func testReversed() {
property("length not changed”) <- forAll { (xs: [Int]) in
return xs.reversed().count == xs.count
}
}
Random values
Slide 43
Slide 43 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
The length does not change
func testReversed() {
property("length not changed”) <- forAll { (xs: [Int]) in
return xs.reversed().count == xs.count
}
}
Describe property
Slide 44
Slide 44 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
The length does not change
func testReversed() {
property("length not changed”) <- forAll { (xs: [Int]) in
return xs.reversed().count == xs.count
}
}
Slide 45
Slide 45 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
The length does not change
func testReversed() {
property("length not changed”) <- forAll { (xs: [Int]) in
return xs.reversed().count == xs.count
}
}
What input values are generated?
Slide 46
Slide 46 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Print generated values
[]
[]
[-2, -2]
[-2]
[2, -1, 4]
[1, -4, -4]
[-5, -4]
...
*** Passed 100 tests
Slide 47
Slide 47 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Print generated values
[]
[]
[-2, -2]
[-2]
[2, -1, 4]
[1, -4, -4]
[-5, -4]
...
*** Passed 100 tests
Every time random
Slide 48
Slide 48 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Print generated values
[]
[]
[-2, -2]
[-2]
[2, -1, 4]
[1, -4, -4]
[-5, -4]
...
*** Passed 100 tests
Test is succeeded
Slide 49
Slide 49 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
When failed
*** Failed! Proposition: length not changed
Falsifiable (after 1 test):
[]
*** Passed 0 tests
Slide 50
Slide 50 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
When failed
*** Failed! Proposition: length not changed
Falsifiable (after 1 test):
[]
*** Passed 0 tests
Summary
Slide 51
Slide 51 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
When failed
*** Failed! Proposition: length not changed
Falsifiable (after 1 test):
[]
*** Passed 0 tests
Failed value
Slide 52
Slide 52 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
2 words
• Arbitrary
• Shrinking
Slide 53
Slide 53 text
2VJDL
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Arbitrary
Slide 54
Slide 54 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Arbitrary
• Protocol to generate random values
• Adopt it can be used as a random value
• Swift standard types are adopted
Slide 55
Slide 55 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Arbitrary Protocol
public protocol Arbitrary {
static var arbitrary : Gen { get }
static func shrink(_ : Self) -> [Self]
}
Slide 56
Slide 56 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Arbitrary Protocol
public protocol Arbitrary {
static var arbitrary : Gen { get }
static func shrink(_ : Self) -> [Self]
}
Slide 57
Slide 57 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Custom type
struct Point {
var x: Int
var y: Int
}
extension Point: Arbitrary {
static var arbitrary: Gen {
return Gen<(Int, Int)>
.zip(Int.arbitrary, Int.arbitrary)
.map(Point.init)
}
}
Slide 58
Slide 58 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Custom type
struct Point {
var x: Int
var y: Int
}
extension Point: Arbitrary {
static var arbitrary: Gen {
return Gen<(Int, Int)>
.zip(Int.arbitrary, Int.arbitrary)
.map(Point.init)
}
}
Has pair of Int
Slide 59
Slide 59 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Custom type
struct Point {
var x: Int
var y: Int
}
extension Point: Arbitrary {
static var arbitrary: Gen {
return Gen<(Int, Int)>
.zip(Int.arbitrary, Int.arbitrary)
.map(Point.init)
}
}
Return generater of Point
Slide 60
Slide 60 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Custom type
for _ in 0...4 {
print(Point.arbitrary.generate)
}
// Point(x: -2, y: -26)
// Point(x: -15, y: 2)
// Point(x: 20, y: -21)
// Point(x: -30, y: -15)
// Point(x: -30, y: 6)
Generated random Points
Slide 61
Slide 61 text
2VJDL
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Shrinking
Slide 62
Slide 62 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Shrinking
• Get more small values
• Reported a smaller failure case
Slide 63
Slide 63 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Arbitrary Protocol
public protocol Arbitrary {
static var arbitrary : Gen { get }
static func shrink(_ : Self) -> [Self]
}
Slide 64
Slide 64 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Arbitrary Protocol
public protocol Arbitrary {
static var arbitrary : Gen { get }
static func shrink(_ : Self) -> [Self]
}
Slide 65
Slide 65 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Shrinking
10
10 is failed
Slide 66
Slide 66 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Shrinking
10
shrink
5
Slide 67
Slide 67 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Shrinking
10
still failed
5
Slide 68
Slide 68 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Shrinking
10 5 2
still failed
Slide 69
Slide 69 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Shrinking
10 5 2 1
still failed
Slide 70
Slide 70 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Shrinking
10
passed !
5 2 1 0
Slide 71
Slide 71 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Shrinking
10 5 2 1 0
reported !
Slide 72
Slide 72 text
2VJDL
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Use cases in Action
Slide 73
Slide 73 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Use cases
• Randomized algorithm
• Symmetric algorithm
• Fast vs Slow
Slide 74
Slide 74 text
2VJDL
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Randomized algorithm
Slide 75
Slide 75 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Example: Maze generation
• Example-based Test is difficult
• Random number seeds can given as
input values, but the algorithm can not
be improved
Slide 76
Slide 76 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Example: Maze generation
• Property
⁃ Maze must always be solved
Slide 77
Slide 77 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Example: Maze generation
func canSolve(_ maze: Maze) -> Bool {
// judge solve or not
}
property("must solve") <- forAll { (seed: UInt64) in
let maze = Maze.generate(seed: seed)
return canSolve(maze)
}
Slide 78
Slide 78 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Example: Maze generation
func canSolve(_ maze: Maze) -> Bool {
// judge solve or not
}
property("must solve") <- forAll { (seed: UInt64) in
let maze = Maze.generate(seed: seed)
return canSolve(maze)
}
generate seed
Slide 79
Slide 79 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Example: Maze generation
func canSolve(_ maze: Maze) -> Bool {
// judge solve or not
}
property("must solve") <- forAll { (seed: UInt64) in
let maze = Maze.generate(seed: seed)
return canSolve(maze)
}
always solve
Slide 80
Slide 80 text
2VJDL
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Symmetric algorithm
Slide 81
Slide 81 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Example: encode / decode
• Encoded and decoded matches the
original value
• The reverse is also the same
Slide 82
Slide 82 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Example: encode / decode
property("encode and decode are synmetry”)
<- forAll { (bird: Bird) in
let bytes = bird.encodeToBytes()
return Bird.decode(bytes: bytes) == bird
}
Slide 83
Slide 83 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Example: encode / decode
property("encode and decode are synmetry”)
<- forAll { (bird: Bird) in
let bytes = bird.encodeToBytes()
return Bird.decode(bytes: bytes) == bird
}
Slide 84
Slide 84 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Example: encode / decode
property("encode and decode are synmetry”)
<- forAll { (bird: Bird) in
let bytes = bird.encodeToBytes()
return Bird.decode(bytes: bytes) == bird
}
Slide 85
Slide 85 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Example: encode / decode
property("encode and decode are synmetry”)
<- forAll { (bird: Bird) in
let bytes = bird.encodeToBytes()
return Bird.decode(bytes: bytes) == bird
}
Slide 86
Slide 86 text
2VJDL
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Fast vs Slow
Slide 87
Slide 87 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Fast vs Slow
• Fast algorithm is complicated to implement
• The obvious algorithm is slow
Slide 88
Slide 88 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Fast vs Slow
• Both should be the same result
• Use explicit (but slow) algorithms for fast
algorithm (but complex) verification
Slide 89
Slide 89 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Fast vs Slow
property("same result") <- forAll { (xs: [Int]) in
return slowSort(xs) == fastSort(xs)
}
Slide 90
Slide 90 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Fast vs Slow
property("same result") <- forAll { (xs: [Int]) in
return slowSort(xs) == fastSort(xs)
}
Slide 91
Slide 91 text
2VJDL
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Conclusion
Slide 92
Slide 92 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Property-based Test
• Describe tests as property
• Randomly generated input values
• SwiftCheck is QuickCheck for Swift
Slide 93
Slide 93 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Is the Example-based Test is Dead?
• Does not replace the Example-based Test
• Reduce anxiety about the miss of test
patterns.
• Combining them makes it a powerful card
Slide 94
Slide 94 text
2VJDL
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Let’s try Property-based test
with SwiftCheck
Slide 95
Slide 95 text
Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
Appendix
• https://en.wikipedia.org/wiki/Property_testing
• https://github.com/typelift/SwiftCheck
• https://www.objc.io/books/functional-swift/