Lin-Check: Testing concurrent data structures in Java

Writing concurrent programs is hard. However, testing them is not easy as well. In this talk, Nikita presents Lin-Check, a tool for testing concurrent data structures in JVM-based languages. Essentially, it takes some declarations of the data structure operations, generates random scenarios, runs them a lot of times, and verifies the corresponding results. During the talk, Nikita mostly focuses on the verification phase.

At first, he explains how LTS (Labeled Transition System) formalism can help us with implementing the verifier efficiently. After that, he extends LTS to support dual data structures; this formalism is useful for data structures with partial methods (synchronous queues, blocking queues, channels, semaphores, ...). However, such an extension is not sufficient, and some small tricks should be added, which restrict the model a bit but make it possible to use it in practice.

Transcript

1. Lincheck: Testing Concurrent Data Structures in Java Nikita Koval, Hydra 2019
2. This is a joint work with Dmitry Tsitelov, Maria Sokolova, Roman Elizarov, and Anton Evdokimov 2
3. Speaker: Nikita Koval • Graduated @ ITMO University • Previously worked as developer and research engineer @ Devexperts • Teaching concurrent programming course @ ITMO University • Researcher @ JetBrains • PhD student @ IST Austria 3 @nkoval_
4. Writing concurrent code is pain 4
5. Writing concurrent code is pain 5 … testing it is not much easier!
6. 6 var i = 0 i.inc() i.inc()
7. 7 var i = 0 i.inc() // 0 // 1 i.inc() // 1 // 0
8. 8 var i = 0 i.inc() // 0 i.inc() // 0
9. 9 We do not expect this! var i = 0 i.inc() // 0 i.inc() // 0
10. 10 Sequential model Concurrent model sequential specification on operations Linearizability (usually)
11. Execution is linearizable ⇔ ∃ equivalent sequential execution wrt happens-before order (a bit harder) 11
12. 12 val q = MSQueue<Int>() q.add(1) q.poll(): 2 q.poll(): 1 q.add(2) Execution is linearizable ⇔ ∃ equivalent sequential execution wrt happens-before order (a bit harder)
13. val q = MSQueue<Int>() q.add(1) q.poll(): 2 q.poll(): 1 q.add(2) 13 Execution is linearizable ⇔ ∃ equivalent sequential execution wrt happens-before order (a bit harder)
14. 14 var i = 0 i.inc() // 0 i.inc() // 0 This counter is not linearizable
15. How to check whether my data structure is linearizable? 15
16. How to check whether my data structure is linearizable? 16 Formal proofs
17. How to check whether my data structure is linearizable? 17 Formal proofs Model checking
18. How to check whether my data structure is linearizable? 18 Formal proofs Model checking Testing
19. How to check whether my data structure is linearizable? 19 Formal proofs Model checking Testing
20. How does the ideal test look?
21. How does the ideal test look? 21 class MSQueueTest { val q = MSQueue<Int>() } Initial state
22. How does the ideal test look? 22 class MSQueueTest { val q = MSQueue<Int>() @Operation fun add(element: Int) = q.add(element) @Operation fun poll() = q.poll() } Operations on the data structure
23. How does the ideal test look? 23 class MSQueueTest { val q = MSQueue<Int>() @Operation fun add(element: Int) = q.add(element) @Operation fun poll() = q.poll() } Operation parameters can be non-fixed!
24. How does the ideal test look? 24 class MSQueueTest { val q = MSQueue<Int>() @Operation fun add(element: Int) = q.add(element) @Operation fun poll() = q.poll() @Test fun runTest() = LinChecker.check(QueueTest::class) } The Magic Button
25. How does the ideal test look? 25 class MSQueueTest { val q = MSQueue<Int>() @Operation fun add(element: Int) = q.add(element) @Operation fun poll() = q.poll() @Test fun runTest() = LinChecker.check(QueueTest::class) } Do we have such instrument?
26. How does the ideal test look? 26 class MSQueueTest { val q = MSQueue<Int>() @Operation fun add(element: Int) = q.add(element) @Operation fun poll() = q.poll() @Test fun runTest() = LinChecker.check(QueueTest::class) } Do we have such instrument? YEEES!
27. Lin-Check Overview Lincheck = Linearizability Checker (supports not only linearizability) https://github.com/Kotlin/kotlinx-lincheck 27
28. Lincheck = Linearizability Checker (supports not only linearizability) https://github.com/Kotlin/kotlinx-lincheck 1. Generates a random scenario 2. Executes it a lot of times 3. Verifies the results 28 Lin-Check Overview
29. Lincheck = Linearizability Checker (supports not only linearizability) https://github.com/Kotlin/kotlinx-lincheck 1. Generates a random scenario 2. Executes it a lot of times 3. Verifies the results 29 ScenarioGenerator Runner Verifier Lin-Check Overview
30. Invalid Execution Example 30 Init part: [poll(): null, add(9)] Parallel part: | poll(): null | add(4) | | add(3) | add(6) | | poll(): 4 | poll(): 3 | Post part: [add(1), poll(): 6]
31. How to check results for correctness? Simplest solution: 1. Generate all possible sequential histories 2. Check whether one of them produces the same results 31
32. How to check results for correctness? Simplest solution: 1. Generate all possible sequential histories 2. Check whether one of them produces the same results 32 2 threads x 15 operations ⇒ OutOfMemoryError
33. How to check results for correctness? Simplest solution: 1. Generate all possible sequential histories 2. Check whether one of them produces the same results Smarter solution: Labeled Transition System (LTS) 33
34. 34 LTS (Labeled Transition System) inc(): 0 Initial state Operation with result dec(): 1 inc(): 1 dec(): 2 inc(): 2 dec(): 3 LTS is infinite
35. 35 LTS (Labeled Transition System) 0 1 2 3 inc(): 0 dec(): 1 inc(): 1 dec(): 2 inc(): 2 dec(): 3
36. 36 LTS-based verification val q = MSQueue<Int>() q.add(4) q.poll(): 9 q.poll(): 4 q.add(9)
37. 37 LTS-based verification 4 add(4) val q = MSQueue<Int>() q.add(4) q.poll(): 9 q.poll(): 4 q.add(9)
38. 38 LTS-based verification 4 add(4) poll(): 4 val q = MSQueue<Int>() q.add(4) q.poll(): 9 q.poll(): 4 q.add(9) Result is different
39. 39 LTS-based verification val q = MSQueue<Int>() q.add(4) q.poll(): 9 q.poll(): 4 q.add(9) 4 add(4) poll(): 4
40. 40 LTS-based verification 9 add(9) val q = MSQueue<Int>() q.add(4) q.poll(): 9 q.poll(): 4 q.add(9) 4 add(4) poll(): 4
41. 41 LTS-based verification poll(): 9 val q = MSQueue<Int>() q.add(4) q.poll(): 9 q.poll(): 4 q.add(9) 9 add(9) 4 add(4) poll(): 4
42. 42 LTS-based verification poll(): 9 val q = MSQueue<Int>() q.add(4) q.poll(): 9 q.poll(): 4 q.add(9) 9 add(9) 4 add(4) poll(): 4 A path is found ⇒ correct
43. Lazy LTS creation 43 • We build LTS lazilly, like on the previous slides • We use sequential implementation
44. Lazy LTS creation 44 • We build LTS lazilly, like on the previous slides • We use sequential implementation 4 add(4) poll(): 4
45. Lazy LTS creation 45 • We build LTS lazilly, like on the previous slides • We use sequential implementation • Equivalence via equals/hashcode implementations 4 add(4) poll(): 4 class MSQueueTest { val q = MSQueue<Int>() // Operations here override fun equals(other: Any?) = ... override fun hashCode() = ... }
46. Lazy LTS creation 46 • We build LTS lazilly, like on the previous slides • We use sequential implementation • Equivalence via equals/hashcode implementations 4 add(4) poll(): 4 class MSQueueTest: VerifierState() { val q = MSQueue<Int>() // Operations here override fun generateState() = q }
47. What if our data structure is blocking by design? 47
48. 48 val c = Channel<Int>() c.send(4) c.receive() // 4 send waits for receive and vice versa
49. Producer 1 val elem = ... c.send(elem) 49 Producer 2 val elem = ... c.send(elem) Consumer while(true) { val elem = c.receive() process(elem) } val c = Channel()
50. Producer 1 val elem = ... c.send(elem) 50 Producer 2 val elem = ... c.send(elem) Consumer while(true) { val elem = c.receive() process(elem) } val c = Channel() Has to wait for send 1
51. Producer 1 val elem = ... c.send(elem) 51 Producer 2 val elem = ... c.send(elem) Consumer while(true) { val elem = c.receive() process(elem) } val c = Channel() 1
52. Producer 1 val elem = ... c.send(elem) 52 Producer 2 val elem = ... c.send(elem) Consumer while(true) { val elem = c.receive() process(elem) } val c = Channel() 1
53. Producer 1 val elem = ... c.send(elem) 53 Producer 2 val elem = ... c.send(elem) Consumer while(true) { val elem = c.receive() process(elem) } val c = Channel() Rendezvous! 1 2
54. Producer 1 val elem = ... c.send(elem) 54 Producer 2 val elem = ... c.send(elem) Consumer while(true) { val elem = c.receive() process(elem) } val c = Channel() 1 2 3
55. Producer 1 val elem = ... c.send(elem) 55 Producer 2 val elem = ... c.send(elem) Consumer while(true) { val elem = c.receive() process(elem) } val c = Channel() 1 2 3
56. Producer 1 val elem = ... c.send(elem) 56 Producer 2 val elem = ... c.send(elem) Consumer while(true) { val elem = c.receive() process(elem) } val c = Channel() 1 2 3 4 Has to wait for receive
57. Producer 1 val elem = ... c.send(elem) 57 Producer 2 val elem = ... c.send(elem) Consumer while(true) { val elem = c.receive() process(elem) } val c = Channel() 1 2 3 4
58. Producer 1 val elem = ... c.send(elem) 58 Producer 2 val elem = ... c.send(elem) Consumer while(true) { val elem = c.receive() process(elem) } val c = Channel() 1 2 3 4 5 Has to wait for receive
59. 59 val c = Channel<Int>() c.send(4) c.receive() // 4 Non-linearizable because of suspension
60. 60 val c = Channel<Int>() c.send(4) c.receive(): // 4 register as a waiter suspend return element
61. 61 val c = Channel<Int>() c.send(4) c.receive(): // 4 register as a waiter suspend return element Dual Data Structures [1] [1] “Nonblocking Concurrent Data Structures with Condition Synchronization” by Scherer, W.N. and Scott, M.L. request follow-up
62. Dual Data Structures val c = Channel<Int>() c.receiveREQ(): tik c.send(4) c.receiveFUP(tik): 4 62 Unique ticket, ∈ℕ
63. val c = Channel<Int>() c.receiveREQ(): tik c.send(4) c.receiveFUP(tik): 4 Dual Data Structures 63 Follow-ups should be invoked after the corresponding requests
64. val c = Channel<Int>() c.receive(0): <s,1> c.send(0, 4) c.receive(1): <4,_> Dual Data Structures 64 Let’s always pass tickets, for simplicity
65. val c = Channel<Int>() c.receive(0): <s,1> c.send(0, 4) c.receive(1): <4,_> Dual Data Structures 65 suspended
66. 66 LTS for Dual Data Structures val c = Channel<Int>() c.receive(0): <s,1> c.send(0, 4) c.receive(1): <4,_>
67. 67 LTS for Dual Data Structures val c = Channel<Int>() c.receive(0): <s,1> c.send(0, 4) c.receive(1): <4,_> c.receive(0):<s,1>
68. 68 LTS for Dual Data Structures val c = Channel<Int>() c.receive(0): <s,1> c.send(0, 4) c.receive(1): <4,_> c.receive(0):<s,1> c.send(0,4) rt={1}
69. 69 LTS for Dual Data Structures val c = Channel<Int>() c.receive(0): <s,1> c.send(0, 4) c.receive(1): <4,_> c.receive(0):<s,1> c.send(0,4) rt={1} c.receive(1):<4,_>
70. 70 LTS for Dual Data Structures val c = Channel<Int>() c.receive(0): <s,1> c.send(0, 4) c.receive(1): <4,_> c.receive(0):<s,1> c.send(0,4) rt={1} c.receive(1):<4,_> Looks similar
71. 71 LTS for Dual Data Structures val c = Channel<Int>() c.send(0, 4): <s,1> c.send(0,4):<s,1>
72. 72 LTS for Dual Data Structures val c = Channel<Int>() c.send(0, 4): <s,1> c.send(0, 4): <s,2> c.receive(0): <4,_> c.send(1, 4) c.send(0,4):<s,1> c.send(0,4):<s,2> c.receive(0): <4,_> c.send(1,4)
73. 73 LTS for Dual Data Structures c.send(0,4):<s,1> c.send(0,4):<s,2> c.receive(0): <4,_> c.send(1,4) c.send(0,4):<s,2> The only difference is in tickets
74. 74 LTS for Dual Data Structures c.send(0,4):<s,1> c.send(0,4):<s,2> c.receive(0): <4,_> c.send(1,4) c.send(0,4):<s,2> The only difference is in tickets Let’s forbid such duplicate transitions
75. 75 LTS for Dual Data Structures c.send(0,4):<s,1> c.send(0,4):<s,2> c.receive(0): <4,_> c.send(1,4) rf={2→1} val c = Channel<Int>() c.send(0, 4): <s,1> c.send(0, 4): <s,2> c.receive(0): <4,_> c.send(1, 4)
76. 76 Verifier for Dual Data Structures val c = Channel<Int>() c.receive(): 4 c.receive(): s c.send(4): s+Unit
77. 77 Verifier for Dual Data Structures val c = Channel<Int>() c.receive(): 4 c.receive(): s c.send(4): s+Unit
78. 78 Verifier for Dual Data Structures val c = Channel<Int>() c.receive(): 4 c.receive(): s c.send(4): s+Unit c.receive(0):<s,1> Results are different
79. 79 Verifier for Dual Data Structures val c = Channel<Int>() c.receive(): 4 c.receive(): s c.send(4): s+Unit c.receive(0):<s,1> c.send(0,4):<s,1> suspended, ticket 1
80. 80 Verifier for Dual Data Structures val c = Channel<Int>() c.receive(): 4 c.receive(): s c.send(4): s+Unit c.receive(0):<s,1> c.send(0,4):<s,1> suspended, ticket 1 resumed c.receive(0):4 rt={1}
81. 81 Verifier for Dual Data Structures val c = Channel<Int>() c.receive(): 4 c.receive(): s c.send(4): s+Unit c.receive(0):<s,1> c.send(0,4):<s,1> suspended, ticket 1 resumed c.receive(0):4 rt={1} c.receive(0):<s,2>
82. 82 Verifier for Dual Data Structures val c = Channel<Int>() c.receive(): 4 c.receive(): s c.send(4): s+Unit c.receive(0):<s,1> c.send(0,4):<s,1> suspended, ticket 1 resumed c.receive(0):4 rt={1} c.receive(0):<s,2> c.send(1,4)
83. 83 Lazy Dual Data Structures LTS creation val c = Channel<Int>() c.receive(0): <s,1> c.send(0, 4) c.receive(1): <4,_> c.send(0,4):<s,1> c.send(0,4):<s,2> c.receive(0): <4,_> c.send(1,4) rf={2→1}
84. 84 Lazy Dual Data Structures LTS creation val c = Channel<Int>() c.receive(0): <s,1> c.send(0, 4) c.receive(1): <4,_> c.send(0,4):<s,1> c.send(0,4):<s,2> c.receive(0): <4,_> c.send(1,4) rf={2→1} 4 4 4 4 Let’s define as an externally observable state as before
85. 85 Lazy Dual Data Structures LTS creation val c = Channel<Int>() c.receive(0): <s,1> c.send(0, 4) c.receive(1): <4,_> c.send(0,4):<s,1> c.send(0,4):<s,2> c.receive(0): <4,_> c.send(1,4) rf={2→1} 4 4 4 4 Let’s define as an externally observable state as before They should be different!
86. st=[c.send(0,4):<s,1>, c.send(0,4):<s,2>] st=[c.send(0,4):<s,1>] st=[c.send(0,4):<s,2>] rt={1 by c.receive(0)} 86 Lazy Dual Data Structures LTS creation c.send(0,4):<s,1> c.send(0,4):<s,2> c.receive(0): <4,_> c.send(1,4) rf={2→1} 4 4 4 4 st = list of suspended operations rt = set of resumed operations
87. 87 Lazy Dual Data Structures LTS creation c.send(0,4):<s,1> c.send(0,4):<s,2> c.receive(0): <4,_> c.send(1,4) rf={2→1} 4 4 4 4 States are equal iff ∃f:ℕ→ℕ that 1. externally observable states 2. st-s wrt rf on tickets (as lists) 3. rt-s wrt rf on tickets (as sets) are equal st=[c.send(0,4):<s,1>, c.send(0,4):<s,2>] st=[c.send(0,4):<s,1>] st=[c.send(0,4):<s,2>] rt={1 by c.receive(0)}
88. 88 Lazy Dual Data Structures LTS creation c.send(0,4):<s,1> c.send(0,4):<s,2> c.receive(0): <4,_> c.send(1,4) rf={2→1} 4 4 4 4 States are equal iff ∃f:ℕ→ℕ that 1. externally observable states 2. st-s wrt rf on tickets (as lists) 3. rt-s wrt rf on tickets (as sets) are equal defined via equals/hashcode maintained by Lin-Check st=[c.send(0,4):<s,1>, c.send(0,4):<s,2>] st=[c.send(0,4):<s,1>] st=[c.send(0,4):<s,2>] rt={1 by c.receive(0)}
89. 89 Channel Test Example class RendezvousChannelTest: LinCheckState() { val c = Channel() @Operation suspend fun send(x: Int) = c.send(x) @Operation suspend fun receive(): Int = c.receive() override fun generateState() = Unit }
90. 90 Channel Test Example class BufferedChannelTest: LinCheckState() { val c = Channel() @Operation suspend fun send(x: Int) = c.send(x) @Operation suspend fun receive(): Int = c.receive() override fun generateState(): Any { val state = ArrayList<Int>() var x: Int? while(true) { x = c.poll() if (x == null) break state += x } return state } }
91. Uncovered topics • Verifiers for several relaxed contracts • How to run scenarios in the most “dangerous” way • API 91
92. Future plans • Smart running strategies • Supporting randomized relaxed contracts 92
93. Questions? https://github.com/Kotlin/kotlinx-lincheck 93