Minimizing Faulty Executions of Distributed Systems

Talk given at NSDI '16, RICON, Microsoft Research, Google, Salesforce, Cornell, UW, UC Berkeley, USC.

Transcript

1. Minimizing Faulty Executions of Distributed Systems Colin Scott, Aurojit Panda, Vjekoslav Brajkovic, George Necula, Arvind Krishnamurthy, Scott Shenker
2. Software Developer
3. (GBs) Software Developer
4. Node1 ? … Node2 Node3 Node4 Node5 Node6 Node7 Node8 Node9 Node10 Node11 Node12 … Software Developer
5. 1 LaToza, Venolia, DeLine, ICSE’ 06 49% of developers’ time spent on debugging!1
6. 1 LaToza, Venolia, DeLine, ICSE’ 06 49% of developers’ time spent on debugging!1 Understanding How Bug Is Triggered Fixing Problematic Code
7. Our Goal Allow Developers To Focus on Fixing the Underlying Bug
8. Problem Statement Identify a minimal causal sequence of events that triggers the bug
9. Why Minimization? Smaller event traces are easier to understand G. A. Miller. The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information. Psychological Review ’56.
10. Outline Introduction Background Node 1 Node N Test Coordinator QA Testbed Software Under Test Fuzz Testing w/ DEMi S2 S3 S1 S3 Computational Model Minimization Evaluation Conclusion
11. Outline Introduction Background Node 1 Node N Test Coordinator QA Testbed Software Under Test Fuzz Testing w/ DEMi S2 S3 S1 S3 Computational Model Minimization Evaluation Conclusion
12. Outline Introduction Background Node 1 Node N Test Coordinator QA Testbed Software Under Test Fuzz Testing w/ DEMi S2 S3 S1 S3 Computational Model Minimization Evaluation Conclusion
13. Outline Introduction Background Node 1 Node N Test Coordinator QA Testbed Software Under Test Fuzz Testing w/ DEMi S2 S3 S1 S3 Computational Model Minimization Evaluation Conclusion
14. Outline Introduction Background Node 1 Node N Test Coordinator QA Testbed Software Under Test Fuzz Testing w/ DEMi S2 S3 S1 S3 Computational Model Minimization Evaluation Conclusion
15. Computational Model Distributed System: Collection of N processes Each process p: Has unbounded memory Starts in a known initial state Changes states deterministically a b c d e
16. Computational Model The network maintains a buffer of sent but not yet delivered messages a b c d e
17. Computational Model The network maintains a buffer of sent but not yet delivered messages a b c d e msg dst: d
18. Computational Model The network maintains a buffer of sent but not yet delivered messages a b c d e msg dst: d
19. Computational Model Message deliveries occur one at a time: destination enters a new state according to old state & message destination sends a finite set of messages to other processes* *May include timer messages to be delivered to itself later a b c d e msg dst: d
20. Computational Model Message deliveries occur one at a time: destination enters a new state according to old state & message destination sends a finite set of messages to other processes* *May include timer messages to be delivered to itself later a b c d e msg dst: d
21. Computational Model Message deliveries occur one at a time: destination enters a new state according to old state & message destination sends a finite set of messages to other processes* *May include timer messages to be delivered to itself later a b c d e timer dst: d msg dst: a
22. Computational Model Message deliveries occur one at a time: destination enters a new state according to old state & message destination sends a finite set of messages to other processes* *May include timer messages to be delivered to itself later a b c d e timer dst: d msg dst: a
23. Computational Model Message deliveries occur one at a time: destination enters a new state according to old state & message destination sends a finite set of messages to other processes* *May include timer messages to be delivered to itself later a b c d e timer dst: d msg dst: a
24. Computational Model Steps may also be external: External message is sent Process is created Process crash-recovers a b c d e timer dst: d msg dst: a
25. Computational Model Steps may also be external: External message is sent Process is created Process crash-recovers a b c d e timer dst: d msg dst: a msg dst: e
26. Computational Model Steps may also be external: External message is sent Process is created Process crash-recovers a b c d e timer dst: d msg dst: a msg dst: e
27. Computational Model A schedule τ is a sequence of events (either external or internal message deliveries) that can be applied in turn starting from the initial configuration. process start message delivery message delivery message delivery external message message delivery e1 i1 i2 i3 i4 e2
28. Invariant Checking An invariant is a predicate P over the state of all processes. a b c d e { ✔ ✗
29. Invariant Checking An invariant is a predicate P over the state of all processes. a b c d e { ✔ ✗ ✗ A faulty execution is one that ends in an invariant violation. e1 i1 i2 i3 i4 e2
30. Formal Problem Statement Find: locally minimal reproducing sequence τ’: τ’ violates P, |τ’| ≤ |τ| τ’ contains a subsequence of the external events of τ if we remove any external event e from τ’, ¬∃ τ’’ containing same external events - e, s.t. τ’’ violates P Given: schedule τ that results in violation of P
31. Formal Problem Statement After finding τ’: remove extraneous message deliveries from τ’
32. Outline Introduction Background Node 1 Node N Test Coordinator QA Testbed Software Under Test Fuzz Testing w/ DEMi S2 S3 S1 S3 Computational Model Minimization Evaluation Conclusion
33. Fuzz Testing with DEMi App RPC lib OS App RPC lib OS App RPC lib OS
34. Fuzz Testing with DEMi App RPC lib OS App RPC lib OS App RPC lib OS
35. Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ
36. Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ msg dst: b
37. Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ msg dst: b
38. Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ msg dst: b
39. Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ msg dst: b
40. Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ msg dst: b message delivery
41. Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ msg dst: b message delivery
42. Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ timer dst: b msg dst: a message delivery
43. Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ timer dst: b msg dst: a message delivery
44. Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ timer dst: b msg dst: a message delivery
45. Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ timer dst: b msg dst: a message delivery crash recovery
46. Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ timer dst: b msg dst: a message delivery crash recovery
47. Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ timer dst: b msg dst: a message delivery crash recovery
48. Outline Introduction Background Node 1 Node N Test Coordinator QA Testbed Software Under Test Fuzz Testing w/ DEMi S2 S3 S1 S3 Computational Model Minimization Evaluation Conclusion
49. Running Example: Raft Consensus a b c d
50. Running Example: Raft Consensus a b c d votes: {a,b,c}
51. Running Example: Raft Consensus a b c d client request
52. Running Example: Raft Consensus a b c d client request
53. Running Example: Raft Consensus a b c d client request client request client request client request
54. Running Example: Raft Consensus a b c d client request client request client request client request
55. Running Example: Raft Consensus a b c d client request client request client request ACK ACK ACK client request
56. Running Example: Raft Consensus a b c d client request client request client request client request
57. Running Example: Raft Consensus a b c d client request client request client request client request
58. Running Example: Raft Consensus a b c d client request client request client request commit commit commit client request
59. Running Example: Raft Consensus a b c d client request client request client request client request
60. RequestVote RequestVote RequestVote RequestVote VoteGranted VoteGranted VoteGranted VoteGranted
61. Minimization τ : Given … ✗ e1 i1 i2 i4 e2 en im
62. Minimization τ : Given Straightforward approach: Enumerate all schedules |τ’| ≤ |τ|, Pick shortest sequence that reproduces ✗ τ Schedule Space … ✗ e1 i1 i2 i4 e2 en im
63. Minimization τ : Given Straightforward approach: Enumerate all schedules |τ’| ≤ |τ|, Pick shortest sequence that reproduces ✗ τ Schedule Space … ✗ e1 i1 i2 i4 e2 en im
64. O(n!)
65. i2 i3 ↛i3 ↛i2 dst(i2) ≠ dst(i3) i3 i2 Observation #1: many schedules are commutative
66. Observation #1: many schedules are commutative i3 i2 Step n: i2 i3 ↛i3 ↛i2 dst(i2) ≠ dst(i3)
67. i3 i2 Step n: Step n+1: i2 i3 ↛i3 ↛i2 dst(i2) ≠ dst(i3) Observation #1: many schedules are commutative
68. i3 i2 i3 Step n: Step n+1: Step n+2: i2 i3 ↛i3 ↛i2 dst(i2) ≠ dst(i3) Observation #1: many schedules are commutative
69. i3 i2 i3 Step n: Step n+1: Step n+2: i2 i3 ↛i3 ↛i2 dst(i2) ≠ dst(i3) Observation #1: many schedules are commutative
70. i3 i2 i2 i3 Step n: Step n+1: Step n+2: i2 i3 ↛i3 ↛i2 dst(i2) ≠ dst(i3) Observation #1: many schedules are commutative
71. Observation #1: many schedules are commutative Adopt DPOR: Dynamic Partial Order Reduction C. Flanagan, P. Godefroid, “Dynamic Partial-Order Reduction for Model Checking Software”, POPL ‘05
72. O( !) n k
73. Approach: prioritize schedule space exploration
74. Approach: prioritize schedule space exploration Assume: fixed time budget Objective: quickly find small failing schedules
76. Given: Prioritization function
77. Given: Prioritization function Produce: Program under test Initial execution s.t. prioritization makes scant progress
78. Conjecture: Systems we care about exhibit program properties amenable with prioritization
80. {x=1,y=2} {x=1,y=3} {x=5,y=5} {x=2,y=2} {x=4,y=1} {x=-1,y=-2} {x=-1,y=-1}
81. {x=1,y=2} {x=1,y=3} {x=5,y=5} {x=4,y=1} {x=-1,y=-2} {x=-1,y=-1} {x=2,y=2}
82. {x=1,y=2} {x=1,y=3} {x=5,y=5} {x=4,y=1} {x=-1,y=-2} {x=-1,y=-1} {x=2,y=2} Invariant defined over small subset of processes’ variables
83. {x=1,y=2} {x=1,y=3} {x=5,y=5} {x=4,y=1} {x=-1,y=-2} {x=-1,y=-1} Each event affects a small subset of receiver’s variables {x=2,y=2} Invariant defined over small subset of processes’ variables
84. {x=1,y=2} {x=1,y=3} {x=5,y=5} {x=4,y=1} {x=-1,y=-2} {x=-1,y=-1} Initial execution contains events that don’t affect invariant {x=2,y=2} Each event affects a small subset of receiver’s variables Invariant defined over small subset of processes’ variables
85. Challenge: Don’t know which events are important Approach: experimentally “infer” important events stay close to the original execution
86. … ✗ e1 i1 i2 i4 e2 en im Observation #2: selectively mask original events τ :
87. … ✗ e1 i1 i2 i4 e2 en im Observation #2: selectively mask original events τ : e1 e2 en e3 e4 ext: e5
88. τ : en e3 ext: e5 e1 e2 e4 … ✗ e1 i1 i2 i4 e2 en im Observation #2: selectively mask original events
89. x τ : en e3 ext: e5 e1 e2 e4 … ✗ e1 i1 i2 i4 e2 en im Observation #2: selectively mask original events
90. x τ : en e3 ext: e5 e1 e2 e4 … ✗ e1 i1 i2 i4 e2 en im (Apply Delta Debugging1) 1A Zeller, R. Hildebrandt, “Simplifying and Isolating Failure-Inducing Input”, IEEE ‘02 Observation #2: selectively mask original events
91. τ : en e3 ext: e5 sub1: e1 e2 e4 … ✗ e1 i1 i2 i4 e2 en im e4 e5 en … (Apply Delta Debugging1) 1A Zeller, R. Hildebrandt, “Simplifying and Isolating Failure-Inducing Input”, IEEE ‘02 Observation #2: selectively mask original events
92. τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im en e5 e4 e1 e2 e3 foreach i in τ: if i is pending: deliver i # ignore unexpected … e5 e4 en Observation #2: selectively mask original events
93. τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im en e5 e4 e1 e2 e3 foreach i in τ: if i is pending: deliver i # ignore unexpected i1 … e5 e4 en Observation #2: selectively mask original events
94. τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im en e5 e4 e1 e2 e3 foreach i in τ: if i is pending: deliver i # ignore unexpected i1 … e5 e4 en Observation #2: selectively mask original events
95. τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im en e5 e4 e1 e2 e3 foreach i in τ: if i is pending: deliver i # ignore unexpected i1 i4 … e5 e4 en im Observation #2: selectively mask original events
96. τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im en e5 e4 e1 e2 e3 foreach i in τ: if i is pending: deliver i # ignore unexpected i1 i4 … e5 e4 en im Observation #2: selectively mask original events
97. τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im en e5 e4 e1 e2 e3 foreach i in τ: if i is pending: deliver i # ignore unexpected i1 i4 ✗ … e5 e4 en im Observation #2: selectively mask original events
98. τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im en e5 e4 e1 e2 e3 foreach i in τ: if i is pending: deliver i # ignore unexpected i1 i4 ✗ … e5 e4 en im Observation #2: selectively mask original events
99. τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im en e5 e4 i1 i4 ✗ … e5 e4 en im Observation #2: selectively mask original events
100. τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im en e5 e4 i1 i4 ✗ … e5 e4 en im Observation #2: selectively mask original events
101. τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im sub2: en e5 e4 i1 i4 ✗ … e5 e4 en im e5 en Observation #2: selectively mask original events
102. τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im sub2: i1 i4 … en e5 e4 i1 i4 ✗ … e5 e4 en im e5 en im Observation #2: selectively mask original events
103. τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im sub2: i1 i4 ✔ … en e5 e4 i1 i4 ✗ … e5 e4 en im e5 en im Observation #2: selectively mask original events
104. τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im sub2: i1 i4 ✔ … Explore backtrack points until (i) ✗ or (ii) time budget for sub2 expired en e5 e4 i1 i4 ✗ … e5 e4 en im e5 en im Observation #2: selectively mask original events
105. τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im sub2: … . . . i1 i4 ✔ … Explore backtrack points until (i) ✗ or (ii) time budget for sub2 expired en e5 e4 i1 i4 ✗ … e5 e4 en im e5 en im Observation #2: selectively mask original events
106. Message contents may differ across executions!
107. a b c d e msg dst: d type:t seq:3 src:a dst:d replicate: [1,2] type:t seq:5 src:a dst:d replicate: [1,2] msg dst: d Original message: Replay:
108. a b c d e msg dst: d Observation #3: some contents should be masked type:t seq:3 src:a dst:d replicate: [1,2] type:t seq:5 src:a dst:d replicate: [1,2] msg dst: d Original message: Replay:
109. Phase 1: choose initial schedule Match messages by user-defined “fingerprint” Observation #3: some contents should be masked
110. Phase 1: choose initial schedule Match messages by user-defined “fingerprint” Phase 2: prioritize backtrack points Match messages by type only Backtrack whenever multiple pending messages match by type Observation #3: some contents should be masked
111. Observation #4: shrink external message contents a b c d e type:bootstrap peers: [a,b,c,d,e] type:bootstrap peers: [a,b,c,d,e] type:bootstrap peers: [a,b,c,d,e]
112. Observation #4: shrink external message contents a b c d e type:bootstrap peers: [a,b,c,d,e] type:bootstrap peers: [a,b,c,d,e] type:bootstrap peers: [a,b,c,d,e]
113. Observation #4: shrink external message contents a b c d e type:bootstrap peers: [a,b,c,d,e] type:bootstrap peers: [a,b,c,d,e] type:bootstrap peers: [a,b,c,d,e]
114. Observation #4: shrink external message contents Observation #1: many schedules are commutative Approach: prioritize schedule space exploration Goal: find minimal schedule that produces violation Minimize internal events after externals minimized Observation #2: selectively mask original events Observation #3: some contents should be masked
115. Outline Introduction Background Node 1 Node N Test Coordinator QA Testbed Software Under Test Fuzz Testing w/ DEMi S2 S3 S1 S3 Computational Model Minimization Evaluation Conclusion
116. Target Systems
117. How well does DEMi work? Total Events 0 300 600 900 1200 1500 1800 2100 2400 2700 3000 Case Study raft-45 raft-46 raft-56 raft-58a raft-58b raft-42 raft-66 spark-2294 spark-3150 spark-9256 11 14 40 77 180 40 226 82 35 23 300 600 1000 400 1710 1500 2850 2380 1250 2160 Initial Execution After Minimization
118. How well does DEMi work? Total Events 0 300 600 900 1200 1500 1800 2100 2400 2700 3000 Case Study raft-45 raft-46 raft-56 raft-58a raft-58b raft-42 raft-66 spark-2294 spark-3150 spark-9256 11 14 40 77 180 40 226 82 35 23 300 600 1000 400 1710 1500 2850 2380 1250 2160 Initial Execution After Minimization Found w/ Fuzz Testing!
119. How well does DEMi work? Total Events 0 300 600 900 1200 1500 1800 2100 2400 2700 3000 Case Study raft-45 raft-46 raft-56 raft-58a raft-58b raft-42 raft-66 spark-2294 spark-3150 spark-9256 11 14 40 77 180 40 226 82 35 23 300 600 1000 400 1710 1500 2850 2380 1250 2160 Initial Execution After Minimization 80% - 97% Reduction!
120. How well does DEMi work? Total Events 0 30 60 90 120 150 180 210 240 270 300 Case Study raft-45 raft-46 raft-56 raft-58a raft-58b raft-42 raft-66 spark-2294spark-3150spark-9256 11 11 25 29 39 28 51 21 23 22 11 14 40 77 180 40 226 82 35 23 After Minimization Smallest Manual Trace
121. How well does DEMi work? Total Events 0 30 60 90 120 150 180 210 240 270 300 Case Study raft-45 raft-46 raft-56 raft-58a raft-58b raft-42 raft-66 spark-2294spark-3150spark-9256 11 11 25 29 39 28 51 21 23 22 11 14 40 77 180 40 226 82 35 23 After Minimization Smallest Manual Trace Factor of 1x - 5x from hand-crafted
122. 69 170 How quickly does DEMi work? Runtime in Seconds 0 400 800 1200 1600 2000 2400 2800 3200 3600 4000 Case Study raft-45 raft-46 raft-56 raft-58a raft-58b raft-42 raft-66 spark-2294 spark-3150 spark-9256 210 245 427 348 10676 69 43482 2132 282 170 Overall Minimization (~12 hours) (~3 hours) (~35 minutes)
123. 69 170 How quickly does DEMi work? Runtime in Seconds 0 400 800 1200 1600 2000 2400 2800 3200 3600 4000 Case Study raft-45 raft-46 raft-56 raft-58a raft-58b raft-42 raft-66 spark-2294 spark-3150 spark-9256 210 245 427 348 10676 69 43482 2132 282 170 Overall Minimization <10 minutes except 3 cases (~12 hours) (~3 hours) (~35 minutes)
124. See the paper for… How we handle non-determinism Handling multithreaded processes Supporting other RPC libraries Sketch for minimizing production traces More in-depth evaluation Related work …
125. Conclusion Open source tool: github.com/NetSys/demi Read our paper! eecs.berkeley.edu/~rcs/research/nsdi_draft.pdf Optimistic that these techniques can be successfully applied more broadly
126. Past Work Internet Troubleshooting: NSDI ’10, SIGCOMM ‘12 SDN Troubleshooting: HotSDN ’13, PODC ’13, SIGCOMM ‘14 Middleboxes & Mobile Devices: SIGCOMM ’12, NSDI ’15 CAP for Networks: HotSDN ‘13
127. Dst’Sys & Networking
128. Dst’Sys & Networking 1Parkinson, VSTTE ‘10 Tools for dst’sys lag sequential tools by ~10 years1
129. SE PL 1Parkinson, VSTTE ‘10 Tools for dst’sys lag sequential tools by ~10 years1 Dst’Sys & Networking
130. SE PL Tools for dst’sys lag sequential tools by ~10 years1 Stable- Multithreading & PCT for dst’sys Routing convergence tradeoffs Testing & debugging async code (mobile,JS) Infer JS defer tags SAT/SMT solvers need systems techniques Test verified dst’sys Program properties for minimization ACID for SDN Synthesizing coordination HCI for configuration hell 1Parkinson, VSTTE ‘10 Dst’Sys & Networking
131. Conclusion Open source tool: github.com/NetSys/demi Read our paper! eecs.berkeley.edu/~rcs/research/nsdi_draft.pdf Optimistic that these techniques can be successfully applied more broadly Thanks for your time! Contact me! cs@cs.berkeley.edu
132. Attributions Inspiration for slide design: Jay Lorch’s IronFleet slides Graphic Icons: thenounproject.org logfile: mantisshrimpdesign magnifying glass: Ricardo Moreira disk: Anton Outkine hook: Seb Cornelius bug report: Lemon Liu devil: Mourad Mokrane Putin: Remi Mercier
133. Production Traces Model: feed partially ordered log into single machine DEMi Require: - Partial ordering of all message deliveries - All crash-recoveries logged to disk
134. Instrumentation Complexity
135. Related Work Thread Schedule Minimization •Isolating Failure-Inducing Thread Schedules. SIGSOFT ’02. •A Trace Simplification Technique for Effective Debugging of Concurrent Programs. FSE ’10. Program Flow Analysis. •Enabling Tracing of Long-Running Multithreaded Programs via Dynamic Execution Reduction. ISSTA ’07. •Toward Generating Reducible Replay Logs. PLDI ’11. Best-Effort Replay of Field Failures •A Technique for Enabling and Supporting Debugging of Field Failures. ICSE ’07. •Triage: Diagnosing Production Run Failures at the User’s Site. SOSP ’07.
136. DDmin in more detail
137. DDmin assumptions
138. Local vs. global minima
139. Minimization Pace
140. Dealing With Threads If you’re lucky: threads are largely independent (Spark) If you’re unlucky: key insight: A write to shared memory is equivalent to a message delivery Approach: •interpose on virtual memory, thread scheduler •pause a thread whenever it writes to shared memory / disk Cf. “Enabling Tracing Of Long-Running Multithreaded Programs Via Dynamic Execution Reduction”, ISSTA ‘07
141. Dealing With Non-Determinism Interpose on: - Timers - Random number generators - Unordered hash values - ID allocation Stop-gap: replay each schedule multiple times
142. Complete Results
143. Runtime Breakdown
144. Integrating with other RPC libs App RPC lib OS App RPC lib OS App RPC lib OS DEMi JVM