Tweet
Tweet

Slide 1

Slide 1 text

Minimizing Faulty Executions of Distributed Systems Colin Scott, Aurojit Panda, Vjekoslav Brajkovic, George Necula, Arvind Krishnamurthy, Scott Shenker

Slide 2

Slide 2 text

Software Developer

Slide 3

Slide 3 text

(GBs) Software Developer

Slide 4

Slide 4 text

Node1 ? … Node2 Node3 Node4 Node5 Node6 Node7 Node8 Node9 Node10 Node11 Node12 … Software Developer

Slide 5

Slide 5 text

1 LaToza, Venolia, DeLine, ICSE’ 06 49% of developers’ time spent on debugging!1

Slide 6

Slide 6 text

1 LaToza, Venolia, DeLine, ICSE’ 06 49% of developers’ time spent on debugging!1 Understanding How Bug Is Triggered Fixing Problematic Code

Slide 7

Slide 7 text

Our Goal Allow Developers To Focus on Fixing the Underlying Bug

Slide 8

Slide 8 text

Problem Statement Identify a minimal causal sequence of events that triggers the bug

Slide 9

Slide 9 text

Why Minimization? Smaller event traces are easier to understand G. A. Miller. The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information. Psychological Review ’56.

Slide 10

Slide 10 text

Outline Introduction Background Node 1 Node N Test Coordinator QA Testbed Software Under Test Fuzz Testing w/ DEMi S2 S3 S1 S3 Computational Model Minimization Evaluation Conclusion

Slide 11

Slide 11 text

Outline Introduction Background Node 1 Node N Test Coordinator QA Testbed Software Under Test Fuzz Testing w/ DEMi S2 S3 S1 S3 Computational Model Minimization Evaluation Conclusion

Slide 12

Slide 12 text

Outline Introduction Background Node 1 Node N Test Coordinator QA Testbed Software Under Test Fuzz Testing w/ DEMi S2 S3 S1 S3 Computational Model Minimization Evaluation Conclusion

Slide 13

Slide 13 text

Outline Introduction Background Node 1 Node N Test Coordinator QA Testbed Software Under Test Fuzz Testing w/ DEMi S2 S3 S1 S3 Computational Model Minimization Evaluation Conclusion

Slide 14

Slide 14 text

Outline Introduction Background Node 1 Node N Test Coordinator QA Testbed Software Under Test Fuzz Testing w/ DEMi S2 S3 S1 S3 Computational Model Minimization Evaluation Conclusion

Slide 15

Slide 15 text

Computational Model Distributed System: Collection of N processes Each process p: Has unbounded memory Starts in a known initial state Changes states deterministically a b c d e

Slide 16

Slide 16 text

Computational Model The network maintains a buffer of sent but not yet delivered messages a b c d e

Slide 17

Slide 17 text

Computational Model The network maintains a buffer of sent but not yet delivered messages a b c d e msg dst: d

Slide 18

Slide 18 text

Computational Model The network maintains a buffer of sent but not yet delivered messages a b c d e msg dst: d

Slide 19

Slide 19 text

Computational Model Message deliveries occur one at a time: destination enters a new state according to old state & message destination sends a finite set of messages to other processes* *May include timer messages to be delivered to itself later a b c d e msg dst: d

Slide 20

Slide 20 text

Computational Model Message deliveries occur one at a time: destination enters a new state according to old state & message destination sends a finite set of messages to other processes* *May include timer messages to be delivered to itself later a b c d e msg dst: d

Slide 21

Slide 21 text

Computational Model Message deliveries occur one at a time: destination enters a new state according to old state & message destination sends a finite set of messages to other processes* *May include timer messages to be delivered to itself later a b c d e timer dst: d msg dst: a

Slide 22

Slide 22 text

Computational Model Message deliveries occur one at a time: destination enters a new state according to old state & message destination sends a finite set of messages to other processes* *May include timer messages to be delivered to itself later a b c d e timer dst: d msg dst: a

Slide 23

Slide 23 text

Computational Model Message deliveries occur one at a time: destination enters a new state according to old state & message destination sends a finite set of messages to other processes* *May include timer messages to be delivered to itself later a b c d e timer dst: d msg dst: a

Slide 24

Slide 24 text

Computational Model Steps may also be external: External message is sent Process is created Process crash-recovers a b c d e timer dst: d msg dst: a

Slide 25

Slide 25 text

Computational Model Steps may also be external: External message is sent Process is created Process crash-recovers a b c d e timer dst: d msg dst: a msg dst: e

Slide 26

Slide 26 text

Computational Model Steps may also be external: External message is sent Process is created Process crash-recovers a b c d e timer dst: d msg dst: a msg dst: e

Slide 27

Slide 27 text

Computational Model A schedule τ is a sequence of events (either external or internal message deliveries) that can be applied in turn starting from the initial configuration. process start message delivery message delivery message delivery external message message delivery e1 i1 i2 i3 i4 e2

Slide 28

Slide 28 text

Invariant Checking An invariant is a predicate P over the state of all processes. a b c d e { ✔ ✗

Slide 29

Slide 29 text

Invariant Checking An invariant is a predicate P over the state of all processes. a b c d e { ✔ ✗ ✗ A faulty execution is one that ends in an invariant violation. e1 i1 i2 i3 i4 e2

Slide 30

Slide 30 text

Formal Problem Statement Find: locally minimal reproducing sequence τ’: τ’ violates P, |τ’| ≤ |τ| τ’ contains a subsequence of the external events of τ if we remove any external event e from τ’, ¬∃ τ’’ containing same external events - e, s.t. τ’’ violates P Given: schedule τ that results in violation of P

Slide 31

Slide 31 text

Formal Problem Statement After finding τ’: remove extraneous message deliveries from τ’

Slide 32

Slide 32 text

Outline Introduction Background Node 1 Node N Test Coordinator QA Testbed Software Under Test Fuzz Testing w/ DEMi S2 S3 S1 S3 Computational Model Minimization Evaluation Conclusion

Slide 33

Slide 33 text

Fuzz Testing with DEMi App RPC lib OS App RPC lib OS App RPC lib OS

Slide 34

Slide 34 text

Fuzz Testing with DEMi App RPC lib OS App RPC lib OS App RPC lib OS

Slide 35

Slide 35 text

Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ

Slide 36

Slide 36 text

Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ msg dst: b

Slide 37

Slide 37 text

Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ msg dst: b

Slide 38

Slide 38 text

Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ msg dst: b

Slide 39

Slide 39 text

Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ msg dst: b

Slide 40

Slide 40 text

Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ msg dst: b message delivery

Slide 41

Slide 41 text

Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ msg dst: b message delivery

Slide 42

Slide 42 text

Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ timer dst: b msg dst: a message delivery

Slide 43

Slide 43 text

Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ timer dst: b msg dst: a message delivery

Slide 44

Slide 44 text

Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ timer dst: b msg dst: a message delivery

Slide 45

Slide 45 text

Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ timer dst: b msg dst: a message delivery crash recovery

Slide 46

Slide 46 text

Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ timer dst: b msg dst: a message delivery crash recovery

Slide 47

Slide 47 text

Fuzz Testing with DEMi App RPC lib OS AspectJ App RPC lib OS AspectJ App RPC lib OS AspectJ timer dst: b msg dst: a message delivery crash recovery

Slide 48

Slide 48 text

Outline Introduction Background Node 1 Node N Test Coordinator QA Testbed Software Under Test Fuzz Testing w/ DEMi S2 S3 S1 S3 Computational Model Minimization Evaluation Conclusion

Slide 49

Slide 49 text

Running Example: Raft Consensus a b c d

Slide 50

Slide 50 text

Running Example: Raft Consensus a b c d votes: {a,b,c}

Slide 51

Slide 51 text

Running Example: Raft Consensus a b c d client request

Slide 52

Slide 52 text

Running Example: Raft Consensus a b c d client request

Slide 53

Slide 53 text

Running Example: Raft Consensus a b c d client request client request client request client request

Slide 54

Slide 54 text

Running Example: Raft Consensus a b c d client request client request client request client request

Slide 55

Slide 55 text

Running Example: Raft Consensus a b c d client request client request client request ACK ACK ACK client request

Slide 56

Slide 56 text

Running Example: Raft Consensus a b c d client request client request client request client request

Slide 57

Slide 57 text

Running Example: Raft Consensus a b c d client request client request client request client request

Slide 58

Slide 58 text

Running Example: Raft Consensus a b c d client request client request client request commit commit commit client request

Slide 59

Slide 59 text

Running Example: Raft Consensus a b c d client request client request client request client request

Slide 60

Slide 60 text

RequestVote RequestVote RequestVote RequestVote VoteGranted VoteGranted VoteGranted VoteGranted

Slide 61

Slide 61 text

Minimization τ : Given … ✗ e1 i1 i2 i4 e2 en im

Slide 62

Slide 62 text

Minimization τ : Given Straightforward approach: Enumerate all schedules |τ’| ≤ |τ|, Pick shortest sequence that reproduces ✗ τ Schedule Space … ✗ e1 i1 i2 i4 e2 en im

Slide 63

Slide 63 text

Minimization τ : Given Straightforward approach: Enumerate all schedules |τ’| ≤ |τ|, Pick shortest sequence that reproduces ✗ τ Schedule Space … ✗ e1 i1 i2 i4 e2 en im

Slide 64

Slide 64 text

O(n!)

Slide 65

Slide 65 text

i2 i3 ↛i3 ↛i2 dst(i2) ≠ dst(i3) i3 i2 Observation #1: many schedules are commutative

Slide 66

Slide 66 text

Observation #1: many schedules are commutative i3 i2 Step n: i2 i3 ↛i3 ↛i2 dst(i2) ≠ dst(i3)

Slide 67

Slide 67 text

i3 i2 Step n: Step n+1: i2 i3 ↛i3 ↛i2 dst(i2) ≠ dst(i3) Observation #1: many schedules are commutative

Slide 68

Slide 68 text

i3 i2 i3 Step n: Step n+1: Step n+2: i2 i3 ↛i3 ↛i2 dst(i2) ≠ dst(i3) Observation #1: many schedules are commutative

Slide 69

Slide 69 text

i3 i2 i3 Step n: Step n+1: Step n+2: i2 i3 ↛i3 ↛i2 dst(i2) ≠ dst(i3) Observation #1: many schedules are commutative

Slide 70

Slide 70 text

i3 i2 i2 i3 Step n: Step n+1: Step n+2: i2 i3 ↛i3 ↛i2 dst(i2) ≠ dst(i3) Observation #1: many schedules are commutative

Slide 71

Slide 71 text

Observation #1: many schedules are commutative Adopt DPOR: Dynamic Partial Order Reduction C. Flanagan, P. Godefroid, “Dynamic Partial-Order Reduction for Model Checking Software”, POPL ‘05

Slide 72

Slide 72 text

O( !) n k

Slide 73

Slide 73 text

Approach: prioritize schedule space exploration

Slide 74

Slide 74 text

Approach: prioritize schedule space exploration Assume: fixed time budget Objective: quickly find small failing schedules

Slide 75

Slide 75 text

No content

Slide 76

Slide 76 text

Given: Prioritization function

Slide 77

Slide 77 text

Given: Prioritization function Produce: Program under test Initial execution s.t. prioritization makes scant progress

Slide 78

Slide 78 text

Conjecture: Systems we care about exhibit program properties amenable with prioritization

Slide 79

Slide 79 text

No content

Slide 80

Slide 80 text

{x=1,y=2} {x=1,y=3} {x=5,y=5} {x=2,y=2} {x=4,y=1} {x=-1,y=-2} {x=-1,y=-1}

Slide 81

Slide 81 text

{x=1,y=2} {x=1,y=3} {x=5,y=5} {x=4,y=1} {x=-1,y=-2} {x=-1,y=-1} {x=2,y=2}

Slide 82

Slide 82 text

{x=1,y=2} {x=1,y=3} {x=5,y=5} {x=4,y=1} {x=-1,y=-2} {x=-1,y=-1} {x=2,y=2} Invariant defined over small subset of processes’ variables

Slide 83

Slide 83 text

{x=1,y=2} {x=1,y=3} {x=5,y=5} {x=4,y=1} {x=-1,y=-2} {x=-1,y=-1} Each event affects a small subset of receiver’s variables {x=2,y=2} Invariant defined over small subset of processes’ variables

Slide 84

Slide 84 text

{x=1,y=2} {x=1,y=3} {x=5,y=5} {x=4,y=1} {x=-1,y=-2} {x=-1,y=-1} Initial execution contains events that don’t affect invariant {x=2,y=2} Each event affects a small subset of receiver’s variables Invariant defined over small subset of processes’ variables

Slide 85

Slide 85 text

Challenge: Don’t know which events are important Approach: experimentally “infer” important events stay close to the original execution

Slide 86

Slide 86 text

… ✗ e1 i1 i2 i4 e2 en im Observation #2: selectively mask original events τ :

Slide 87

Slide 87 text

… ✗ e1 i1 i2 i4 e2 en im Observation #2: selectively mask original events τ : e1 e2 en e3 e4 ext: e5

Slide 88

Slide 88 text

τ : en e3 ext: e5 e1 e2 e4 … ✗ e1 i1 i2 i4 e2 en im Observation #2: selectively mask original events

Slide 89

Slide 89 text

x τ : en e3 ext: e5 e1 e2 e4 … ✗ e1 i1 i2 i4 e2 en im Observation #2: selectively mask original events

Slide 90

Slide 90 text

x τ : en e3 ext: e5 e1 e2 e4 … ✗ e1 i1 i2 i4 e2 en im (Apply Delta Debugging1) 1A Zeller, R. Hildebrandt, “Simplifying and Isolating Failure-Inducing Input”, IEEE ‘02 Observation #2: selectively mask original events

Slide 91

Slide 91 text

τ : en e3 ext: e5 sub1: e1 e2 e4 … ✗ e1 i1 i2 i4 e2 en im e4 e5 en … (Apply Delta Debugging1) 1A Zeller, R. Hildebrandt, “Simplifying and Isolating Failure-Inducing Input”, IEEE ‘02 Observation #2: selectively mask original events

Slide 92

Slide 92 text

τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im en e5 e4 e1 e2 e3 foreach i in τ: if i is pending: deliver i # ignore unexpected … e5 e4 en Observation #2: selectively mask original events

Slide 93

Slide 93 text

τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im en e5 e4 e1 e2 e3 foreach i in τ: if i is pending: deliver i # ignore unexpected i1 … e5 e4 en Observation #2: selectively mask original events

Slide 94

Slide 94 text

τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im en e5 e4 e1 e2 e3 foreach i in τ: if i is pending: deliver i # ignore unexpected i1 … e5 e4 en Observation #2: selectively mask original events

Slide 95

Slide 95 text

τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im en e5 e4 e1 e2 e3 foreach i in τ: if i is pending: deliver i # ignore unexpected i1 i4 … e5 e4 en im Observation #2: selectively mask original events

Slide 96

Slide 96 text

τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im en e5 e4 e1 e2 e3 foreach i in τ: if i is pending: deliver i # ignore unexpected i1 i4 … e5 e4 en im Observation #2: selectively mask original events

Slide 97

Slide 97 text

τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im en e5 e4 e1 e2 e3 foreach i in τ: if i is pending: deliver i # ignore unexpected i1 i4 ✗ … e5 e4 en im Observation #2: selectively mask original events

Slide 98

Slide 98 text

τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im en e5 e4 e1 e2 e3 foreach i in τ: if i is pending: deliver i # ignore unexpected i1 i4 ✗ … e5 e4 en im Observation #2: selectively mask original events

Slide 99

Slide 99 text

τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im en e5 e4 i1 i4 ✗ … e5 e4 en im Observation #2: selectively mask original events

Slide 100

Slide 100 text

τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im en e5 e4 i1 i4 ✗ … e5 e4 en im Observation #2: selectively mask original events

Slide 101

Slide 101 text

τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im sub2: en e5 e4 i1 i4 ✗ … e5 e4 en im e5 en Observation #2: selectively mask original events

Slide 102

Slide 102 text

τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im sub2: i1 i4 … en e5 e4 i1 i4 ✗ … e5 e4 en im e5 en im Observation #2: selectively mask original events

Slide 103

Slide 103 text

τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im sub2: i1 i4 ✔ … en e5 e4 i1 i4 ✗ … e5 e4 en im e5 en im Observation #2: selectively mask original events

Slide 104

Slide 104 text

τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im sub2: i1 i4 ✔ … Explore backtrack points until (i) ✗ or (ii) time budget for sub2 expired en e5 e4 i1 i4 ✗ … e5 e4 en im e5 en im Observation #2: selectively mask original events

Slide 105

Slide 105 text

τ : ext: sub1: … ✗ e1 i1 i2 i4 e2 en im sub2: … . . . i1 i4 ✔ … Explore backtrack points until (i) ✗ or (ii) time budget for sub2 expired en e5 e4 i1 i4 ✗ … e5 e4 en im e5 en im Observation #2: selectively mask original events

Slide 106

Slide 106 text

Message contents may differ across executions!

Slide 107

Slide 107 text

a b c d e msg dst: d type:t seq:3 src:a dst:d replicate: [1,2] type:t seq:5 src:a dst:d replicate: [1,2] msg dst: d Original message: Replay:

Slide 108

Slide 108 text

a b c d e msg dst: d Observation #3: some contents should be masked type:t seq:3 src:a dst:d replicate: [1,2] type:t seq:5 src:a dst:d replicate: [1,2] msg dst: d Original message: Replay:

Slide 109

Slide 109 text

Phase 1: choose initial schedule Match messages by user-defined “fingerprint” Observation #3: some contents should be masked

Slide 110

Slide 110 text

Phase 1: choose initial schedule Match messages by user-defined “fingerprint” Phase 2: prioritize backtrack points Match messages by type only Backtrack whenever multiple pending messages match by type Observation #3: some contents should be masked

Slide 111

Slide 111 text

Observation #4: shrink external message contents a b c d e type:bootstrap peers: [a,b,c,d,e] type:bootstrap peers: [a,b,c,d,e] type:bootstrap peers: [a,b,c,d,e]

Slide 112

Slide 112 text

Observation #4: shrink external message contents a b c d e type:bootstrap peers: [a,b,c,d,e] type:bootstrap peers: [a,b,c,d,e] type:bootstrap peers: [a,b,c,d,e]

Slide 113

Slide 113 text

Observation #4: shrink external message contents a b c d e type:bootstrap peers: [a,b,c,d,e] type:bootstrap peers: [a,b,c,d,e] type:bootstrap peers: [a,b,c,d,e]

Slide 114

Slide 114 text

Observation #4: shrink external message contents Observation #1: many schedules are commutative Approach: prioritize schedule space exploration Goal: find minimal schedule that produces violation Minimize internal events after externals minimized Observation #2: selectively mask original events Observation #3: some contents should be masked

Slide 115

Slide 115 text

Outline Introduction Background Node 1 Node N Test Coordinator QA Testbed Software Under Test Fuzz Testing w/ DEMi S2 S3 S1 S3 Computational Model Minimization Evaluation Conclusion

Slide 116

Slide 116 text

Target Systems

Slide 117

Slide 117 text

How well does DEMi work? Total Events 0 300 600 900 1200 1500 1800 2100 2400 2700 3000 Case Study raft-45 raft-46 raft-56 raft-58a raft-58b raft-42 raft-66 spark-2294 spark-3150 spark-9256 11 14 40 77 180 40 226 82 35 23 300 600 1000 400 1710 1500 2850 2380 1250 2160 Initial Execution After Minimization

Slide 118

Slide 118 text

How well does DEMi work? Total Events 0 300 600 900 1200 1500 1800 2100 2400 2700 3000 Case Study raft-45 raft-46 raft-56 raft-58a raft-58b raft-42 raft-66 spark-2294 spark-3150 spark-9256 11 14 40 77 180 40 226 82 35 23 300 600 1000 400 1710 1500 2850 2380 1250 2160 Initial Execution After Minimization Found w/ Fuzz Testing!

Slide 119

Slide 119 text

How well does DEMi work? Total Events 0 300 600 900 1200 1500 1800 2100 2400 2700 3000 Case Study raft-45 raft-46 raft-56 raft-58a raft-58b raft-42 raft-66 spark-2294 spark-3150 spark-9256 11 14 40 77 180 40 226 82 35 23 300 600 1000 400 1710 1500 2850 2380 1250 2160 Initial Execution After Minimization 80% - 97% Reduction!

Slide 120

Slide 120 text

How well does DEMi work? Total Events 0 30 60 90 120 150 180 210 240 270 300 Case Study raft-45 raft-46 raft-56 raft-58a raft-58b raft-42 raft-66 spark-2294spark-3150spark-9256 11 11 25 29 39 28 51 21 23 22 11 14 40 77 180 40 226 82 35 23 After Minimization Smallest Manual Trace

Slide 121

Slide 121 text

How well does DEMi work? Total Events 0 30 60 90 120 150 180 210 240 270 300 Case Study raft-45 raft-46 raft-56 raft-58a raft-58b raft-42 raft-66 spark-2294spark-3150spark-9256 11 11 25 29 39 28 51 21 23 22 11 14 40 77 180 40 226 82 35 23 After Minimization Smallest Manual Trace Factor of 1x - 5x from hand-crafted

Slide 122

Slide 122 text

69 170 How quickly does DEMi work? Runtime in Seconds 0 400 800 1200 1600 2000 2400 2800 3200 3600 4000 Case Study raft-45 raft-46 raft-56 raft-58a raft-58b raft-42 raft-66 spark-2294 spark-3150 spark-9256 210 245 427 348 10676 69 43482 2132 282 170 Overall Minimization (~12 hours) (~3 hours) (~35 minutes)

Slide 123

Slide 123 text

69 170 How quickly does DEMi work? Runtime in Seconds 0 400 800 1200 1600 2000 2400 2800 3200 3600 4000 Case Study raft-45 raft-46 raft-56 raft-58a raft-58b raft-42 raft-66 spark-2294 spark-3150 spark-9256 210 245 427 348 10676 69 43482 2132 282 170 Overall Minimization <10 minutes except 3 cases (~12 hours) (~3 hours) (~35 minutes)

Slide 124

Slide 124 text

See the paper for… How we handle non-determinism Handling multithreaded processes Supporting other RPC libraries Sketch for minimizing production traces More in-depth evaluation Related work …

Slide 125

Slide 125 text

Conclusion Open source tool: github.com/NetSys/demi Read our paper! eecs.berkeley.edu/~rcs/research/nsdi_draft.pdf Optimistic that these techniques can be successfully applied more broadly

Slide 126

Slide 126 text

Past Work Internet Troubleshooting: NSDI ’10, SIGCOMM ‘12 SDN Troubleshooting: HotSDN ’13, PODC ’13, SIGCOMM ‘14 Middleboxes & Mobile Devices: SIGCOMM ’12, NSDI ’15 CAP for Networks: HotSDN ‘13

Slide 127

Slide 127 text

Dst’Sys & Networking

Slide 128

Slide 128 text

Dst’Sys & Networking 1Parkinson, VSTTE ‘10 Tools for dst’sys lag sequential tools by ~10 years1

Slide 129

Slide 129 text

SE PL 1Parkinson, VSTTE ‘10 Tools for dst’sys lag sequential tools by ~10 years1 Dst’Sys & Networking

Slide 130

Slide 130 text

SE PL Tools for dst’sys lag sequential tools by ~10 years1 Stable- Multithreading & PCT for dst’sys Routing convergence tradeoffs Testing & debugging async code (mobile,JS) Infer JS defer tags SAT/SMT solvers need systems techniques Test verified dst’sys Program properties for minimization ACID for SDN Synthesizing coordination HCI for configuration hell 1Parkinson, VSTTE ‘10 Dst’Sys & Networking

Slide 131

Slide 131 text

Conclusion Open source tool: github.com/NetSys/demi Read our paper! eecs.berkeley.edu/~rcs/research/nsdi_draft.pdf Optimistic that these techniques can be successfully applied more broadly Thanks for your time! Contact me! [email protected]

Slide 132

Slide 132 text

Attributions Inspiration for slide design: Jay Lorch’s IronFleet slides Graphic Icons: thenounproject.org logfile: mantisshrimpdesign magnifying glass: Ricardo Moreira disk: Anton Outkine hook: Seb Cornelius bug report: Lemon Liu devil: Mourad Mokrane Putin: Remi Mercier

Slide 133

Slide 133 text

Production Traces Model: feed partially ordered log into single machine DEMi Require: - Partial ordering of all message deliveries - All crash-recoveries logged to disk

Slide 134

Slide 134 text

Instrumentation Complexity

Slide 135

Slide 135 text

Related Work Thread Schedule Minimization •Isolating Failure-Inducing Thread Schedules. SIGSOFT ’02. •A Trace Simplification Technique for Effective Debugging of Concurrent Programs. FSE ’10. Program Flow Analysis. •Enabling Tracing of Long-Running Multithreaded Programs via Dynamic Execution Reduction. ISSTA ’07. •Toward Generating Reducible Replay Logs. PLDI ’11. Best-Effort Replay of Field Failures •A Technique for Enabling and Supporting Debugging of Field Failures. ICSE ’07. •Triage: Diagnosing Production Run Failures at the User’s Site. SOSP ’07.

Slide 136

Slide 136 text

DDmin in more detail

Slide 137

Slide 137 text

DDmin assumptions

Slide 138

Slide 138 text

Local vs. global minima

Slide 139

Slide 139 text

Minimization Pace

Slide 140

Slide 140 text

Dealing With Threads If you’re lucky: threads are largely independent (Spark) If you’re unlucky: key insight: A write to shared memory is equivalent to a message delivery Approach: •interpose on virtual memory, thread scheduler •pause a thread whenever it writes to shared memory / disk Cf. “Enabling Tracing Of Long-Running Multithreaded Programs Via Dynamic Execution Reduction”, ISSTA ‘07

Slide 141

Slide 141 text

Dealing With Non-Determinism Interpose on: - Timers - Random number generators - Unordered hash values - ID allocation Stop-gap: replay each schedule multiple times

Slide 142

Slide 142 text

Complete Results

Slide 143

Slide 143 text

Runtime Breakdown

Slide 144

Slide 144 text

Integrating with other RPC libs App RPC lib OS App RPC lib OS App RPC lib OS DEMi JVM