Tweet
Tweet

Slide 1

Slide 1 text

How Did I Get Here? Building Confidence in a Distributed Stream Processor

Slide 2

Slide 2 text

Sean T. Allen

Slide 3

Slide 3 text

T

Slide 4

Slide 4 text

T

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

Experience Report

Slide 10

Slide 10 text

Stream Processor

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

Prototype Started January 2016

Slide 13

Slide 13 text

Prototype Started January 2016

Slide 14

Slide 14 text

Production Started April 2016

Slide 15

Slide 15 text

Production Started April 2016

Slide 16

Slide 16 text

America is all about speed. Hot, nasty, bad-ass speed. — Eleanor Roosevelt

Slide 17

Slide 17 text

High Throughput Buffy: Goals

Slide 18

Slide 18 text

Low Latency Buffy: Goals

Slide 19

Slide 19 text

Less Hardware Buffy: Goals

Slide 20

Slide 20 text

America is all about data quality. Quiet, demure data quality. — Andrew Jackson

Slide 21

Slide 21 text

High Fidelity Buffy: Goals

Slide 22

Slide 22 text

Stream Processing

Slide 23

Slide 23 text

Message at a time

Slide 24

Slide 24 text

Never ending

Slide 25

Slide 25 text

Failure

Slide 26

Slide 26 text

Machine Failure

Slide 27

Slide 27 text

Slow Machine

Slide 28

Slide 28 text

Segfaulting Process

Slide 29

Slide 29 text

GC Pause

Slide 30

Slide 30 text

Network Error

Slide 31

Slide 31 text

Failure Happens

Slide 32

Slide 32 text

Delivery Guarantees

Slide 33

Slide 33 text

At-Most-Once

Slide 34

Slide 34 text

At-Most-Once Best Effort

Slide 35

Slide 35 text

At-Least-Once

Slide 36

Slide 36 text

At-Least-Once ACK or resend

Slide 37

Slide 37 text

Exactly-Once

Slide 38

Slide 38 text

Exactly-Once At-Least-Once + Idempotence

Slide 39

Slide 39 text

Exactly-Once

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

No content

Slide 62

Slide 62 text

Confidence

Slide 63

Slide 63 text

Black Box Testing

Slide 64

Slide 64 text

Black Box Testing

Slide 65

Slide 65 text

Black Box Testing

Slide 66

Slide 66 text

Black Box Testing

Slide 67

Slide 67 text

Black Box Testing

Slide 68

Slide 68 text

System Under Test Black Box Testing

Slide 69

Slide 69 text

Input Source Black Box Testing

Slide 70

Slide 70 text

Output Receiver Black Box Testing

Slide 71

Slide 71 text

Unit Testing because isn't enough Black Box Testing

Slide 72

Slide 72 text

Integration Testing because isn't enough Black Box Testing

Slide 73

Slide 73 text

composed components because have interesting new failure modes Black Box Testing

Slide 74

Slide 74 text

Test The Entire System Black Box Testing

Slide 75

Slide 75 text

Test The Entire System end to end Black Box Testing

Slide 76

Slide 76 text

Test The Entire System end to end Black Box Testing and verify your expectations

Slide 77

Slide 77 text

Wesley Expectation verification for Buffy

Slide 78

Slide 78 text

Wesley

Slide 79

Slide 79 text

Wesley Input

Slide 80

Slide 80 text

Wesley Output

Slide 81

Slide 81 text

Wesley Input Output

Slide 82

Slide 82 text

Input Source Wesley

Slide 83

Slide 83 text

Input Source Wesley Output Receiver

Slide 84

Slide 84 text

Wesley Input Source Records sent data 1,2,3,4

Slide 85

Slide 85 text

Wesley Input Source Records sent data Records received data 2,4,6,8 1,2,3,4 Output Receiver

Slide 86

Slide 86 text

Wesley

Slide 87

Slide 87 text

Wesley Analyze!

Slide 88

Slide 88 text

Wesley

Slide 89

Slide 89 text

Wesley

Slide 90

Slide 90 text

Wesley

Slide 91

Slide 91 text

Wesley

Slide 92

Slide 92 text

Wesley

Slide 93

Slide 93 text

Wesley

Slide 94

Slide 94 text

Wesley

Slide 95

Slide 95 text

Wesley

Slide 96

Slide 96 text

Wesley It Works!

Slide 97

Slide 97 text

Spike Fault injection for Buffy

Slide 98

Slide 98 text

Fault Injection

Slide 99

Slide 99 text

Lineage-driven fault injection

Slide 100

Slide 100 text

Start from a good result Spike: LDFI

Slide 101

Slide 101 text

Input Spike: LDFI

Slide 102

Slide 102 text

Output Spike: LDFI

Slide 103

Slide 103 text

Figure out what can go wrong Spike: LDFI

Slide 104

Slide 104 text

Nemesis Spike: LDFI Each "wrong" is a possible

Slide 105

Slide 105 text

The Network Spike: LDFI Our first nemesis:

Slide 106

Slide 106 text

Determinism is key Spike

Slide 107

Slide 107 text

Repeated runs with different results == Mostly Useless Spike

Slide 108

Slide 108 text

Spike

Slide 109

Slide 109 text

Spike Inject failures as informed by TCP

Slide 110

Slide 110 text

Spike TCP Guarantees:

Slide 111

Slide 111 text

Spike TCP Guarantees: Per connection in order delivery

Slide 112

Slide 112 text

Spike Per connection in order delivery Per connection duplicate detection TCP Guarantees:

Slide 113

Slide 113 text

Spike Per connection in order delivery Per connection duplicate detection Per connection retransmission of lost data TCP Guarantees:

Slide 114

Slide 114 text

TCP in Pony: Event Driven

Slide 115

Slide 115 text

TCP in Pony: Event Driven

Slide 116

Slide 116 text

TCP in Pony: Event Driven

Slide 117

Slide 117 text

TCP in Pony: Event Driven

Slide 118

Slide 118 text

TCP in Pony: Event Driven

Slide 119

Slide 119 text

Useless Notifier

Slide 120

Slide 120 text

Useless Notifier

Slide 121

Slide 121 text

Useless Notifier

Slide 122

Slide 122 text

Dropped Connections Nemesis #1:

Slide 123

Slide 123 text

Spike: Drop Connection

Slide 124

Slide 124 text

Spike: Drop Connection

Slide 125

Slide 125 text

Spike: Drop Connection

Slide 126

Slide 126 text

Spike: Drop Connection

Slide 127

Slide 127 text

Spike: Drop Connection

Slide 128

Slide 128 text

Spike: Drop Connection • Incoming connection accepted

Slide 129

Slide 129 text

Spike: Drop Connection • Incoming connection accepted • Attempting outgoing connection

Slide 130

Slide 130 text

Spike: Drop Connection • Incoming connection accepted • Attempting outgoing connection • Connection established

Slide 131

Slide 131 text

Spike: Drop Connection • Incoming connection accepted • Attempting outgoing connection • Connection established • Data sent

Slide 132

Slide 132 text

Spike: Drop Connection • Incoming connection accepted • Attempting outgoing connection • Connection established • Data sent • Data received

Slide 133

Slide 133 text

Integrating Spike "Double and Halve" app

Slide 134

Slide 134 text

Integrating Spike "Double and Halve" app

Slide 135

Slide 135 text

Integrating Spike "Double and Halve" app

Slide 136

Slide 136 text

Integrating Spike "Double and Halve" app

Slide 137

Slide 137 text

Integrating Spike "Double and Halve" app

Slide 138

Slide 138 text

Integrating Spike "Double and Halve" app

Slide 139

Slide 139 text

Integrating Spike "Double and Halve" app

Slide 140

Slide 140 text

Integrating Spike "Double and Halve" app

Slide 141

Slide 141 text

Integrating Spike "Double and Halve" app

Slide 142

Slide 142 text

Integrating Spike "Double and Halve" app

Slide 143

Slide 143 text

• Easy to verify Integrating Spike "Double and Halve" app

Slide 144

Slide 144 text

• Easy to verify • Messages cross process boundary Integrating Spike "Double and Halve" app

Slide 145

Slide 145 text

• Easy to verify • Messages cross process boundary • Messages cross network boundary Integrating Spike "Double and Halve" app

Slide 146

Slide 146 text

Integrating Spike • Double and Halve App

Slide 147

Slide 147 text

Integrating Spike • Double and Halve App • No Spiking

Slide 148

Slide 148 text

Integrating Spike • Double and Halve App • No Spiking • Test, Test, Test

Slide 149

Slide 149 text

Integrating Spike • Double and Halve App • No Spiking • Test, Test, Test • Wesley: It passes! It passes! It passes!

Slide 150

Slide 150 text

Integrating Spike • Double and Halve App

Slide 151

Slide 151 text

Integrating Spike • Double and Halve App • Spike with “drop connection”

Slide 152

Slide 152 text

Integrating Spike • Double and Halve App • Spike with “drop connection” • Test, Test, Test

Slide 153

Slide 153 text

Integrating Spike • Double and Halve App • Spike with “drop connection” • Test, Test, Test • Wesley: It fails! It fails! It fails!

Slide 154

Slide 154 text

Integrating Spike

Slide 155

Slide 155 text

Integrating Spike == Session Recovery!

Slide 156

Slide 156 text

Integrating Spike • Double and Halve App

Slide 157

Slide 157 text

Integrating Spike • Double and Halve App • Spike with “drop connection”

Slide 158

Slide 158 text

Integrating Spike • Double and Halve App • Spike with “drop connection” • Test, Test, Test

Slide 159

Slide 159 text

Integrating Spike • Double and Halve App • Spike with “drop connection” • Test, Test, Test • Wesley: It passes! It passes! It passes!

Slide 160

Slide 160 text

Repeated runs with different results == Mostly Useless Spike

Slide 161

Slide 161 text

Determinism & Spike

Slide 162

Slide 162 text

It's easy to get wrong Determinism & Spike

Slide 163

Slide 163 text

Determinism & Spike TCP delivery is not deterministic

Slide 164

Slide 164 text

Determinism & Spike TCP guarantees: Per connection in order delivery

Slide 165

Slide 165 text

Determinism & Spike Per connection in order delivery Per connection duplicate detection TCP guarantees:

Slide 166

Slide 166 text

Determinism & Spike Per connection in order delivery Per connection duplicate detection Per connection retransmission of lost data TCP guarantees:

Slide 167

Slide 167 text

Determinism & Spike Per connection in order delivery Per connection duplicate detection Per connection retransmission of lost data but it doesn't guarantee determinism TCP guarantees:

Slide 168

Slide 168 text

Determinism & Spike TCP delivery is not deterministic

Slide 169

Slide 169 text

Determinism & Spike TCP delivery is not deterministic

Slide 170

Slide 170 text

Determinism & Spike TCP delivery is not deterministic

Slide 171

Slide 171 text

Determinism & Spike TCP delivery is not deterministic Per method call Spiking won't work

Slide 172

Slide 172 text

Determinism & Spike TCP delivery is not deterministic Per method call Spiking won't work unless we make it work…

Slide 173

Slide 173 text

Determinism & Spike TCP message framing

Slide 174

Slide 174 text

Determinism & Spike TCP message framing

Slide 175

Slide 175 text

Determinism & Spike TCP message framing

Slide 176

Slide 176 text

Determinism & Spike TCP message framing

Slide 177

Slide 177 text

Determinism & Spike TCP message framing

Slide 178

Slide 178 text

Determinism & Spike TCP message framing

Slide 179

Slide 179 text

Determinism & Spike TCP message framing

Slide 180

Slide 180 text

Determinism & Spike TCP message framing

Slide 181

Slide 181 text

Determinism & Spike TCP message framing

Slide 182

Slide 182 text

Determinism & Spike Expect in action

Slide 183

Slide 183 text

Determinism & Spike Expect in action

Slide 184

Slide 184 text

Determinism & Spike Expect in action

Slide 185

Slide 185 text

Determinism & Spike Expect in action

Slide 186

Slide 186 text

Determinism & Spike Expect in action

Slide 187

Slide 187 text

Determinism & Spike Expect in action

Slide 188

Slide 188 text

Determinism & Spike Expect in action

Slide 189

Slide 189 text

Determinism & Spike Expect in action

Slide 190

Slide 190 text

Determinism & Spike Expect in action

Slide 191

Slide 191 text

Determinism & Spike Expect in action

Slide 192

Slide 192 text

Determinism & Spike Expect makes received deterministic

Slide 193

Slide 193 text

Determinism & Spike Expect makes received deterministic

Slide 194

Slide 194 text

Determinism & Spike Expect makes received deterministic

Slide 195

Slide 195 text

Determinism & Spike Expect makes received deterministic

Slide 196

Slide 196 text

Determinism & Spike Expect makes received deterministic

Slide 197

Slide 197 text

Determinism & Spike Expect makes received deterministic

Slide 198

Slide 198 text

Determinism & Spike Expect makes received deterministic

Slide 199

Slide 199 text

Determinism & Spike Received gets called with

Slide 200

Slide 200 text

Determinism & Spike then…

Slide 201

Slide 201 text

Determinism & Spike and then another…

Slide 202

Slide 202 text

Determinism & Spike and finally…

Slide 203

Slide 203 text

Same number of notifier method calls Determinism & Spike no matter how the data arrives

Slide 204

Slide 204 text

Drop Connection & Expect fast deterministic friends Determinism & Spike Determinism & Spike

Slide 205

Slide 205 text

Slow Connections Nemesis #1:

Slide 206

Slide 206 text

Spike: Delay

Slide 207

Slide 207 text

Spike: Delay

Slide 208

Slide 208 text

Spike: Delay

Slide 209

Slide 209 text

Spike: Delay

Slide 210

Slide 210 text

Spike: Delay Delay overrides expect

Slide 211

Slide 211 text

Spike: Delay Delay overrides expect and controls the flow of bytes

Slide 212

Slide 212 text

Spike: Delay Delay overrides expect and controls the flow of bytes to maintain determinism

Slide 213

Slide 213 text

Spike: Delay

Slide 214

Slide 214 text

Spike: Delay

Slide 215

Slide 215 text

Spike: Delay

Slide 216

Slide 216 text

Spike: Delay

Slide 217

Slide 217 text

Spike: Delay r TCP Spike

Slide 218

Slide 218 text

Spike: Delay r TCP Spike

Slide 219

Slide 219 text

Spike: Delay r TCP Spike

Slide 220

Slide 220 text

Spike: Delay TCP

Slide 221

Slide 221 text

Spike: Delay TCP TCP Spike

Slide 222

Slide 222 text

Spike: Delay TCP TCP TCP Spike Spike

Slide 223

Slide 223 text

Early Results

Slide 224

Slide 224 text

Early Results • Bugs in Session Recovery Found…

Slide 225

Slide 225 text

Early Results • Bugs in Session Recovery • Bug in Pony standard library Found…

Slide 226

Slide 226 text

Early Results • Bugs in Session Recovery • Bug in Pony standard library • Bugs in Spike Found…

Slide 227

Slide 227 text

Early Results • Bugs in Session Recovery • Bug in Pony standard library • Bugs in Spike • And more bugs… Found…

Slide 228

Slide 228 text

Determinism is key Early Results Found…

Slide 229

Slide 229 text

Determinism is key Early Results but hard to achieve Found…

Slide 230

Slide 230 text

Data Lineage

Slide 231

Slide 231 text

WARNING!!! Vaporware ahead

Slide 232

Slide 232 text

Output Data Lineage How did I get here?

Slide 233

Slide 233 text

Output Data Lineage

Slide 234

Slide 234 text

Data Lineage Input: 1,2,3

Slide 235

Slide 235 text

Data Lineage Input: 1,2,3 Expect: 2,4,6

Slide 236

Slide 236 text

Data Lineage Input: 1,2,3 Expect: 2,4,6 Get: 4,6

Slide 237

Slide 237 text

Data Lineage Input: 1,2,3 Expect: 2,4,6 Get: 4,6 How did we get here? these are not our beautiful results

Slide 238

Slide 238 text

Data Lineage Input: 1,2,3

Slide 239

Slide 239 text

Data Lineage Input: 1,2,3 Expect: 2,4,6

Slide 240

Slide 240 text

Data Lineage Input: 1,2,3 Expect: 2,4,6 Get: 2,6,12

Slide 241

Slide 241 text

Data Lineage Input: 1,2,3 Expect: 2,4,6 Get: 2,6,12 ¯\_(ϑ)_/¯

Slide 242

Slide 242 text

Data Lineage to the Rescue!

Slide 243

Slide 243 text

Data Lineage Externally verify determinism

Slide 244

Slide 244 text

Data Lineage Externally verify determinism is it REALLY deterministic?

Slide 245

Slide 245 text

Data Lineage Find incorrect executions

Slide 246

Slide 246 text

Data Lineage Find incorrect executions bugs in Buffy

Slide 247

Slide 247 text

Data Lineage Input: 1 Expected: 2 Got: 4 ¯\_(ϑ)_/¯

Slide 248

Slide 248 text

Data Lineage Execution path was… when it should have been

Slide 249

Slide 249 text

Data Lineage when it should have been Execution path was…

Slide 250

Slide 250 text

Data Lineage Useful outside of development

Slide 251

Slide 251 text

Data Lineage Production Debugging

Slide 252

Slide 252 text

Data Lineage Production Debugging how did I get here?

Slide 253

Slide 253 text

Data Lineage Audit Log

Slide 254

Slide 254 text

Data Lineage Audit Log why did you do that?

Slide 255

Slide 255 text

Data Lineage Hindsight Machine

Slide 256

Slide 256 text

Building Confidence is difficult

Slide 257

Slide 257 text

and frustrating

Slide 258

Slide 258 text

No content

Slide 259

Slide 259 text

Don't be this dog

Slide 260

Slide 260 text

Be this dog

Slide 261

Slide 261 text

No content

Slide 262

Slide 262 text

Peter Alvaro http://www.cs.berkeley.edu/~palvaro/molly.pdf @palvaro https://www.youtube.com/watch?v=ggCffvKEJmQ Lineage-driven Fault Injection: Outwards from the Middle of the Maze:

Slide 263

Slide 263 text

Kyle Kingsbury https://aphyr.com/tags/Jepsen @aphyr Jepsen:

Slide 264

Slide 264 text

Will Wilson https://www.youtube.com/watch?v=4fFDFbi3toc Testing Distributed Systems w/ Deterministic Simulation:

Slide 265

Slide 265 text

Catie McCaffrey http://queue.acm.org/detail.cfm?ref=rss&id=2889274 @caitie The Verification of a Distributed System The Verification of a Distributed System: A practitioner's guide to increasing confidence in system correctness 2:55 PM Tomorrow in Salon E

Slide 266

Slide 266 text

Inés Sombra https://www.youtube.com/watch?v=KSdNYi55kjg Testing in a Distributed World: @randommood

Slide 267

Slide 267 text

http://principlesofchaos.org Principles of Chaos Engineering: Chaos Engineering

Slide 268

Slide 268 text

No content

Slide 269

Slide 269 text

Thanks Peter Alvaro Sylvan Clebsch Zeeshan Lakhani John Mumm Rob Roland Andrew Turley

Slide 270

Slide 270 text

@SeanTAllen Note: The 'T' is very important