Four years of breaking things in production, on purpose.

7067ff85573929e5257aa9e9c1069de9?s=47 Eric Sigler
November 09, 2017
Technology
0
23

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

7067ff85573929e5257aa9e9c1069de9?s=128

Eric Sigler

November 09, 2017
Tweet

More Decks by Eric Sigler

See All by Eric Sigler
7067ff85573929e5257aa9e9c1069de9?s=48 esigler
0
140
7067ff85573929e5257aa9e9c1069de9?s=48 esigler
0
76
7067ff85573929e5257aa9e9c1069de9?s=48 esigler
0
60
7067ff85573929e5257aa9e9c1069de9?s=48 esigler
0
130
7067ff85573929e5257aa9e9c1069de9?s=48 esigler
0
170
7067ff85573929e5257aa9e9c1069de9?s=48 esigler
1
98
7067ff85573929e5257aa9e9c1069de9?s=48 esigler
0
140
7067ff85573929e5257aa9e9c1069de9?s=48 esigler
0
140
7067ff85573929e5257aa9e9c1069de9?s=48 esigler
0
40

Other Decks in Technology

See All in Technology
966e29b84ae7dbde170f66b0bc75b7a6?s=48 ishiayaya
1
220
3115a782126be714b5f94d24073c957d?s=48 oracle4engineer
3
300
79dcb04101764d7bf66f898dd446838e?s=48 tsukubachallenge
0
260
A645e7c3a6635ead8fa323c583769591?s=48 hololab
0
310
5b47136bedcba2799edf4fcd27ea66d7?s=48 yaegashi
0
320
4b09160b087e45103b210ef95079cee4?s=48 gakumura
0
130
5d715b858124e0dd59af1a92e4397acd?s=48 y_temp4
0
110
B3d836132393f607053b041f70dafbfc?s=48 horiuchie
1
130
9e7a653ed313b3da4b263a772bfc5026?s=48 clustervr
0
200
Ac099b28fee346b831ba8afcdb9e7d06?s=48 ericgpks
0
130
54bcc1069d75636cc735e72877dd30ec?s=48 taichinakamura
0
230
75fd80a8ace6ab292a7de563e018ae11?s=48 takashi_toyosaki
0
250

Featured

See All Featured
79acae287842acf29084005533a7fb3b?s=48 hursman
103
9k
Fde6c3a50ca518a909ef35c22c0ee0e4?s=48 destraynor
222
46k
78b475797a14c84799063c7cd073962f?s=48 holman
458
270k
1b75d89772b7eea1f6c89fbf362607d9?s=48 moore
123
21k
D47656e20ff5e42f125fc5ea0fd636c6?s=48 tmm1
61
7.4k
E6c6e133e74c3b83f04d2861deaa1c20?s=48 searls
202
35k
6804f1775cb4babfcc3851298566fbce?s=48 tanoku
256
24k
D36d2c1821af9249b69ff7f5ed60529b?s=48 tammielis
236
23k
F383c6a4dc55e331bbe2987b622cee6b?s=48 stephaniewalter
256
10k
C51b39fa3c79ccb99dcb8a076bc98287?s=48 brianwarren
83
4.5k
8081b26e05bb4354f7d65ffc34cbbd67?s=48 chriscoyier
681
180k
C157b16234f1e75e8eac3698c1d4414a?s=48 ddemaree
274
31k

Transcript

  1. Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of

    breaking things in production, on purpose.

  2. @esigler Obligatory disclaimer: This is what works for us. Take

    away ideas, not dogmas.

  3. @esigler

  4. @esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016

    2017

  5. @esigler 2013 2014 2015 2016 2017

  6. None

  7. @esigler 2014: Expanding Scope 2013 2014 2015 2016 2017

  8. @esigler 2013 2014 2015 2016 2017

  9. @esigler 2015: Automation 2013 2014 2015 2016 2017

  10. @esigler 2013 2014 2015 2016 2017

  11. @esigler 2013 2014 2015 2016 2017

  12. @esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017

  13. @esigler 2013 2014 2015 2016 2017

  14. @esigler Also 2016: Putting It All Together 2013 2014 2015

    2016 2017

  15. @esigler 2013 2014 2015 2016 2017

  16. @esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017

  17. @esigler 2013 2014 2015 2016 2017

  18. @esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections

    resulting in a public postmortem: 3

  19. @esigler Simulated full AZ failures: 4 Simulated full Region failures:

    3 Simulated partial Disaster Recovery: 2

  20. @esigler Tickets created from Failure Friday: over 225 Distinct services

    that had faults injected: 49

  21. @esigler

  22. @esigler Optimized for learning first, tooling second Built the toolchain

    to enable other teams Distributed chaos engineering knowledge

  23. @esigler