Four years of breaking things in production, on...

November 09, 2017

53

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Eric Sigler

November 09, 2017

Tweet

More Decks by Eric Sigler

See All by Eric Sigler

Instrumenting The Rest Of The Company: Hunting For Metrics

0

340

A Brief Introduction To DevOps

0

100

Humans are terrible compilers: A User's Guide

0

110

Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.

0

160

"Is there any strong objection?"

0

210

Fear, Uncertainty, and Continuous Deployment

1

110

0

210

Strategies For Being On Call & Keeping Your Sanity At The Same Time

0

160

Engineering for Engineers

0

84

Other Decks in Technology

See All in Technology

株式会社EventHub・エンジニア採用資料

0

4.3k

現場の種を事業の芽にする - エンジニア主導のイノベーションを事業戦略に装着する方法 -

2

2.1k

AndroidデバイスにFTPサーバを建立する

0

250

偶然 × 行動で人生の可能性を広げよう / Serendipity × Action: Discover Your Possibilities

1

1.1k

個人開発から公式機能へ: PlaywrightとRailsをつなげた3年の軌跡

11

3k

ホワイトボードチャレンジ説明&実行資料

0

130

一度 Expo の採用を断念したけど、再度 Expo の導入を検討している話

1

170

あれは良かった、あれは苦労したB2B2C型SaaSの新規開発におけるCloud Spanner

2

570

分解して理解する Aspire

1

110

Developer Summit 2025 [14-D-1] Yuki Hattori

19

6.2k

滅・サービスクラス🔥 / Destruction Service Class

6

1.6k

30分でわかる『アジャイルデータモデリング』

9

2.7k

Featured

See All Featured

RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub

133

33k

Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything

30

2.2k

Chrome DevTools: State of the Union 2024 - Debugging React & Beyond

4

330

Embracing the Ebb and Flow

84

4.6k

Six Lessons from altMBA

27

3.6k

Automating Front-end Workflow

1368

200k

ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020

114

50k

Bash Introduction

611

210k

Adopting Sorbet at Scale

74

9.2k

Gamification - CAS2011

80

5.1k

"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)

226

22k

Fight the Zombie Pattern Library - RWD Summit 2016

233

17k

Transcript

Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of
breaking things in production, on purpose.
@esigler Obligatory disclaimer: This is what works for us. Take
away ideas, not dogmas.
@esigler
@esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016
2017
@esigler 2013 2014 2015 2016 2017
None
@esigler 2014: Expanding Scope 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2015: Automation 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler Also 2016: Putting It All Together 2013 2014 2015
2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections
resulting in a public postmortem: 3
@esigler Simulated full AZ failures: 4 Simulated full Region failures:
3 Simulated partial Disaster Recovery: 2
@esigler Tickets created from Failure Friday: over 225 Distinct services
that had faults injected: 49
@esigler
@esigler Optimized for learning first, tooling second Built the toolchain
to enable other teams Distributed chaos engineering knowledge
@esigler