Four years of breaking things in production, on purpose.

November 09, 2017

45

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Eric Sigler

November 09, 2017

Tweet

More Decks by Eric Sigler

See All by Eric Sigler

Instrumenting The Rest Of The Company: Hunting For Metrics

0

290

A Brief Introduction To DevOps

0

97

Humans are terrible compilers: A User's Guide

0

110

Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.

0

160

"Is there any strong objection?"

0

200

Fear, Uncertainty, and Continuous Deployment

1

110

0

210

Strategies For Being On Call & Keeping Your Sanity At The Same Time

0

160

Engineering for Engineers

0

81

Other Decks in Technology

See All in Technology

.NET Profiler in 2024.

2

2.2k

Building a RAG-poweredAI chat appwith Python and VS Code

0

170

Handling focus in 2024

0

480

Cloud Service Mesh に触れ合う

1

270

Building Dashboards as a Hobby

0

410

ゼロから始めるVue.jsコミュニティ貢献 / first-vuejs-community-contribution-link-and-motivation

1

150

コードファーストの考え方。 Amplify Gen2から学ぶAWS次世代のWeb開発体験

2

510

DX企業CTOとして考える技術への向き合い方

0

100

M&A戦略を支えるデータマネジメント (MIDAS Tech Study #16 GENDA Komiyama)

1

150

One engineer company with Ruby on Rails

2

450

非同期推論システムによるコスト削減と信頼性向上

1

380

認知症フレンドリーテックとスタックチャン

0

340

Featured

See All Featured

Sharpening the Axe: The Primacy of Toolmaking

22

1.4k

Mobile First: as difficult as doing things right

217

8.6k

The World Runs on Bad Software

61

6.7k

The Illustrated Children's Guide to Kubernetes

32

47k

Being A Developer After 40

67

580k

Java REST API Framework Comparison - PWX 2021

18

6.9k

Designing with Data

96

4.8k

457

140k

A designer walks into a library…

pauljervisheath

201

23k

How to train your dragon (web standard)

75

5.2k

StorybookのUI Testing Handbookを読んだ

13

4.6k

Creatively Recalculating Your Daily Design Routine

211

11k

Transcript

Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of
breaking things in production, on purpose.
@esigler Obligatory disclaimer: This is what works for us. Take
away ideas, not dogmas.
@esigler
@esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016
2017
@esigler 2013 2014 2015 2016 2017
None
@esigler 2014: Expanding Scope 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2015: Automation 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler Also 2016: Putting It All Together 2013 2014 2015
2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections
resulting in a public postmortem: 3
@esigler Simulated full AZ failures: 4 Simulated full Region failures:
3 Simulated partial Disaster Recovery: 2
@esigler Tickets created from Failure Friday: over 225 Distinct services
that had faults injected: 49
@esigler
@esigler Optimized for learning first, tooling second Built the toolchain
to enable other teams Distributed chaos engineering knowledge
@esigler