Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
dojo.pdf
Search
Rich Burroughs
April 18, 2019
Technology
0
130
dojo.pdf
Rich Burroughs
April 18, 2019
Tweet
Share
More Decks by Rich Burroughs
See All by Rich Burroughs
Virtual_Kubernetes_Clusters__Tips_and_Tricks_-_Rejekts.pdf
richburroughs
0
1.2k
What On-Call Does to Us
richburroughs
1
120
Other Decks in Technology
See All in Technology
衛星運用をソフトウェアエンジニアに依頼したときにできあがるもの
sankichi92
1
1k
[SRE NEXT] ARR150億円_エンジニア140名_27チーム_17プロダクトから始めるSLO.pdf
satos
5
3k
AIでテストプロセス自動化に挑戦する
sakatakazunori
1
530
60以上のプロダクトを持つ組織における開発者体験向上への取り組み - チームAPIとBackstageで構築する組織の可視化基盤 - / sre next 2025 Efforts to Improve Developer Experience in an Organization with Over 60 Products
vtryo
3
1.9k
対話型音声AIアプリケーションの信頼性向上の取り組み
ivry_presentationmaterials
3
1k
Four Keysから始める信頼性の改善 - SRE NEXT 2025
ozakikota
0
410
全部AI、全員Cursor、ドキュメント駆動開発 〜DevinやGeminiも添えて〜
rinchsan
10
5.1k
PHPからはじめるコンピュータアーキテクチャ / From Scripts to Silicon: A Journey Through the Layers of Computing
tomzoh
2
120
Introduction to Bill One Development Engineer
sansan33
PRO
0
260
AI Ready API ─ AI時代に求められるAPI設計とは?/ AI-Ready API - Designing MCP and APIs in the AI Era
yokawasa
6
1.9k
ソフトウェアQAがハードウェアの人になったの
mineo_matsuya
3
200
Contract One Engineering Unit 紹介資料
sansan33
PRO
0
6.9k
Featured
See All Featured
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
60k
Code Reviewing Like a Champion
maltzj
524
40k
Being A Developer After 40
akosma
90
590k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2.2k
Bash Introduction
62gerente
613
210k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Faster Mobile Websites
deanohume
308
31k
A designer walks into a library…
pauljervisheath
207
24k
How to Ace a Technical Interview
jacobian
278
23k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
750
Rails Girls Zürich Keynote
gr2m
95
14k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
2.9k
Transcript
Learning Through Failure Rich Burroughs Community Manager Gremlin, Inc. @richburroughs
None
None
Complexity is constantly increasing
None
None
None
What's changed?
None
None
"Catastrophe is always just around the corner"
"Change introduces new forms of failure"
"All practitioner actions are gambles"
None
None
What are some ways we can learn more about systems?
None
None
None
Chaos Engineering
"The science of performing intentional experimentation on a system by
injecting precise and measured amounts of harm to observe how the system responds for the purpose of improving the system’s resilience."
None
Prerequisites —Observability —Blameless Culture
Scientific Method —Ask a question —Research —Form a hypothesis —Experiment
to test the hypothesis —Analyze data and draw a conclusion —Share the results
Types of attacks —Shutdown —CPU —Memory —I/O —Network Latency —Packet
Loss —DNS —Blackhole
None
The goal is to experiment in Production
None
Example experiment —Application: Front End —Attack: CPU —Hypothesis: Adding CPU
load will cause additional hosts to spin up in our Autoscaling Group —Abort condition: Latency increases by 20%
Example experiment #2 —Application: Front End —Attack: Blackhole —Hypothesis: Blackholing
the hostname for the Twilio API will cause the SMS transmissions to time out —Abort condition: Error rate increases by 20%
Don't experiment on things you know are broken
None
Questions —Were we able to measure the results? —Did the
system respond the way we expected? —Are there things we need to fix?
Run experiments to simulate an incident you've had
What comes after Game Days?
Continuous Chaos
Maturity model —Running manual experiments —Running experiments using Chaos Engineering
tools —Regularly scheduled Game Days —Experimenting in Production —Continuous Chaos
Next steps: —Join our Chaos Engineering Slack: gremlin.com/ slack —Read
tutorials: gremlin.com/community —Chaos Conf: chaosconf.io —Gremlin Free: go.gremlin.com/richchaos
Thank you! Twitter: @richburroughs Email:
[email protected]
Slides: https://github.com/richburroughs/ dojo201904