Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
How Netflix does Failovers in 7 minutes
Search
Amjith
May 12, 2018
Programming
750
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
How Netflix does Failovers in 7 minutes
Amjith
May 12, 2018
Transcript
62 slides
None
None
Whoops, something went wrong… Netflix Streaming Error We’re having trouble
playing this title right now. Please try again later or select a different title.
None
None
Cloud Prod Engineering AMJITH RAMANUJAM (@amjithr) Sr. Software Engineer, Traffic
Team
Regional Failover in 7 minutes AMJITH RAMANUJAM (@amjithr) Sr. Software
Engineer, Traffic Team
Regional Standby system takes over when the main system fails
Failover
None
Amazon Web Services Everything else Netflix Open Connect All video
delivery
Regional Failover
Christmas Eve 2012
None
None
None
None
ELB Outage 7 Hours
16,000 years in 1 day content watched ~14,000 BC: First
colonization of America 2018 0 AD 4,600 years in 7 hours
Regional Failover
Active - standby system is also serving traffic Active vs
Passive Passive - standby system is NOT serving traffic
Stateless services Prerequisites Regional replication of data
Infrastructure problem isolated to one region Failover Candidate Problem won’t
follow if we move traffic Bad code deploy in a region
Detect the problem Regional Failover Process Scale the savior regions
Shift traffic
None
Detect the problem
Is it working?
None
One metric to rule them all - Dumbledore
SPS Stream Starts Per Second
None
Scale Saviors
None
None
None
Scaling Pattern Linear Regression
Shift Traffic
Proxy Traffic Traffic Shift Switch DNS
None
Detect the problem - 5 minutes Regional Failover Process Scale
the savior regions - 35 minutes Shift traffic - 10 minutes Total = 45 mins
Nimble Goals • Fast failover (<10mins) ◦ Pre-scale • Transparent
to service owners ◦ No code changes for service owners ◦ No auto-scaling changes
API lolomo nccp Playready us-west
API lolomo nccp nimble_API nimble_lolomo nimble_nccp Playready nimble_Playready us-west
API lolomo nccp nimble_API nimble_lolomo nimble_nccp Playready nimble_Playready us-west
API lolomo nccp nimble_API nimble_lolomo nimble_nccp starting starting starting starting
Playready nimble_Playready us-west
Failover API lolomo nccp Playready nimble_API nimble_Playready nimble_lolomo nimble_nccp
Failover API lolomo nccp Playready nimble_API nimble_Playready nimble_lolomo nimble_nccp
Failover API lolomo nccp Playready nimble_API nimble_Playready nimble_lolomo nimble_nccp
Failover API lolomo nccp nimble_API nimble_lolomo nimble_nccp Playready nimble_Playready
Detect the problem - 2 minutes Regional Failover Process Scale
the savior regions - 4 minutes Shift traffic - 3 minutes Total = 7 mins
Architecture
Actions Periodic Tasks Triggered Tasks
Periodic Tasks Fetch historical data Predict cluster sizes Manage dark
clusters
Triggered Tasks Ungate dark instances Transplant instances Traffic shift
Nanoservices Python Flask RQ - Redis
Characteristics Anticipate Failure Eventually Consistent Garbage Collect
Anticipate Failure Multi-region Rebuild state from scratch Fallbacks
Fallbacks AWS State - EDDA Historical data - ATLAS Local
Cache - Redis, Filesystem
Eventual Consistency AWS is eventually consistent Favor idempotent actions
Orphan Cleaner • Terminate detached instances • Safety features ◦
Terminate slowly ◦ Don’t terminate large volume of instances
FAQs How often do you failover? Why not have dark
clusters take traffic? How much did Nimble cost?
Suggestions Fallbacks, Fallbacks, Fallbacks Exercise it often Provide visibility
We’re hiring! @amjithr
More Decks by Amjith
See All by Amjith
Awesome Commandline Tools
amjith
0
570
FuzzyFind
amjith
0
130
Awesome Command Line Tools
amjith
1
190
Modern Command Line
amjith
2
410
Introduction to Docker
amjith
8
1.5k
Thread Profiling in Python
amjith
7
890
Python Profiling
amjith
3
390
Statistical Thread Profiler
amjith
1
180
Debugging Live Python Web Applications
amjith
8
1.9k
Other Decks in Programming
See All in Programming
Ruby から Emacs を操作したい
phigasui
0
350
Python Lambdaにおけるテストしやすい設計とAI支援の活用 / Designing Testable Python Lambda Functions with AI
kitano_yuichi
1
340
Crash Auto Pipelines
veronikapj
1
430
エラーチャンネルのボールを拾い続けて鍛えるエンジニアの地力 / phpcon-niigata2026
meihei3
1
450
Materialが消えた日 〜真っ白なFlutterキャンバスにRenderObjectで絵を描く〜
umigishiaoi
0
170
조합이폭발하는디자인시스템이걸눈으로도테스트하라고요
giljongyu
0
780
小さなアプリを作って公開するまで
onozaty
0
200
AIがコードを書いてくれる時代、我々は何を確かめるのか
yosuke_furukawa
PRO
13
6.1k
あなたはMCPの機能を、まだ知らない。
4geru
1
680
Zero-Tap Sign-In対応、結局何をすればいいの?
mitohato14
0
2.4k
作れるエンジニアから、価値を届けるエンジニアへ
ryunakayama
1
310
テストだけで見つかる? Rubyでつくるメソッド発掘器
izawa
0
140
Featured
See All Featured
End of SEO as We Know It (SMX Advanced Version)
ipullrank
3
4.5k
Why Mistakes Are the Best Teachers: Turning Failure into a Pathway for Growth
auna
0
360
Introduction to Domain-Driven Design and Collaborative software design
baasie
1
1.1k
New Earth Scene 8
popppiees
5
2.9k
SEOcharity - Dark patterns in SEO and UX: How to avoid them and build a more ethical web
sarafernandez
0
340
Groundhog Day: Seeking Process in Gaming for Health
codingconduct
0
490
The Hidden Cost of Media on the Web [PixelPalooza 2025]
tammyeverts
2
580
Skip the Path - Find Your Career Trail
mkilby
1
290
Believing is Seeing
oripsolob
1
280
Un-Boring Meetings
codingconduct
0
500
XXLCSS - How to scale CSS and keep your sanity
sugarenia
250
1.3M
Building Adaptive Systems
keathley
44
3.3k