How Netflix does Failovers in 7 minutes

Transcript

3. Whoops, something went wrong… Netflix Streaming Error We’re having trouble playing this title right now. Please try again later or select a different title.
6. Cloud Prod Engineering AMJITH RAMANUJAM (@amjithr) Sr. Software Engineer, Traffic Team
7. Regional Failover in 7 minutes AMJITH RAMANUJAM (@amjithr) Sr. Software Engineer, Traffic Team
8. Regional Standby system takes over when the main system fails Failover
10. Amazon Web Services Everything else Netflix Open Connect All video delivery
11. Regional Failover
12. Christmas Eve 2012
17. ELB Outage 7 Hours
18. 16,000 years in 1 day content watched ~14,000 BC: First colonization of America 2018 0 AD 4,600 years in 7 hours
19. Regional Failover
20. Active - standby system is also serving traffic Active vs Passive Passive - standby system is NOT serving traffic
21. Stateless services Prerequisites Regional replication of data
22. Infrastructure problem isolated to one region Failover Candidate Problem won’t follow if we move traffic Bad code deploy in a region
23. Detect the problem Regional Failover Process Scale the savior regions Shift traffic
25. Detect the problem
26. Is it working?
28. One metric to rule them all - Dumbledore
29. SPS Stream Starts Per Second
31. Scale Saviors
35. Scaling Pattern Linear Regression
36. Shift Traffic
37. Proxy Traffic Traffic Shift Switch DNS
39. Detect the problem - 5 minutes Regional Failover Process Scale the savior regions - 35 minutes Shift traffic - 10 minutes Total = 45 mins
40. Nimble Goals • Fast failover (<10mins) ◦ Pre-scale • Transparent to service owners ◦ No code changes for service owners ◦ No auto-scaling changes
41. API lolomo nccp Playready us-west
42. API lolomo nccp nimble_API nimble_lolomo nimble_nccp Playready nimble_Playready us-west
43. API lolomo nccp nimble_API nimble_lolomo nimble_nccp Playready nimble_Playready us-west
44. API lolomo nccp nimble_API nimble_lolomo nimble_nccp starting starting starting starting Playready nimble_Playready us-west
45. Failover API lolomo nccp Playready nimble_API nimble_Playready nimble_lolomo nimble_nccp
46. Failover API lolomo nccp Playready nimble_API nimble_Playready nimble_lolomo nimble_nccp
47. Failover API lolomo nccp Playready nimble_API nimble_Playready nimble_lolomo nimble_nccp
48. Failover API lolomo nccp nimble_API nimble_lolomo nimble_nccp Playready nimble_Playready
49. Detect the problem - 2 minutes Regional Failover Process Scale the savior regions - 4 minutes Shift traffic - 3 minutes Total = 7 mins
50. Architecture
51. Actions Periodic Tasks Triggered Tasks
52. Periodic Tasks Fetch historical data Predict cluster sizes Manage dark clusters
53. Triggered Tasks Ungate dark instances Transplant instances Traffic shift
54. Nanoservices Python Flask RQ - Redis
55. Characteristics Anticipate Failure Eventually Consistent Garbage Collect
56. Anticipate Failure Multi-region Rebuild state from scratch Fallbacks
57. Fallbacks AWS State - EDDA Historical data - ATLAS Local Cache - Redis, Filesystem
58. Eventual Consistency AWS is eventually consistent Favor idempotent actions
59. Orphan Cleaner • Terminate detached instances • Safety features ◦ Terminate slowly ◦ Don’t terminate large volume of instances
60. FAQs How often do you failover? Why not have dark clusters take traffic? How much did Nimble cost?
61. Suggestions Fallbacks, Fallbacks, Fallbacks Exercise it often Provide visibility
62. We’re hiring! @amjithr