2020-01-08 Thinking after that disaster in cloud

Transcript

1. 5"*1&*
2. 5IJOLJOH
   BGUFS
   UIBU
   EJTBTUFS
   JO
   DMPVE 4BNNZ
   -JO 4S
   &OHJOFFSJOH
   .BOBHFS
   43&
   
   
   .FEJB
5. Which cloud?
6. What if service down…
7. I implemented deployment strategy like blue/green and canary deployment. That would never happen.
8. I implemented deployment strategy like blue/green and canary deployment. That would never happen.
10. A leopard never changes its spots.
12. P .S This is a dummy SLA model.
13. ༻⼾戶
15. The more coming users, the more complicated actions.
16. "WBJMBCJMJUZ 100 x (Calendar Time ‒ Downtime) / Calendar Time 3FMJBCJMJUZ MTBF(Mean Time Between Failures) = Operating time (hours) / Number of Failures
17. 17Media SRE Journey 2015 • 17Media founded • Builded on AWS 2016/8 • First DevOps Engineer joined 2018/5 • Migrated to GCP 2017/1 • Migrated from beanstalk to ECS 2018/6 • Migrated from node.js to golang 2019/8 • Building GRE
18. 17Media SRE Journey 2015 • 17Media founded • Builded on AWS 2016/8 • First DevOps Engineer joined 2018/5 • Migrated to GCP 2017/1 • Migrated from beanstalk to ECS 2018/6 • Migrated from node.js to golang 2019/8 • Building GRE 2019 started to thinking and planning in terms of Disaster Recovery
21. 1. Someone drops database or workload because of fat-fingering error. 2. The entire region is down (Not considering global services) Assumptions
22. Split Components Run Labs Run Production
23. When last November…
25. Our multi-regions DR plan cannot be used.
27. Timeline • Investigating 12:00 PHASE 1
28. Timeline • Investigating • Identified 12:00 PHASE 1 15:00 PHASE 2
29. Timeline • Investigating • Identified • Started DR 12:00 PHASE 1 15:00 PHASE 2 PHASE 3 17:00
30. Timeline • Investigating • Identified • Still Resolving • Started DR 12:00 PHASE 1 15:00 PHASE 2 18:00 PHASE 4 PHASE 3 17:00
31. Timeline • Investigating • Identified • Still Resolving • Still Resolving • Enabled throttle • Started DR 12:00 PHASE 1 15:00 PHASE 2 18:00 PHASE 4 20:00 PHASE 5 PHASE 3 17:00
32. Timeline • Investigating • Identified • Still Resolving • Still Resolving • Enabled throttle • Resolved • Started DR 12:00 PHASE 1 15:00 PHASE 2 18:00 PHASE 4 20:00 PHASE 5 22:30 PHASE 6 PHASE 3 17:00
33. Timeline • Investigating • Identified • Still Resolving • Still Resolving • Enabled throttle • Resolved • Started DR 12:00 PHASE 1 15:00 PHASE 2 18:00 PHASE 4 20:00 PHASE 5 22:30 PHASE 6 PHASE 3 17:00
34. Hypothesis
35. How to resolve the current situation?
36. 1. Set up Emergency Response Team & SOP Developer Lead ! " # Communication Lead Owner BU Lead BU Lead Developer Developer ! # ! #
37. 1. Cold DR 2. Hot DR 3. HA 2. Phased DR implementation Image from https://picsart.com/i/sticker-cute-rainbow-cloud-sprinkle-sparkle-glitter-kawaii-293803594026211
38. 2-1. Cold DR implementation VPC VPC Private VPN Gateway Customer Gateway VPN Connection MySQL Mongo Redis Amazon Route 53 Weighted Records Amazon EKS Kubernetes Elastic Load Balancing Elastic Load Balancing 100%
39. 2-1. Cold DR implementation VPC VPC Private VPN Gateway Customer Gateway VPN Connection MySQL Mongo Redis Amazon Route 53 Weighted Records Amazon EKS Kubernetes Elastic Load Balancing Elastic Load Balancing 100%
40. 2-1. Cold DR implementation VPC VPC Private VPN Gateway Customer Gateway VPN Connection MySQL Mongo Redis Amazon Route 53 Weighted Records Amazon EKS Kubernetes Elastic Load Balancing Elastic Load Balancing 60% 40%
41. 2-1. Cold DR implementation MTTR: approximately 30 Mins Pros: Very cheap Cons High latency Break down if infra change
42. 2-2. Hot DR implementation
43. 2-2. Hot DR implementation VPC VPC Private VPN Gateway Customer Gateway VPN Connection MySQL Mongo Redis Amazon Route 53 Weighted Records Amazon EKS Kubernetes Elastic Load Balancing Elastic Load Balancing
44. 2-2. Hot DR implementation VPC VPC Private VPN Gateway Customer Gateway VPN Connection MySQL Mongo Redis Amazon Route 53 Weighted Records Amazon EKS Kubernetes Elastic Load Balancing Elastic Load Balancing 90% 10%
45. 2-2. Hot DR implementation VPC VPC Private VPN Gateway Customer Gateway VPN Connection MySQL Mongo Redis Amazon Route 53 Weighted Records Amazon EKS Kubernetes Elastic Load Balancing Elastic Load Balancing 90% 10%
46. 2-2. Hot DR implementation VPC VPC Private VPN Gateway Customer Gateway VPN Connection MySQL Mongo Redis Amazon Route 53 Weighted Records Amazon EKS Kubernetes Elastic Load Balancing Elastic Load Balancing 90% 10% Redis (slave) Mongo (secondary) MySQL
 (slave)
47. 2-2. Hot DR implementation MTTR: approximately 5 Mins Pros: Low latency of read Immediate notice if infra change Cons High latency of write Expensive
48. 2-3. HA implementation of multi-region/cloud
49. 2-3. HA implementation of multi-region/cloud Coming soon...
50. 2-3. HA implementation of multi-region/cloud MTTR: Without downtime Pros: Low latency of read/write Immediate notice if infra change Cons Very expensive Very complicated
51. Reference : https://medium.com/@sachinkagarwal/public-cloud-inter-region-network-latency-as-heat-maps-134e22a5ff19 Muti-region Ping latency heat-map
52. Traffic Throughput Latency MTTR Cost
53. 不是你選擇那一刻，而是那一刻選擇你 - 災難演練 @ AWS 實戰分享 Rick Hwang 研發資深經理, 91APP More
54. 5IBOL
    ZPV ˜
    
    "NB[PO
    8FC
    4FSWJDFT
    *OD
    PS
    JUT
    BGGJMJBUFT
    "MM
    SJHIUT
    SFTFSWFE Check our job page! 17 Media We're hiring Site Reliability Engineer - Infrastructure Site Reliability Engineer - Automation & Tools
 Backend Engineer