Road to four nines

Learn the journey to achieving 99.99% uptime (four nines). This talk covers reliability engineering practices, monitoring strategies, incident response, and the architectural decisions that enable systems to achieve and maintain exceptional availability.

Transcript

1. Road to Four Nines
2. Dr. Ilya Kaznacheev Cloud-Native Architect Ex. GDE on Google Cloud 13 years in software engineering
3. 99.99%
4. how do we hit 99.99%?
5. microservices!
6. microservices?
7. Redundancy
8. eliminate single point of failure
9. replica 1 replica 2 availability load + replica
10. service 1 service 2 availability load + dependency
12. proper application design
13. • stateless • horizontally scalable • graceful shutdown • health checks
14. redundant services
15. replica 1 replica 2 load replica 3
16. replica 1 replica 2 load replica 3 ❌
17. Load Management
18. load balancing
19. replica 1 replica 2 load replica 3 ❌
20. replica 1 replica 2 load replica 3 ❌ load balancer
21. replica 1 replica 2 load replica 3 ❌ load balancer health monitoring
22. proper connectivity mechanisms
24. Data Replication
25. database replication
26. master replica replication service
27. decoupling via replication
28. service a service b
29. service a service b ❌ ❌ ❌
30. service a service b service b meta replication
31. service a service b service b meta replication ❌ ❌ ❌
32. object store
34. Fault Tolerance
35. payload distribution
36. node 2 node 1 node 3
37. node 2 node 1 node 3 ❌
38. node 2 node 1 node 3 ❌
39. network
40. service service service load balancer service service service load balancer load balancer client
41. Self-Healing
42. • autonomous detection • autonomous diagnosis • recovery • replication, scaling, balancing • predictive maintenance
44. compute
45. storage
46. IAM identity and access management
48. servers & racks
50. switches & routers
53. data storage systems
56. Disaster Tolerance
57. multi-zone
58. multi-region
59. global routing
60. multi-cloud
61. Failover & Recovery
64. CI/CD & Supply Chain
65. • repeatable builds • automated deployment • autonomous rollbacks • dependency vendoring (local storage) • security and vulnerability monitoring • infrastructure automation (IaC)
66. disaster recovery drills
67. chaos engineering
69. L1/L2/L3 support
70. • teams and responsibilities • ITSM support system • escalation work fl ow • scripts / knowledge base • alerts, monitoring, root-cause analysis • on-call rotation
73. build vs. buy
74. build vs. buy own vs. rent
75. so, how to reach 99.99%?
77. • redundancy • load management • data replication • fault tolerance • self-healing • disaster tolerance • failover & recovery • CI/CD & supply chain
78. Thank You! Any questions now?
79. Follow me on