How Patroni solved Database Reliability at Gojek

Transcript

1. How Patroni solved Database Reliability at Gojek Kumar Abhijeet Cloud Platforms
2. Fabricating DBaaS@Gojek DevOps/Platforms Home Gym Owner Budding Musician
3. Agenda Gojek - Scale and Microservices Databases & Reliability Patroni & 5 9s of Availability Deep dive into Patroni Managing Patroni on production - Lessons & Experiences
4. ~600 microservices running on production ~400 have databases
5. 600k RPM 12000 WALs/hour
6. 18Bn record inserts/month 85Bn records fetched/month
7. Will a conventional master-slave PostgreSQL system be able to support country-level scale?
8. App Server Workloads PostgreSQL VMs API Traffic LB
9. App Server Workloads PostgreSQL VMs API Traffic LB
10. Cloud Provider’s Compute Uptime >= 99.9% < 8h 41m of downtime/year Across multiple zones >= 99.99% < 52m of downtime/year
11. Database Uptime ≅ App Uptime
12. Target >= 99.999% Uptime Less than 5m of downtime/year
13. App Server Workloads PostgreSQL VMs API Traffic LB New Master Old Master Replica
15. App Server Workloads PostgreSQL VMs API Traffic LB New Master Old Master Replica shared_buffers=16MB shared_buffers=2GB
16. Enter Patroni!
17. Patroni Open Source and actively maintained by Zalando Converts PostgreSQL systems into Highly Available Fault Tolerant Disaster Ready
21. Patroni Almost instantaneous failovers (~1-2s) Way cheaper than running managed DB solutions Cluster Management made easy Multi Region HA Deployments
23. HA Loop Flow
26. Downtime in Seconds≈0.0000315576
28. Patroni at Gojek 200+ clusters running on Production ~60 TB of data flows in/out every day Guarantees less than 10MBs of data loss Consul as DCS and service discovery IAC everywhere!
29. Patroni at Gojek TF Modules for Provisioning/Chef for configuration Sync/Async replication choices All round observability! Secure and granular role-based access PR based workflow for infra provisioning
33. Thank you!