Zero downtime with little effort

In this presentation the Authlete will explain how they achieve Database update without downtime using YugabyteDB

Transcript

1. Zero downtime with little effort www.authlete.com
2. Presenters Henrik Falck VP of Engineering Zolo Jargalsaikhan DevOps & Support Arnaud Bruyer SRE & DevOps
3. Authlete introduction - Founded 2015 in Tokyo - About 30 members from/in a dozen countries - Identity component as a service - Authorization backend for backends - Managed (SaaS) or on-premise
4. Authlete infrastructure (dedicated cloud) GKE kubernetes cluster (multi-AZ) Authlete server CloudSQL proxy Authlete server CloudSQL proxy Authlete server CloudSQL proxy CloudSQL MySQL HA Primary Failover
5. Background on problem (1/2) CloudSQL Maintenance - approx. quarterly - 1~2 minutes downtime - notified in advance - need to notify customers - customers often want to reschedule (possible within 28 days of date) - high support overhead ~5 min/year downtime (0.999%) is itself not a big issue, but highly opinionated customers are.
6. Background on problem (2/2) Performance - We are still far from hitting a limit that scaling up cannot solve - But we have had on-premise customers who are looking for very high scalability High availability - Ability to distribute cross-region (and globally) opens up new options for us - Being able to scale up/out the DB without downtime is an advantage
7. Solutions we explored a.k.a. “Why don’t you just use Spanner?” We have discussed Vitess, Spanner, CochroachDB, YugabyteDB, TiDB since 2020 - Prefer managed solutions (don’t want to hire DBA team) - Need good compatibility with our Java DB middleware (Mysql/Postgres) - Ideally should offer on-premise DB solution for our on-premise customers - Good support and partnership opportunities
8. Initial Tests and Performance tuning
9. Getting Started (Initial tests) To make sure YugabyteDB was compatible with Authlete, first step was to pass our unit tests with Yugabyte as a DB. PostgreSQL / MySQL Authlete YugabyteDB Authlete
10. Initial tests Just had to set one flag to pass all tests. Authlete 🥲 (Some tests fail) YugabyteDB Authlete 😄 (All tests pass) YugabyteDB yb_enable_read_committed_isolation =true
11. 3 Authlete pods on GKE Performance tuning Authlete Authlete Yugabyte Managed Cluster on GCP Authlete perftest client VPC peering Next, we tried running load test. - We want to observe the behavior under stress (Errors, panics, timeouts) - Measure performance and draw conclusion Authlete • Create a cluster on Yugabyte Managed • Add Yugabyte smart driver
12. Performance tuning Performance advisor: https://docs.yugabyte.com/preview/yugabyte-cloud/cloud-monitor/cloud-advisor/
13. Performance tuning Thanks to Gwenn, we tuned the performance even further: - Enabled bunch of flags (for the most part, I didn’t know what these flags did) - We tested upgrading the Yugabyte Managed cluster while running load test - We achieved satisfactory results
14. Compatibility and Integeration
15. Column type change - Liquibase rollback was failing due to unsupported data type change - At that time, PostgreSQL supported this one, not Yugabyte - Yugabyte team provided support and helped us to fix our issue in recent version
16. Retries and refresh timeout - Reworked DB calls to handle retries - Lowered Yugabyte Smart driver node list refresh timeout YB Node YB Node YB Node Authlete Node list: 1, 2, 3 YB Node YB Node YB Node Authlete Node list: 1, 3 After refresh and retry Zero downtime retry
17. Integration - Easy integration in CI/CD pipelines - Almost same arguments as PostgreSQL - Built-in tools in docker image to manage DB
18. Conclusions and next steps Pilot project for SaaS - We are planning on running a pilot project with customer cooperation - Evaluation of real-world use of Yugabyte-backed Authlete SaaS Yugabyte officially supported for on-premise customers - We are communicating to our customers that Yugabyte is a supported DB option General availability as an option for SaaS customers - Upon completion of pilot project, we are expecting to offer Yugabyte as an HA option
19. Thank You! Questions? www.authlete.com