Distributed SQL Operational Best Practices with YugabyteDB

Amey Banarse, Principal Data Architect at YugabyteDB walked through an overview of YugabyteDB's architecture and how to apply operational best practices to its administrations. Topics covered included:

* Self-service deployments
* Deployment topologies
* Automation
* Data recoverability
* Config management
* Monitoring & health checks
* Security integration

Recorded on 12/18/19
video webinar: https://vimeo.com/380552852

Transcript

1. YugabyteDB Operational Best Practices Amey Banarse Principal Data Architect, Yugabyte
2. 2 © 2019 All rights reserved. Introduction 2  Amey Banarse Principal Data Architect, YugabyteDB ♦ Pivotal ♦ FINRA University of Pennsylvania @ameybanarse http://about.me/amey
3. © 2020 All Rights Reserved PostgreSQL-compatible, high-performance, open-source, cloud-native distributed SQL database 100% Apache 2.0 Low Latency & High Throughput Built for Kubernetes & Cloud Native Ecosystem
4. 4 Yugabyte Confidential © 2019 All rights reserved. • Database Reliability Engineering ◦ Inspired by Google’s SRE model ◦ Blending DevOps culture with DBA teams ◦ Infrastructure as code ◦ Automation is the key Introducing DBRE model
5. 5 Yugabyte Confidential © 2019 All rights reserved. • Responsibility of the data shared by cross-functional teams • Provide patterns and knowledge to support other team’s processes to facilitate their work • Defining reference architectures and configurations for data stores that are approved for operations, and can be deployed by teams. DBRE Guiding Principles
6. 6 Yugabyte Confidential © 2019 All rights reserved. “Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure and declarative APIs exemplify this approach. These techniques enable loosely coupled systems that are resilient, manageable and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil.” Cloud Native - cncf.io definition
7. 7 Yugabyte Confidential © 2019 All rights reserved. Designed for Cloud Native Microservices Sharding & Load Balancing Raft Consensus Replication Distributed Transaction Manager & MVCC Document Storage Layer Custom RocksDB Storage Engine DocDB Distributed Document Store Yugabyte Query Layer YCQL YSQL PostgreSQL Google Spanner YugabyteDB SQL Ecosystem ✓ Massively adopted ✘ New SQL flavor ✓ Reuse PostgreSQL RDBMS Features ✓ Advanced Complex ✘ Basic cloud-native ✓ Advanced Complex and cloud-native Highly Available ✘ ✓ ✓ Horizontal Scale ✘ ✓ ✓ Distributed Txns ✘ ✓ ✓ Data Replication Async Sync Sync + Async
8. © 2020 All Rights Reserved Yugabyte Cloud Fully Managed DBaaS Yugabyte offers flexible consumption models 24 x 7 Enterprise Support Yugabyte Platform UI Operational Excellence DBaaS out of the box Yugabyte Platform YugabyteDB Self managed Self Service UI Yugabyte managed Public DBaaS Community supported Yugabyte DB Transactional Distributed SQL DB 100% Open Source Apache 2.0 PostgreSQL compatible Enterprise grade RDBMS Cloud Native Self or Yugabyte managed https://download.yugabyte.com
9. 9 Yugabyte Confidential © 2019 All rights reserved. • Comprehensive operations lifecycle manager • Cloud Native deployments on public & private cloud • Robust automation for Day 2 ops with self healing • Built-in monitoring and alerts • Configuration & Change Management Yugabyte Platform
10. 10 Yugabyte Confidential © 2019 All rights reserved. • Choosing the right topology for business ◦ Business Continuity or data regulations ◦ Single or Multi Regional cluster • Multi & Hybrid Cloud deployments • Self Service deployment on any form factor - Containers, Virtual Machine(VM), Bare Metal • Ability to expand to new Regions Self-Service Deployment
11. 11 Yugabyte Confidential © 2019 All rights reserved. 1. Single Region, Multi-Zone Availability Zone 1 Availability Zone 2 Availability Zone 3 Consistent Across Zones No WAN Latency But No Region-Level Failover/Repair 2. Single Cloud, Multi-Region Region 1 Region 2 Region 3 Consistent Across Regions with Auto Region-Level Failover/Repair 3. Multi-Cloud, Multi-Region Cloud 1 Cloud 2 Cloud 3 Consistent Across Clouds with Auto Cloud-Level Failover/Repair Resilient and strongly consistent across failure domains
12. 12 © 2018 All rights reserved. • On AWS, 7 Production clusters, 3 are 15 Nodes each. ~4 non-Prod. • Deployed across 3 AZs, 5 Nodes per AZ, rf=3. Can handle AZ failures. • In Prod, they did zero downtime upgrade of YB from v1.1.9 to 1.2.6 • ~400 GB compressed data per node = 1 TB per node • Write-heavy clusters, 60K to 80K writes/sec at peak with CPU utilization < 20%. YugabyteDB Customers & Sample Deployment Topologies Customer A • 18-node Production cluster with 2 TB of compressed data per node. • 300+ days of uninterrupted uptime and availability • Zero-downtime node repair • Cluster expansion with zero-downtime and in minutes Customer B
13. 13 Yugabyte Confidential © 2019 All rights reserved. • Changes incorporated into deployment and infrastructure automation, with focus on testing, fallback, and impact mitigation • Rolling Upgrades, Scale Up/Down & CI tools • Self healing • Operational Efficiency with minimal toil Automation
14. 14 Yugabyte Confidential © 2019 All rights reserved. • Standardized and automated backup and recovery processes • Business Continuity and Disaster Recovery ◦ Defining RPO and RTO • Defining the right backup strategy with data validation pipelines Data Recoverability
15. 15 Yugabyte Confidential © 2019 All rights reserved. Data Recoverability • Auto Failover • Resilient to AZ/Rack outages • No downtime for client apps
16. 16 Yugabyte Confidential © 2019 All rights reserved. • IaaS Configurations • Centralized Store for Cluster Config • Security - KMS or SmartKey integration • Backup/Restore - Object Store or NFS Config Management
17. 17 Yugabyte Confidential © 2019 All rights reserved. • Monitoring KPI(s) on various levels ◦ Node, Network, Memory usage & Storage layer • Build Alerting and integration w/ Incident Response platforms like PagerDuty • Automated health-checks for regular activities Monitoring & Health Checks
18. 18 Yugabyte Confidential © 2019 All rights reserved. Enterprise Security Hardening 18 • Third-party security audit • Audit Logging for DB operations • YSQL Kerberos and GSSAPI (in development) • LDAP support for better user management • YSQL support is GA since YB v2.4 • YCQL support will be available in YB v2.6.x • Data in Transit Encryption Enhancements • TLS v1.2 across the database • Simplified TLS workflows in YB Platform • Data at Rest Encryption with Vormetric TDE and AWS KMS Intg.
19. 19 Yugabyte Confidential © 2019 All rights reserved. Case Study
20. 20 Yugabyte Confidential © 2019 All rights reserved. • 27 billion operations per day with 30-40 ms of latency • 35 TBs of data managed by a single cluster • Rolling upgrades with zero downtime • Less time spent managing databases and more time spent on core business • YugabyteDB can support Plume’s next phase of growth which is predicted to reach 75+ billion ops per day Plume Design
21. 21 Yugabyte Confidential © 2019 All rights reserved. Demo