Building PostgreSQL as a Service with Kubernetes PGConf.Asia 2019 2019/9/9 @tzkb

3 My Actitvities PGConf.Asia 2018 @Tokyo A guide of PostgreSQL on Kuberntes - In terms of Storage - CloudNativeDays Tokyo 2019 The Future of Database on Kubernetes - What run with Cloud Native Storage -  ＋ ＝∞

4 Agenda 1. Recap: What is Kubernetes? 2. The Issues for Database on Kubernetes 3. How to run your PostgreSQL on K8s 4. Kubernetes becomes The Platform

6 1. Recap: What is Kubernetes?

7 What is Kubernetes? Pod Pod Pod Pod Pod • Kubernetes(K8s) is the orchestrator tool for containers. It has 3 features below. • Declarative config • Auto-healing • Immutable Database is not Immutable.

8 Better to handle the database system by Kubernetes? Node Node Node Master Slave Replicate • The database usually has a state that is not easy to maintain by Kubernetes. • Necessary to startup in turn. • Must never lose their data. • Handle the database as pets.

9 Example of Database on Kubernetes: Vitess VTtablet VTtablet VTtablet VTgate app app app SQL SQL SQL • Vitess that used on YouTube is the CNCF incubating project. • Vitess provides MySQL sharding in K8s. • VTgate and VTtablet can scale by K8s. • When terminating a component abnormally, Kubernetes repair it automatically.

10 The choice: How to manage your database Compute Storage Managed Amazon Aurora Amazon Redshift Amazon RDS on Cloud on Kubernetes • You can choose to manage the database by yourself or else.

17 2. The Issues for Database on Kubernetes

18 Kubernetes is the Distributed Systems • Developed as following a distributed architecture. • When doesn’t a node reply – Network partition? – Process failure? – Node failure? • If the disk resource attached, harder to determine. FailOver?

20 Database Architects are familiar with Clustering  “If you don’t know the status, it’s okay. We act on the premise of failsafe.”  “No need to share resources. Right?”  “Both have long been known for database clustering.”

21 Basic: Database Clustering HA (Active/Standby) １ Sharding Replication (Active/Active) ２or more Instances Redundancy 2 or more Shared Disk Log Shipping --- × Scaleout? Read Read/ Write Failover (Fencing) Availability Promotion (Election) --- • There are differences to build a DB cluster with some nodes.

22 Clustering #1: HA • With Linux-HA • Use high-available shared storage • Multiple writes to storage • Fencing VIP Linux-HA Controller Controller • It's been used since before Linux but helpful.

23 Note: Fencing VIP Linux-HA Controller Controller < When Detecting Node Failure > 1. Forced node power off i. Definite processes stop ii. Unmount storage iii. Detach virtual IP 2. PostgreSQL starts to run on the standby node. • Failed node is isolated from resources = Fencing

24 Clustering #2: Replication WAL • The master can Read/Write, Slaves are Read-Only. • Data synchronization by WAL transmission • 2 or more Masters • Leader Election • Redundancy built into PostgreSQL = Streaming Replication Master Slave Slave

25 Note: Leader Election WAL Be promoted as a master, The other is still a slave. • Always one master • The former master joins as a slave. 1. The remaining one slave is elected as the leader 2. The leader is promoted as a master. • Algorithms such as Paxos and Raft are used. Master Slave

26 Clustering #3 Sharding • Divide data between nodes and operates as one DB. • Dispatches queries to relevant nodes. • Basically no availability. • Problems with the transaction. • For rather scalability than availability. Coordinator

27 3. How to Run your on Kubernetes

28 Implemetation Overview : on Kubernetes # Category OSS used Description ⅰ HA • Use Rook/Ceph as Shared Storage. ⅱ • Use LINSTOR/DRBD as Shared Storage. ⅲ Replication • Use Streaming Replication, without Shared Storage. ⅳ Operator • Building and Operating Replication automatically. • We can see following four patterns.

29 • K8s manages everything(DB,storage) • Shared-Storage: Ceph • Fenced by kube-fencing < Disadvantage > • Complicated • Insufficient IO HA (i)： Replicas:1 • is deployed as StatefulSet using Rook/Ceph. kube-fencing

30 Note: Without Fencing Replicas:1 • When a node goes down, never failover. • To avoid network partition. • It is by design.

31 Note: What is • Rook is Kubernetes Operator managing Ceph or others. operator agent/discover agent/discover agent/discover osd osd osd mon mon mon CSI csi-provisioner csi-rbdplugin csi-rbdplugin csi-rbdplugin Rook • Rook makes easy to build Ceph cluster. • Also easy to deploy CSI modules. • CSI: Containar Storage Interface

32 HA (ii)： Replicas:1 kube-fencing • LINSTOR is Software-Defined Storage based on DRBD. • K8s manages everything(DB,storage) • Redundancy: DRBD • Simple, Read IO without Network < Disadvantage > • Limited to Scale

33 Benchmark Results Single(with EBS) Rook/Ceph DRBD 1nodes 5nodes 2nodes 100 37.8 77.1 • Measured by pgbench for 3 patterns. TPS

34 Replication : proxy proxy proxy keeper keeper keeper sentinel sentinel sentinel • Builds Streaming Replication on top of Kubernetes. • 3 types of processes have different roles • Without Shared- Resources < Disadvantage > • Not builtin Read Off- loading

36 Operator : • KubeDB operates not only but also others. kubedb-operator -0 -1 -2 postgres snapshot dormantdabases • Database Operator for – PostgreSQL – MySQL – Redis • Kubedb-operator builds SR. • Able to get/restore snapshot easily.

37 Example : PostgreSQL Configration by KubeDB apiVersion: kubedb.com/v1alpha1 kind: Postgres metadata: name: ha-postgres namespace: demo spec: version: “10.6-v2" replicas: 3 storageType: Durable storage: storageClassName: "standard" accessModes: - ReadWriteOnce resources: requests: storage: 100Gi  spec.version – Choose PostgreSQL version.  spec.replicas – The number of Instances.  spec.storage – Define storage type/size, etc. • Allows to define Streaming Replication with a simple YAML.

38 Example : Snapshot by KubeDB apiVersion: kubedb.com/v1alpha1 kind: Snapshot metadata: name: snapshot-to-s3 labels: kubedb.com/kind: Postgres spec: databaseName: ha-postgres storageSecretName: s3-secret s3: endpoint: 's3.amazonaws.com' bucket: kubedb-qa prefix: demo • Write declarative Snapshot settings by YAML. • Simple backup that applies only this YAML. • You can select storage, – S3 – Swift – Kubernetes Persistent Volume

39 Note : Backup with PostgreSQL + Ceph $ kubectl exec -it -n rook-ceph rook-ceph-tools-seq -- rbd -p replicapool ls pvc-bdbc6e53-f6e9-11e8-b0d9-02f062df6b48 $ kubectl exec -it pg-rook-sf-0 -- psql -h localhost -U postgres -c "SELECT pg_start_backup(now()::text);" pg_start_backup ----------------- 0/C000028 (1 row) $ kubectl exec -it -n rook-ceph rook-ceph-tools-seq -- rbd snap create replicapool/img@snap $ kubectl exec -it pg-rook-sf-0 -- psql -h localhost -U postgres -c "SELECT pg_stop_backup();" NOTICE: pg_stop_backup complete, all required WAL segments have been archived pg_stop_backup ---------------- 0/D000050 (1 row) • Need to know PostgreSQL & Ceph Commands.

41 4. Kubernetes becomes The Platform

42 To Recap  The components of database clustering with Kubernetes Native are already available.  You can see some operators for DBA task automation.  However, it is not over yet. Cloud Native Storage ＋ ＋ ＝ ???

43 The Signs I. Pluggable Storage  Optimized Storage system for DB on K8s? II. Forked and Cloud-Oriented PostgreSQL  AWS Aurora, Azure Hyperscale

44 THE LOG IS THE DATABASE. SQL Transactions Caching Storage Logging Storage Logging Storage Logging CPU Memory Cache(SSD) Page Cache(SSD) Log AWS Aurora(PostgreSQL) Azure Hyperscale • Both divide RDBMS functions and are extended by each cloud.

45 As the platform for PostgreSQL as a Service DBaaS by Kubernetes STaaS by Kubernetes What we got for DBaaS • HA • Streaming Replication • DB Operator Also for STaaS • Simple Redundancy • Distributed Storage • Interoperable IF(CSI) • Kubernetes will be "The Platform for Platforms."

46 Questions? @tzkb @tzkoba

47 Appendix