"Smooth Operator" on TiDB Operator

This deck was delivered at the Bay Area In-Memory Computing meetup to share some experiences, learning, and pitfalls in building TiDB-Operator (https://github.com/pingcap/tidb-operator), which is an open source implementation of the Kubernetes Operator pattern to deploy and manage a TiDB cluster in any cloud environment.

A video of the talk can be viewed here: https://www.youtube.com/watch?v=a-Ehehrgiw0

Transcript

1. “Smooth” Operator Kevin Xu (@kevinsxu; kevin@pingcap.com)
2. Agenda • History and Community • Technical Walkthrough • Kubernetes Operator ◦ to run cloud-native stateful app
3. A little about PingCAP • Founded in April 2015 by 3 infrastructure engineers • Created and maintains TiDB, TiKV • Offices throughout North America and China
4. PingCAP.com Community Stars • TiDB: 17,000+ • TiKV: 4,500+ Contributors • TiDB: 245+ • TiKV: 130+ TiDB DevCon: • 700+
5. PingCAP.com Who’s Using TiDB? 300+ Deployments
6. Use Cases 1. Approaching the maximum size for MySQL on a single server. Debating whether or not to shard. 2. Already sharded MySQL, but having a hard time doing analytics on up-to-date data.
7. PingCAP.com Mobike + TiDB • 200 million users • 200 cities • 9 million smart bikes • ~35 TB in TiDB
8. Technical Walkthrough
9. Technical Inspiration TiDB is a NewSQL database that speaks the MySQL protocol It is not based on the MySQL source code It is an ACID/strongly consistent database The inspiration is Google Spanner + F1 It separates SQL processing and storage into separate components Both of them are independently scalable The SQL processing layer is stateless It is designed for both Transaction and Analytical Processing (HTAP)
10. TiDB TiDB Region 1 L TiKV Node 1 Region 2 Region 3 Region 4 Region 2 L TiKV Node 3 Region 3 Region 4 L Region 1 Region 4 TiKV Node 2 Region 3 L Region 2 Region 1 TiKV Cluster PD Cluster
11. Row + Column Storage (Announced Jan 2019) Spark Cluster TiDB TiDB Region 1 TiKV Node 1 Region 2 Region 3 Region 4 Region 2 TiKV Node 3 Region 3 Region 4 Region 1 Region 4 TiKV Node 2 Region 3 Region 2 Region 1 TiFlash Node 2 TiFlash Extension Cluster TiKV Cluster TiSpark Worker TiSpark Worker TiFlash Node 1
12. Kubernetes -- Operator
13. Kubernetes on 1 slide • Pod: Smallest unit of work (analogous to a node, an instance) ◦ can hold 1 or more containers ◦ it dies? it dies. just get a new pod (cattle v. pet) • Deployment: API for a Pod ◦ great for stateless apps, like RESTful API ◦ use pod resources, declare # of replicas, rolling deployment, etc. • StatefulSet: Another API for a Pod, designed for stateful apps ◦ maintain order, “sticky” identity, no interchangeability, etc.
15. Operator History • Operator pattern pioneered by CoreOS...now Redhat...now IBM • Introduced in 2016, Operator Framework in 2018 ◦ First 2: etcd operator, Prometheus operator • TiDB Operator (2017); Predated Operator Framework • OperatorHub.io
16. Why Do We (As in TiDB) Care? • Manage multiple clusters (multi-tenancy) • Safe scaling (up or down, in or out) • Use different types of Network or Local Storage (different performance) • Automatic monitoring • Rolling updates • Automatic failover • *Multi-Cloud* (as long as it has k8s)
17. Why Should YOU Care? • Manages *stateful* applications: ◦ databases, caches, monitoring system, etc. • Encodes application domain knowledge ◦ Extension of your SRE team • Kubernetes-enabled Hybrid / Multi-Cloud • Growing popularity in database community: ◦ https://thenewstack.io/databases-operators-bring-stateful-workloads-to-kubernet es/
18. PingCAP.com High Level Resources => Controllers => Clusters
19. Resources -- CRD • Custom Resource Definition (CRD): ◦ An application-specific YAML file ◦ End user writes the domain operation logic in CRD ◦ Simple to implement and deploy • (There is another way): ◦ API Aggregation: ▪ More control, more powerful but… ▪ Hard to deploy, not well-supported by k8s engines
20. Cluster State -- StatefulSet StatefulSet... • Guarantees ordering and uniqueness of pods ◦ pd -> tikv -> tidb • Gives “sticky” identity -- network and storage • *No* interchangeable pods ◦ always map the same volume to the same pod • Stable since k8s 1.9 (we are at 1.13 1.14 now)
21. How TiDB manages state -- Custom Controller Spec: component: image: replicas: ... Status: image replicas state CRD (provided by user) Custom Controller Cluster State
22. How TiDB manages state apiVersion: pingcap.com/v1alpha1 kind: TidbCluster metadata: name: demo spec: pd: image: pingcap/pd:v2.1.0 replicas: 3 requests: cpu: “4” memory: “8Gi” … tikv: image: pingcap/tikv:v2.1.0 … status: tikv: stores: “5”: podName: demo-tikv-2 state: Up ... type Manager interface { Sync(*TidbCluster) error }
23. “Smooth” Operator
24. “Smooth” Operator
25. PingCAP.com All Open Sourced https://github.com/pingcap/tidb-operator/blob/ master/docs/user-guide.md
26. PingCAP.com On GCP Marketplace
27. PingCAP.com TiDB, Managed As A Cloud Service Early Access: https://www.pingcap.com/tidb-cloud/
28. Thank You @kevinsxu; kevin@pingcap.com