Where TiDB Meets Mobike

This deck was delivered at the SF/Bay Area Big Data Applications Meetup to talk about TiDB is fundamental architecture, motivation, and use case with Mobike, one of the largest dockless bikesharing platforms in the world.

A video of the talk can be viewed here: https://www.youtube.com/watch?v=D_tfFQz0IK8

Transcript

1. Where TiDB Meets Mobike Make Data Creative Kevin Xu (@pingcap; @kevinsxu) April 18, 2018
2. • History and Community • Technical Overview • Use Case with Mobike • DIY HTAP (Demo) • Q&A Agenda
3. History • Founded in April 2015 by 3 infrastructure engineers • TiDB platform: (Ti = Titanium) ◦ TiDB (stateless SQL layer compatible with MySQL) ◦ TiKV (distributed transactional key-value store) ◦ TiSpark (Apache Spark plug-in) • Open source from Day 1 ◦ GA 1.0: October 2017 ◦ Current version: 2.0-RC4
4. TiDB in the Community This Afternoon (04/18):
5. TiDB in the news 2018-03-29 Chaos Tools and Techniques for Testing the TiDB Distributed NewSQL Database 2018-02-08 TiDB: Performance-tuning a distributed NewSQL database 2018-02-22 The Hybrid Database Capturing Perishable Insights at Yiguo 2018-01-31 Tick or Tock? Keeping Time and Order in Distributed Databases 2017-04-26 TiDB Brings Distributed Scalability to SQL
6. • Hybrid OLTP & OLAP (No More ETL) • Horizontal Scalability • MySQL Compatible • Distributed Transaction (ACID Compliant) • High Availability • Cloud-Native TiDB Core Features
7. Architecture TiDB TiDB Worker Spark Driver TiKV Cluster (Storage) Metadata TiKV TiKV TiKV Data location Job TiSpark DistSQL API TiKV TiDB TSO/Data location Worker Worker Spark Cluster TiDB Cluster TiDB DistSQL API PD PD Cluster TiKV TiKV TiDB KV API Application Syncer SparkSQL PD PD
8. TiDB: SQL above KV (OLTP) Node1 Node2 Node3 Node4 MySQL Network Protocol SQL Parser Cost-based Optimizer Distributed Executor (Coprocessor) ODBC/JDBC MySQL Client Any ORM which supports MySQL TiDB TiKV
9. ID Name Email 1 Edward h@pingcap.com 2 Tom tom@pingcap.com ... user/1 Edward,h@pingcap.com user/2 Tom,tom@pingcap.com ... In TiKV -∞ +∞ (-∞, +∞) Sorted map “User” Table TiDB: SQL above KV (OLTP)
10. TiSpark: Spark above KV (OLAP) Spark Exec Spark Exec Spark Driver Spark Exec TiKV TiKV TiKV TiKV TiSpark TiSpark TiSpark TiSpark TiKV Placement Driver (PD) gRPC Distributed Storage Layer gRPC retrieve data location retrieve real data from TiKV
11. TiKV: The Foundation RocksDB Raft Transaction Txn KV API Coprocessor API RocksDB Raft Transaction Txn KV API Coprocessor API RocksDB Raft Transaction Txn KV API Coprocessor API Raft Group Client gRPC TiKV Instance TiKV Instance TiKV Instance gRPC gRPC PD Cluster
12. Dynamic Split and Merge Region A Region A Region B Region A Region A Region B Split Region A Region A Region B Merge TiKV_1 TiKV_2 TiKV_2 TiKV_1
13. Hotspot Scheduling Region A’ Region B’ Region A Region B Workload Region A’ Region B Region A Region B’ Workload Workload Hotspot Schedule (Raft leader transfer) TiKV_1 TiKV_2 TiKV_2 TiKV_1
14. Support: • Hash Join • Sort Merge Join • Index Lookup Join Chosen based on Cost-base Optimizer: Join Network cost Memory cost CPU cost
15. • Timestamp Oracle (TSO) based on Google’s Percolator paper • 2-Phase commit protocol (2PC) • Avoid single point of failure with PD cluster ◦ Organized by Raft protocol Transaction Model
16. Cloud-Native ... ...
17. Who’s using TiDB?
18. Mobike + TiDB • 200 million users • 200 cities • 9 milllion smart bikes • ~30 TB / day
19. • Locking and unlocking of smart bikes generate massive log • Smooth experience is key to user retention • TiDB supports this system by alerting administrators when success rate of locking/unlocking drops, within minutes • Quickly find malfunctioning bikes Scenario #1: Locking/Unlocking
20. • Synchronize TiDB with MySQL instances using Syncer (proprietary tool) • TiDB + TiSpark empower real-time analysis with horizontal scalability • No need for Hadoop + Hive Scenario #2: Real-Time Analysis
21. • An innovative loyalty program that must be on 24 x 7 x 356 • TiDB handles: ◦ High-concurrency for peak or promotional season ◦ Permanent storage ◦ Horizontal scalability • No interruption as business evolves Scenario #3: Mobike Store
22. • Massive log data from different sources: ◦ Parking histories ◦ Notifications from locking/unlocking • TiDB keeps log for analysis, supports new service development • No need for ETL Scenario #4: Log Collection
23. DIY HTAP Demo
24. Thank You! Visit: Pingcap.com (Request Private Demo) Just Google: “TiDB Docker Compose” Twitter: @PingCAP; @kevinsxu https://github.com/pingcap (Give us a Watch/Star!) Kevin Xu (kevin@pingcap.com)