Lessons Learnt From Building a Globally Distributed Database System From The Ground Up (Keynote version)

Abridged version of the Lessons Learnt From Building Cosmos DB presented at O'Reilly Velocity Conference, Santa Clara, June 2017.

1. Azure #CosmosDB Lessons learnt from building a globally distributed database from the ground up Dharma Shukla (@dharmashukla), Distinguished Engineer, Microsoft
2. Cosmos DB
3. 2010 Project Florence 2017 Cosmos DB Blood, sweat and tears Requirements (2010) • Turnkey global distribution • Low latency at the 99th percentile, worldwide • Guaranteed high availability • Programmable consistency • Elastically scale throughput and storage, globally, on demand • Operate at the lowest possible cost
4. Database hosted in the cloud != Database born in the cloud
5. Global Distribution Exploiting the cloud properties to the extreme IaaS hosted managed database offerings cannot match this! Millions of trans/sec Petabytes of data Elastic and unlimited scalability Cost efficiencies with fine grained multi-tenancy
6. Typical activity of an application
7.  Elastically scaling throughput from 10 to 100s of millions of transactions/sec across multiple regions  Fully resource governed stack  Highly responsive partition management  Modular, resource governed nested consensus  Multiple granularities of throughput (e.g. sec, min, hour) at different price points Elastically scaling throughput, anywhere, anytime 9 PM PST Less throughput More throughput More throughput Less throughput 11 PM PST
8. At global scale CREATE INDEX, DROP INDEX, ALTER TABLE
9. • Logical index layouts (inverted, tree, columnar, …) • Automatic and synchronous indexing of all ingested content • No schemas or secondary indices ever needed • Resource governed, write optimized database engine with latch free and log structured techniques Schema agnostic database engine locations headquarter exports 0 1 country Germany city Berlin country France city Paris city Moscow city Athens Belgium 0 1 { "locations": [ { "country": "Germany", "city": "Berlin" }, { "country": "France", "city": "Paris" } ], "headquarter": "Belgium", "exports":[{"city":"Moscow"}, {"city":"Athens"}] }
10. Real world consistency is not a binary choice
11. The wild west of consistency models…
12. The state of commercial databases Strong consistency High latency Eventual consistency, Low latency
13. Consistency models in Cosmos DB 5 well-defined consistency levels with clear tradeoffs Strong Bounded-stateless Session Consistent prefix Eventual Most real-life applications do not fall into these two extremes
14. Insights from production workloads 4 18 73 2 3 Usage (%) Strong Bounded Staleness Session Consistent Prefix Eventual 0 0.2 0.4 0.6 0.8 1 1.2 Throughput Consistency distribution among customers Consistency vs. Throughput
15. High availability SLA is not good enough
16. Microsoft Azure
17.  Global distribution, horizontal partitioning and fine-grained multi-tenancy cannot be an afterthought while building a cloud database  Schema agnostic database engine design is crucial for a globally distributed database  Intermediate consistency models are extremely useful  A globally distributed database must provide comprehensive SLAs beyond just high availability  Throughput, latency at 99th percentile, consistency and high availability Lessons learned
18. #Azure #CosmosDB We are Hiring