How to scale large database

Bài nói về các kĩ thuật để mở rộng một database lớn.

Transcript

1. How To Scale Large Database Phạm Tùng Dương – CIO03 Course: Advanced Database
2. Overview •  First glance about Large Database •  Typical techniques to scale •  Database sharding •  Database sharding in MySQL
3. First glance about Large Database  
4. When You Talk about Large Database
5. Example Tumblr @2012
6. Example •  400 million active users •  5 billion pieces of content per week • 3 billion photos uploaded per month Facebook@2010
7. Example •  1 billion tweets per week •  140 million tweets sent per day •  456 tweets per second @MJ death •  6939 tweets per second on NY day Twitter@2011
8. What is The Large Database •  Large working data sets •  I/O write intensive
9. Typical approaches
10. What is The Bottleneck? I/O, I/O and I/O
11. We have a job which is called Performance Tuning
12. Scale up •  Adding more RAM, more CPU •  High I/O HDD
13. Scale topo Replication (Master – Slave) Master Slave Client Read/Write Read Only Master Master Storage Client Cluster (shared storage)  
14. Caching •  Memcached •  Redis
15. Finally, Everything in RAM is a Dream!
16. But, No Silver Bullet!
17. Database Sharding
18. What is Database Sharding •  Horizontal Partitioning •  Data is stored in small chunks and distributed across many computers •  Often use with Replication
19. Database sharding topo Primary  DB   Shard1   Shard2   Shard3   Slave1   Slave2   Slave3  
20. 3 types •  Range sharding •  List sharding (Lookup table) •  Hash sharding
21. Range sharding •  Distributed by the range of Primary Key •  Example – Primary Key: user_id (1..1000) user_shard1 (1..500) user_shard2 (501..1000)
22. List sharding •  Distributed data by the attribute of the data •  Example: database of people in VN – Sharded by the city_name (Ha_Noi, Hai_Phong, Da_Nang,…)
23. Hash sharding (modulus) •  Distributed data by using a hash function on primary key. •  Example: primary_key mod N
24. Pros of Database Sharding •  Easy to scale (data, write I/O) •  Using commodity hardware •  Minimum effect when system failed
25. Cons of Database sharding •  You MUST implement by yourselves •  Operation is harder •  Handle join operation is very difficult •  Data denormalization – > Don’t do it because it’s COOL!
26. Database Sharding in MySQL
27. Sharding Solutions •  Application layer •  Storage layer •  Heavy middleware •  Lightweight middleware
28. Application layer •  Hibernate Shards •  HiveDB
29. Storage layer •  MySQL Spider – Requires to change storage engine of MySQL
30. Heavy Middleware •  Twitter Gizzard •  dbShards –  Each db has an agent
31. Lightweight Middleware •  Acts like a proxy •  Route the request •  Spock, CUBRID
32. You Will Do It Because You Have To … not because it’s Cool!
33. Q&A