Sharding Strategies

Let's explore the three main sharding strategies - Lookup, Range, and Hash - and dive into their unique characteristics, benefits, and ideal use cases with simple examples.

Transcript

1. Sharding S t r a t e g i e s Shard C Shard B Shard A Horizontal Scalability Thread/Shard per core Architecture
2. Partitioning data for enhanced scalability Sharding
3. Sharding
4. Sharding
5. Sharding
6. Sharding Horizontal Scalability
7. Sharding Scaling your system by adding more machines to handle increased load or data volume. Horizontal Scalability
8. Thread/Shard per core Architecture Scaling your system by adding more machines to handle increased load or data volume. Sharding Horizontal Scalability
9. Thread/Shard per core Architecture Distributing data for scalable performance Scaling your system by adding more machines to handle increased load or data volume. Sharding Horizontal Scalability
10. Sharding Strategies
11. The Lookup Strategy
12. Sharding distributes data across shards using a routing map that directs requests based on a specific key. allowing logical categorization/segregation The Lookup Strategy
13. Sharding Shard A The Lookup Strategy distributes data across shards using a routing map that directs requests based on a specific key. allowing logical categorization/segregation
14. Sharding Shard B Shard A The Lookup Strategy distributes data across shards using a routing map that directs requests based on a specific key. allowing logical categorization/segregation
15. Sharding Shard C Shard B Shard A The Lookup Strategy distributes data across shards using a routing map that directs requests based on a specific key. allowing logical categorization/segregation
16. Sharding Key Shard Tenant1:News Shard A Tenant1:* Shard B Tenant2:Products Shard C Tenant2:Users Shard C Tenant2:* Shard B Tenant3:* Shard A Tenant4:* Shard C Shard C Shard B Shard A The Lookup Strategy distributes data across shards using a routing map that directs requests based on a specific key. allowing logical categorization/segregation
17. Sharding Key Shard Tenant1:News Shard A Tenant1:* Shard B Tenant2:Products Shard C Tenant2:Users Shard C Tenant2:* Shard B Tenant3:* Shard A Tenant4:* Shard C “Tenant2:Users:Th30z:…” Shard C Shard B Shard A The Lookup Strategy distributes data across shards using a routing map that directs requests based on a specific key. allowing logical categorization/segregation
18. Sharding Key Shard Tenant1:News Shard A Tenant1:* Shard B Tenant2:Products Shard C Tenant2:Users Shard C Tenant2:* Shard B Tenant3:* Shard A Tenant4:* Shard C “Tenant2:Users:Th30z:…” -> getShard(“Tenant2:Users”) Shard C Shard B Shard A The Lookup Strategy distributes data across shards using a routing map that directs requests based on a specific key. allowing logical categorization/segregation
19. Sharding Key Shard Tenant1:News Shard A Tenant1:* Shard B Tenant2:Products Shard C Tenant2:Users Shard C Tenant2:* Shard B Tenant3:* Shard A Tenant4:* Shard C “Tenant2:Users:Th30z:…” -> getShard(“Tenant2:Users”) Shard C Shard B Shard A The Lookup Strategy distributes data across shards using a routing map that directs requests based on a specific key. allowing logical categorization/segregation = Shard C
20. Sharding The Lookup Strategy Key Shard Tenant1:News Shard A Tenant1:* Shard B Tenant2:Products Shard C Tenant2:Users Shard C Tenant2:* Shard B Tenant3:* Shard A Tenant4:* Shard C “Tenant2:Users:Th30z:…” -> getShard(“Tenant2:Users”) Shard C Shard B Shard A “Tenant2:Users:Th30z:…” distributes data across shards using a routing map that directs requests based on a specific key. allowing logical categorization/segregation = Shard C
21. Sharding Key Shard Tenant1:News Shard A Tenant1:* Shard B Tenant2:Products Shard C Tenant2:Users Shard C Tenant2:* Shard B Tenant3:* Shard A Tenant4:* Shard C “Tenant3:Users:Foo:…” -> getShard(“Tenant2:Users”) Shard C Shard B Shard A “Tenant2:Users:Th30z:…” “Tenant3:Users:Foo:…” The Lookup Strategy distributes data across shards using a routing map that directs requests based on a specific key. allowing logical categorization/segregation “Tenant2:Users:Th30z:…” -> getShard(“Tenant2:Users”) = Shard C = Shard A
22. Sharding Key Shard Tenant1:News Shard A Tenant1:* Shard B Tenant2:Products Shard C Tenant2:Users Shard C Tenant2:* Shard B Tenant3:* Shard A Tenant4:* Shard C “Tenant2:Products:PC:…” -> getShard(“Tenant2:Products”) Shard C Shard B Shard A “Tenant2:Users:Th30z:…” “Tenant3:Users:Foo:…” “Tenant2:Products:PC:…” The Lookup Strategy distributes data across shards using a routing map that directs requests based on a specific key. allowing logical categorization/segregation “Tenant2:Users:Th30z:…” -> getShard(“Tenant2:Users”) = Shard C “Tenant3:Users:Foo:…” -> getShard(“Tenant2:Users”) = Shard A = Shard C
23. Sharding Key Shard Tenant1:News Shard A Tenant1:* Shard B Tenant2:Products Shard C Tenant2:Users Shard C Tenant2:* Shard B Tenant3:* Shard A Tenant4:* Shard C Shard C Shard B Shard A “Tenant2:Users:Th30z:…” “Tenant3:Users:Foo:…” “Tenant2:Products:PC:…” “Tenant2:News:IPO:…” -> getShard(“Tenant2:News”) “Tenant2:News:IPO:…” The Lookup Strategy distributes data across shards using a routing map that directs requests based on a specific key. allowing logical categorization/segregation “Tenant2:Users:Th30z:…” -> getShard(“Tenant2:Users”) = Shard C “Tenant3:Users:Foo:…” -> getShard(“Tenant2:Users”) = Shard A “Tenant2:Products:PC:…” -> getShard(“Tenant2:Products”) = Shard C = Shard B
24. Sharding The Lookup Strategy Key Shard Tenant1:News Shard A Tenant1:* Shard B Tenant2:Products Shard C Tenant2:Users Shard C Tenant2:* Shard B Tenant3:* Shard A Tenant4:* Shard C Shard C Shard B Shard A “Tenant2:Users:Th30z:…” -> getShard(“Tenant2:Users”) = Shard C “Tenant2:Users:Th30z:…” “Tenant3:Users:Foo:…” -> getShard(“Tenant2:Users”) = Shard A “Tenant3:Users:Foo:…” “Tenant2:Products:PC:…” -> getShard(“Tenant2:Products”) = Shard C “Tenant2:Products:PC:…” “Tenant2:News:IPO:…” -> getShard(“Tenant2:News”) = Shard B “Tenant2:News:IPO:…” “Tenant1:Products:Car:…” -> getShard(“Tenant1:Products”) = Shard B “Tenant1:Products:Car:…” distributes data across shards using a routing map that directs requests based on a specific key. allowing logical categorization/segregation
25. Sharding The Lookup Strategy Key Shard Tenant1:News Shard A Tenant1:* Shard B Tenant2:Products Shard C Tenant2:Users Shard C Tenant2:* Shard B Tenant3:* Shard A Tenant4:* Shard C Shard C Shard B Shard A “Tenant2:Users:Th30z:…” -> getShard(“Tenant2:Users”) = Shard C “Tenant2:Users:Th30z:…” “Tenant3:Users:Foo:…” -> getShard(“Tenant2:Users”) = Shard A “Tenant3:Users:Foo:…” “Tenant2:Products:PC:…” -> getShard(“Tenant2:Products”) = Shard C “Tenant2:Products:PC:…” “Tenant2:News:IPO:…” -> getShard(“Tenant2:News”) = Shard B “Tenant2:News:IPO:…” “Tenant1:Products:Car:…” -> getShard(“Tenant1:Products”) = Shard B “Tenant1:Products:Car:…” distributes data across shards using a routing map that directs requests based on a specific key. allowing logical categorization/segregation
26. The Range Strategy
27. Sharding distributes data across shards in a sorted manner, allowing efficient sequential scans and range-based queries. The Range Strategy
28. Sharding The Range Strategy distributes data across shards in a sorted manner, allowing efficient sequential scans and range-based queries. Shard 1 Shard 2 Shard 3
29. Sharding Shard 1 Key Shard aaaa-jjjj Shard 1 kkkk-ssss Shard 2 tttt-zzzz Shard 3 aaaa-jjjj The Range Strategy Shard 2 Shard 3 kkkk-ssss tttt-zzzz distributes data across shards in a sorted manner, allowing efficient sequential scans and range-based queries.
30. Sharding “Hello” Key Shard aaaa-jjjj kkkk-ssss tttt-zzzz aaaa-jjjj The Range Strategy kkkk-ssss tttt-zzzz distributes data across shards in a sorted manner, allowing efficient sequential scans and range-based queries. Shard 1 Shard 1 Shard 2 Shard 3 Shard 2 Shard 3
31. Sharding “Hello” -> getShard(“Hello”) Key Shard aaaa-jjjj kkkk-ssss tttt-zzzz aaaa-jjjj The Range Strategy kkkk-ssss tttt-zzzz distributes data across shards in a sorted manner, allowing efficient sequential scans and range-based queries. Shard 1 Shard 1 Shard 2 Shard 3 Shard 2 Shard 3
32. Sharding Hello “Hello” -> getShard(“Hello”) = Shard 1 Key Shard aaaa-jjjj kkkk-ssss tttt-zzzz aaaa-jjjj kkkk-ssss tttt-zzzz The Range Strategy distributes data across shards in a sorted manner, allowing efficient sequential scans and range-based queries. Shard 1 Shard 1 Shard 2 Shard 3 Shard 2 Shard 3
33. Sharding aaaa-jjjj kkkk-ssss tttt-zzzz Hello World “World” -> getShard(“World”) = Shard 3 Key Shard aaaa-jjjj kkkk-ssss tttt-zzzz The Range Strategy distributes data across shards in a sorted manner, allowing efficient sequential scans and range-based queries. Shard 1 Shard 1 Shard 2 Shard 3 Shard 2 Shard 3 “Hello” -> getShard(“Hello”) = Shard 1
34. Sharding aaaa-jjjj kkkk-ssss tttt-zzzz Hello World Test “Test” -> getShard(“Test”) = Shard 3 Key Shard aaaa-jjjj kkkk-ssss tttt-zzzz The Range Strategy distributes data across shards in a sorted manner, allowing efficient sequential scans and range-based queries. Shard 1 Shard 1 Shard 2 Shard 3 Shard 2 Shard 3 “World” -> getShard(“World”) = Shard 3 “Hello” -> getShard(“Hello”) = Shard 1
35. Sharding aaaa-jjjj kkkk-ssss tttt-zzzz Hello World Test Demo “Demo” -> getShard(“Demo”) = Shard 1 Key Shard aaaa-jjjj kkkk-ssss tttt-zzzz The Range Strategy distributes data across shards in a sorted manner, allowing efficient sequential scans and range-based queries. Shard 1 Shard 1 Shard 2 Shard 3 Shard 2 Shard 3 “Test” -> getShard(“Test”) = Shard 3 “World” -> getShard(“World”) = Shard 3 “Hello” -> getShard(“Hello”) = Shard 1
36. Sharding aaaa-jjjj kkkk-ssss tttt-zzzz Hello World Test Demo Showcase “Showcase” -> getShard(“Showcase”) = Shard 2 Key Shard aaaa-jjjj kkkk-ssss tttt-zzzz The Range Strategy distributes data across shards in a sorted manner, allowing efficient sequential scans and range-based queries. Shard 1 Shard 1 Shard 2 Shard 3 Shard 2 Shard 3 “Demo” -> getShard(“Demo”) = Shard 1 “Test” -> getShard(“Test”) = Shard 3 “World” -> getShard(“World”) = Shard 3 “Hello” -> getShard(“Hello”) = Shard 1
37. Sharding aaaa-jjjj kkkk-ssss tttt-zzzz Hello World Test Demo Showcase “Showcase” -> getShard(“Showcase”) = Shard 2 “Demo” -> getShard(“Demo”) = Shard 1 “Test” -> getShard(“Test”) = Shard 3 “World” -> getShard(“World”) = Shard 3 “Hello” -> getShard(“Hello”) = Shard 1 The Range Strategy Key Shard aaaa-jjjj kkkk-ssss tttt-zzzz distributes data across shards in a sorted manner, allowing efficient sequential scans and range-based queries. Shard 1 Shard 1 Shard 2 Shard 3 Shard 2 Shard 3
38. Sharding aaaa-jjjj kkkk-ssss tttt-zzzz Hello World Test Demo Showcase “Showcase” -> getShard(“Showcase”) = Shard 2 “Demo” -> getShard(“Demo”) = Shard 1 “Test” -> getShard(“Test”) = Shard 3 “World” -> getShard(“World”) = Shard 3 “Hello” -> getShard(“Hello”) = Shard 1 The Range Strategy Key Shard aaaa-jjjj kkkk-ssss tttt-zzzz distributes data across shards in a sorted manner, allowing efficient sequential scans and range-based queries. Get anything from A to E -> getShard(A-E) = Shard 1 Shard 1 Shard 1 Shard 2 Shard 3 Shard 2 Shard 3
39. Hash Strategy
40. Sharding distributes data across shards based on a hash function, ensuring even data distribution and load balancing. The Hash Strategy
41. Sharding Shard 0 The Hash Strategy Shard 1 Shard 2 distributes data across shards based on a hash function, ensuring even data distribution and load balancing.
42. Sharding Shard = Shard 0 The Hash Strategy Shard 1 Shard 2 distributes data across shards based on a hash function, ensuring even data distribution and load balancing.
43. Sharding Shard = (Key) hash Shard 0 The Hash Strategy Shard 1 Shard 2 distributes data across shards based on a hash function, ensuring even data distribution and load balancing.
44. Sharding Shard = % NumShards (Key) hash Shard 0 The Hash Strategy Shard 1 Shard 2 distributes data across shards based on a hash function, ensuring even data distribution and load balancing.
45. Sharding xxh3 murmur3 sipHash spookyHash city64 … Shard = % NumShards (Key) hash Non-cryptographic hash functions Shard 0 The Hash Strategy Shard 1 Shard 2 distributes data across shards based on a hash function, ensuring even data distribution and load balancing.
46. Sharding xxh3 murmur3 sipHash spookyHash city64 … Shard = % NumShards (Key) hash Non-cryptographic hash functions “Hello” Shard 0 The Hash Strategy Shard 1 Shard 2 distributes data across shards based on a hash function, ensuring even data distribution and load balancing.
47. Sharding xxh3 murmur3 sipHash spookyHash city64 … Shard = % NumShards (Key) hash Non-cryptographic hash functions “Hello” -> hash(“Hello”) % 3 Shard 0 The Hash Strategy Shard 1 Shard 2 distributes data across shards based on a hash function, ensuring even data distribution and load balancing.
48. Sharding xxh3 murmur3 sipHash spookyHash city64 … Shard = % NumShards (Key) hash Non-cryptographic hash functions “Hello” -> hash(“Hello”) % 3 = Shard 1 Shard 0 The Hash Strategy Shard 1 Shard 2 distributes data across shards based on a hash function, ensuring even data distribution and load balancing.
49. Sharding Shard 0 Shard 1 Shard 2 xxh3 murmur3 sipHash spookyHash city64 … Shard = % NumShards (Key) hash Non-cryptographic hash functions Hello “Hello” -> hash(“Hello”) % 3 = Shard 1 The Hash Strategy distributes data across shards based on a hash function, ensuring even data distribution and load balancing.
50. Sharding Shard 0 Shard 1 Shard 2 “World” -> hash(“World”) % 3 = Shard 0 xxh3 murmur3 sipHash spookyHash city64 … Shard = % NumShards (Key) hash Non-cryptographic hash functions Hello World “Hello” -> hash(“Hello”) % 3 = Shard 1 The Hash Strategy distributes data across shards based on a hash function, ensuring even data distribution and load balancing.
51. Sharding Shard 0 Shard 1 Shard 2 “Test” -> hash(“Test”) % 3 = Shard 0 xxh3 murmur3 sipHash spookyHash city64 … Shard = % NumShards (Key) hash Non-cryptographic hash functions Hello World Test “World” -> hash(“World”) % 3 = Shard 0 “Hello” -> hash(“Hello”) % 3 = Shard 1 The Hash Strategy distributes data across shards based on a hash function, ensuring even data distribution and load balancing.
52. Sharding Shard 0 Shard 1 Shard 2 “Demo” -> hash(“Demo”) % 3 = Shard 2 xxh3 murmur3 sipHash spookyHash city64 … Shard = % NumShards (Key) hash Non-cryptographic hash functions Hello World Test Demo “Test” -> hash(“Test”) % 3 = Shard 0 “World” -> hash(“World”) % 3 = Shard 0 “Hello” -> hash(“Hello”) % 3 = Shard 1 The Hash Strategy distributes data across shards based on a hash function, ensuring even data distribution and load balancing.
53. Sharding Shard 0 Shard 1 Shard 2 xxh3 murmur3 sipHash spookyHash city64 … Shard = % NumShards (Key) hash Non-cryptographic hash functions Hello World Test Demo “Showcase” -> hash(“Showcase”) % 3 = Shard 1 Showcase “Demo” -> hash(“Demo”) % 3 = Shard 2 “Test” -> hash(“Test”) % 3 = Shard 0 “World” -> hash(“World”) % 3 = Shard 0 “Hello” -> hash(“Hello”) % 3 = Shard 1 The Hash Strategy distributes data across shards based on a hash function, ensuring even data distribution and load balancing.
54. Sharding Shard 0 Shard 1 Shard 2 The Hash Strategy xxh3 murmur3 sipHash spookyHash city64 … Shard = % NumShards (Key) hash Non-cryptographic hash functions Hello World Test Demo “Showcase” -> hash(“Showcase”) % 3 = Shard 1 Showcase “Demo” -> hash(“Demo”) % 3 = Shard 2 “Test” -> hash(“Test”) % 3 = Shard 0 “World” -> hash(“World”) % 3 = Shard 0 “Hello” -> hash(“Hello”) % 3 = Shard 1 distributes data across shards based on a hash function, ensuring even data distribution and load balancing.
55. The Hash Strategy Shard = % NumShards (Key) hash Sharding Thread/Shard per core Architecture
56. Thread/Shard per core Architecture The Hash Strategy Shard = % NumShards (Key) hash Sharding Consistent Lookup Performance
57. Thread/Shard per core Architecture The Hash Strategy Shard = % NumShards (Key) hash Sharding Consistent Lookup Performance Balanced Data Distribution
58. Thread/Shard per core Architecture The Hash Strategy Shard = % NumShards (Key) hash Sharding Consistent Lookup Performance Balanced Data Distribution
59. Thread/Shard per core Architecture The Hash Strategy Shard = % NumShards (Key) hash Sharding Consistent Lookup Performance Balanced Data Distribution
60. Thread/Shard per core Architecture The Hash Strategy Shard = % NumShards (Key) hash Sharding Consistent Lookup Performance Balanced Data Distribution
61. Thread/Shard per core Architecture The Hash Strategy Shard = % NumShards (Key) hash Sharding Consistent Lookup Performance Balanced Data Distribution
62. Thread/Shard per core Architecture The Hash Strategy Shard = % NumShards (Key) hash Sharding Consistent Lookup Performance Balanced Data Distribution
63. Thread/Shard per core Architecture The Hash Strategy Shard = % NumShards (Key) hash Sharding Consistent Lookup Performance Balanced Data Distribution
64. Thread/Shard per core Architecture The Hash Strategy Shard = % NumShards (Key) hash Sharding Consistent Lookup Performance Balanced Data Distribution
65. Thread/Shard per core Architecture The Hash Strategy Shard = % NumShards (Key) hash Sharding Consistent Lookup Performance Balanced Data Distribution
66. Thread/Shard per core Architecture The Hash Strategy Shard = % NumShards (Key) hash Sharding Consistent Lookup Performance Balanced Data Distribution
67. Thread/Shard per core Architecture The Hash Strategy Shard = % NumShards (Key) hash Sharding Consistent Lookup Performance Balanced Data Distribution
68. Thread/Shard per core Architecture The Hash Strategy Shard = % NumShards (Key) hash Sharding Consistent Lookup Performance Balanced Data Distribution
69. Thread/Shard per core Architecture The Hash Strategy Shard = % NumShards (Key) hash Sharding Consistent Lookup Performance Balanced Data Distribution
70. Thread/Shard per core Architecture The Hash Strategy Shard = % NumShards (Key) hash Sharding Consistent Lookup Performance Balanced Data Distribution
71. Sharding Mix multiple Strategies Lookup Strategy to isolate a Tenant Resources Hash Strategy to distribute data across them +
72. Sharding Easy when numShards/machines are fixed
73. Sharding Easy when numShards/machines are fixed Not so Easy When “Machines are moving”
74. Sharding Easy when numShards/machines are fixed Not so Easy When “Machines are moving” - Data Migration - Adjust Shard Mapping Consistent Hashing - Rebalance Data