Consistent Hashing

What is Consistent Hashing? What’s the motivation behind it, and how distributed caches and storages are taking advantage of it?
We’ll first review the general concept of hashing and its purpose, followed by a description of distributed hashing and the problems it entails. In turn, that will lead us to us to our title subject and its benefits.

Transcript

1. Consistent Hashing Teodora Petkova
2. Introduction ü Teodora Petkova ü Lidl Digital CoE Bulgaria ü Team Battery ü Java Developer 3>4 years ü @teodorrrra 7+ followers Hey everybody!
3. Agenda Ø The Motivation Behind Ø Real-World Applications Ø Distributed Hashing Ø A Simple Approach Ø A Not So Simple Approach using Consistent Hashing
4. The Motivation Behind Web Caching
5. The Motivation Behind Distributed Cache
6. The Motivation Behind Distributed Storage
7. Real-World Applications Who uses Consistent Hashing? ØAmazon Dynamo ØAkamai ØCassandra ØTwitter Twemproxy ØVarnish Self-Routing Cluster Ø…
8. Real – World Applications The Dynamo Paper from 2007
9. Real – World Applications Varnish Self Routing Cluster
10. Distributed Hashing Let’s solve the problem J
11. Distributed Hashing Hash Function We could first apply a hash function to ProductIDs and URLs. What this would give us is: 1) a more unified problem 2) working with fixed-size values hashFunction(1234)=0 hashFunction(2345)=1 … hashFunction(XXXX)=7
12. A Simple Approach To Distributed Hashing Hashes ∈ {0...7} N = 4 (servers) ServerIndex ∈ {0...3} ServerIndex = Hash % N
13. A Simple Approach To Distributed Hashing The Rehashing Problem: Adding a server Hashes ∈ {0...7} N = 5 (servers) ServerIndex ∈ {0...4} ServerIndex = Hash % N
14. A Simple Approach To Distributed Hashing The Rehashing Problem: Removing a server Hashes ∈ {0...7} N = 3 (servers) ServerIndex ∈ {0...2} ServerIndex = Hash % N
15. A Simple Approach To Distributed Hashing Results ü A server can be dynamically added or removed qMinimal amount of data is moved between the servers when we add/remove a server
16. Consistent Hashing Introduction
17. Consistent Hashing Adding a server
18. Consistent Hashing Removing a server
19. Consistent Hashing Even better!
20. Consistent Hashing Results Consistent Hashing helps distributed caches and storages with the following things: ü A server can be dynamically added or removed ü Minimal amount of data is moved between the servers when we add/remove a server
21. Resources Ø https://web.stanford.edu/class/cs168/l/l1.pdf Ø https://info.varnish-software.com/blog/creating-self-routing-varnish-cluster Ø https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html Ø https://www.toptal.com/big-data/consistent-hashing