Probabilistic Data Structures

Bloom Filters, Count Min Sketches, LogLog, SuperLogLog, HyperLogLog, HyperLogLog++, BigData, Analytics

Transcript

1. Probabilistic Data Structures
2. Standard Data Structures • List • Set • Map • Queue • Priority Queue • Stack • Deque • Sorted Set • Sorted Map • Multiset • Multimap
3. Probabilistic Data Structures • Bloom Filter • Count-Min Sketch • LogLog
4. Why? - Space Efficiency / Big Data
5. Hashing - Definition “A hash function is any function that can be used to map digital data of arbitrary size to digital data of fixed size, with slight differences in input data producing very big differences in output data.” - 62 United States Attorneys’ Bulletin 44-82
6. Hashing - Properties • Deterministic • Uniform • Defined Range • Continuity • Non-invertible
7. Membership “Does this data set contain a particular element?”
8. Membership - Use Cases • NoSQL Databases ◦ Does this data segment contain this key? • Chrome ◦ Is this url malicious? • Squid Caching Proxy ◦ Does a cached version exist?
9. Bloom Filter - What is it? • Can answer the membership question • False positives are possible • False negatives are not possible
10. Bloom Filter - Implementation • Setup ◦ Set of k different hash functions ◦ Bit array of size m with all bits initialized to 0 • Insertion ◦ Hash the input with all k hash functions ◦ Set each corresponding bit in the array to 1 • Query ◦ Hash the input with all k hash functions ◦ All corresponding bits are set to 1 ? “Yes” : “No”
11. Bloom Filter - Insertion • Hash1 = 4 • Hash2 = 7 • Hash3 = 8 • Hash4 = 14 ‘This is my input’ 0 0 0 1 0 0 1 1 0 0 0 0 0 1 0 0
12. Bloom Filter - Insertion • Hash1 = 5 • Hash2 = 7 • Hash3 = 9 • Hash4 = 11 ‘This is another input’ 0 0 0 1 1 0 1 1 1 0 1 0 0 1 1 0
13. Bloom Filter - Query • Hash1 = 4 • Hash2 = 7 • Hash3 = 8 • Hash4 = 14 ‘This is my input’ => Yes 0 0 0 1 1 0 1 1 1 0 1 0 0 1 1 0
14. Bloom Filter - Query • Hash1 = 2 • Hash2 = 5 • Hash3 = 10 • Hash4 = 12 ‘I have not seen this’ => No 0 0 0 1 1 0 1 1 1 0 1 0 0 1 1 0
15. Bloom Filter - Collision • Hash1 = 5 • Hash2 = 7 • Hash3 = 8 • Hash4 = 11 ‘Is this in the set?’ => Yes (oops) 0 0 0 1 1 0 1 1 1 0 1 0 0 1 1 0
16. Heavy Hitters “What are the most frequent elements?”
17. Heavy Hitters - Use Cases • Top-k of anything ◦ Users who X the most ◦ Users who score the most ◦ Leaderboards • DDOS detection / Rate Limiting ◦ Track the abusive IP addresses
18. Count Min Sketch - What is it? • Can answer the heavy hitters question • Answer comes with ◦ estimation error (count will be ± some value) ◦ probability (x% of the time)
19. Count Min Sketch - Implementation • Setup ◦ Set of k different hash functions ◦ Matrix of size m x k with all values initialized to 0 • Insertion ◦ Hash the input with all k hash functions ◦ Increment each corresponding value by 1 • Query ◦ Hash the input with all k hash functions ◦ Return the minimum of all the corresponding values
20. Count Min Sketch - Insertion • Hash1 = 4 • Hash2 = 7 • Hash3 = 8 • Hash4 = 14 ‘This is my input’ (5 times) 0 0 0 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0 0
21. Count Min Sketch - Insertion ‘This is another input’ (2 times) • Hash1 = 5 • Hash2 = 7 • Hash3 = 9 • Hash4 = 11 0 0 0 5 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 5 0 0
22. Count Min Sketch - Query 0 0 0 5 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 5 0 0 • Hash1 = 4 • Hash2 = 7 • Hash3 = 8 • Hash4 = 14 ‘This is my input’ => 5
23. Cardinality “How many distinct elements are in the data set?”
24. Cardinality - Use Cases • Count Distinct ◦ Number of users who X ◦ Number of users who logged in ◦ Daily active users
25. Linear Counting - What is it? • Can answer the cardinality question • Answer comes with ◦ standard error (percentage error rate)
26. Linear Counting - Implementation • Setup ◦ A hash function ◦ Bit array of size m with all bits initialized to 0 • Insertion ◦ Hash the input with the hash function ◦ Set the corresponding bit in the array to 1 • Query ◦ Count the number of 1s in the array
27. Linear Counting - Insertion • Hash = 4 ‘This is my input’ 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0
28. Linear Counting - Insertion • Hash = 9 ‘This is another input’ 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0
29. Linear Counting - Query 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0
30. LogLog Counting - What is it? • Can answer the cardinality question • Much more efficient than Linear Counting • Answer comes with ◦ standard error (percentage error rate)
31. LogLog Counting - Intuition • Hashing the input creates a (pseudo) random, evenly distributed output • In such a random, evenly distributed series, you would expect to see a value with k leading zeros once in every 2^k elements
32. LogLog Counting - Implementation • Setup ◦ def hash(input) ◦ m = 2^k ◦ buckets = [0] * m • Insertion ◦ hashed = hash(input) ◦ index = get_bits(hashed, 0, k) ◦ observation = get_bits(hashed, k, hashed.size) ◦ rank = num_leading_zeros(observation) ◦ buckets[index] = max(buckets[index], rank) • Query ◦ 2^(sum(buckets)/ m) * m * 0.79402
33. LogLog Counting - Insertion ‘This is my input’ hashed = 0b10000000001010101010000111101 index = 0b1000 (i.e. 8) observation = 0b0000001010101010000111101 rank = 6 buckets[8] = max(buckets[8], 6)
34. LogLog Counting - Variants • SuperLogLog ◦ Outliers skew results - Toss out largest 30% • HyperLogLog ◦ Use harmonic mean instead of geometric mean ◦ Special case logic for low cardinality • HyperLogLog++ ◦ Google’s version of HLL ◦ Use 64-bit hash function to count larger cardinalities ◦ Different special case logic for low cardinality
35. Resources • http://highlyscalable.wordpress.com/2012/05/01/probabilistic-structures-web-analytics-data-mining/ • http://research.neustar.biz/tag/probabilistic-sketching/ • http://blog.notdot.net/2012/09/Dam-Cool-Algorithms-Cardinality-Estimation • http://en.wikipedia.org/wiki/Category:Probabilistic_data_structures • https://gist.github.com/debasishg/8172796