HyperLogLog

Lightning Talk on HyperLogLog, a probabilistic data structure to count unique elements in a collection.

Further Reading:
http://antirez.com/news/75
http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf

Transcript

1. Robin Roth PyCon.DE 2018 HyperLogLog Know Your Data Structures
2. Counting Things Goal • Count unique elements in a collection (count-distinct problem) Idea • Use a set: len(set(mydata)) Downsides • O(n) space needed • For large datasets limited by memory • No “streaming” mode Want • Algorithm to deal with billions of entries
3. LogLog Idea • Calculate approximate result • Use less storage O(1), O(1/) Algorithm • Hash object => 64 bit representation • count the number of leading 0s • estimate: 2**max(num_zeros(elem) for elem in data) 00110110 => 2 01010111 => 1 00010110 => 3 … max => 3 => 23 = 8 elements in set Storage: 1 number ✓ CPU: Hash + Max ✓ Accuracy: ❌ Works with streaming data
4. HyperLogLog Idea • Improve accuracy Algorithm • Hash object => 64 bit representation • Take the first bits to choose a bucket • Take the remaining bits and count leading zero • estimate: average(bucket_max) 0011 0110 => bucket[0011] = 1 0101 0111 => bucket[0101] = 1 0001 0010 => bucket[0010] = 2 … Storage: m buckets CPU: bitop + average ✓ Accuracy: 1/sqrt(m) Still works with streaming Trivial to parallelize
5. HyperLogLog in Redis Memory: 12k ✓ (16k buckets of 6 bits) Error: <1% ✓ Number of objects: unlimited* (< 250) Usage: PFADD languages python java lisp PFCOUNT languages PFMERGE languages interpreted compiled Further Reading: http://antirez.com/news/75 http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf Know your data structures