Redisconf 2018: Probabilistic Data Structures

Real Time Log Analysis using Probabilistic Data Structures in Redis. Presented at Redisconf 2018.

Transcript

1. Probabilistic Data Structures in Redis Srinivasan Rangarajan @cnu
2. Srinivasan Rangarajan • srinivasanr@gmail.com • @cnu • https://cnu.name
3. Log Analysis
4. User Events Kinesis Firehose ELK
5. Sample Event Data { "ip": "123.123.123.123", "client_id": 232, "user_id": "35827", "email": "foo@example.com", "product_id": "ABC-12345", "image_id": 3, "action": "pageview", "datetime": "2017-06-29T12:42:53Z", }
6. Challenges • 100s of Millions of events processed every day • Peak of ~10 Million events in an hour • Needed Real Time processing • Low memory/storage requirements
8. User Events Kinesis Firehose ELK AWS Lambda Redis
9. Cost Accuracy Scale
10. Probabilistic Data Structures
11. xkcd/1132
12. Loading Modules • ./redis-server --loadmodule /path/to/module.so • redis.conf loadmodule /path/to/module.so • MODULE LOAD /path/to/module.so
13. Execute custom commands >>> import redis >>> r = redis.Redis() >>> out = r.execute_command('CMD param1 param2')
14. Data Structures • HyperLogLog • TopK • CountMinSketch • Bloom Filters
15. HyperLogLog Count the Cardinality of a Set
16. Count Unique Visitors/hour >>> r.pfadd('users:2017083120', 123, 456, 789) 1 >>> r.pfcount('users:2017083120') 3 >>> r.pfadd('users:2017083120', 456) 0
17. Merge Hourly into Daily >>> r.pfadd('users:2017083121', 121, 454, 787) 1 >>> r.pfmerge('users:20170831', 'users:2017083120', 'users:2017083121') True >>> r.pfcount('users:20170831’) 6
18. Links • https://redis.io/commands#hyperloglog • http://antirez.com/news/75
19. TopK Get top K elements in a set
20. Top K IP Addresses >>> r.execute_command('TOPK.ADD ip:20170831 3 123.45.67.89') >>> r.execute_command('TOPK.ADD ip:20170831 3 123.45.67.90') >>> r.execute_command('TOPK.ADD ip:20170831 3 123.45.67.91') 1L >>> r.execute_command('TOPK.ADD ip:20170831 3 123.45.67.92') -1L
21. Top K IP Addresses >>> r.zrange('ip:20170831’, 0, -1, withscores=True) [('TOPK:1.0.1:1.0:\xff\xff\xff\xff\xff\xff\xff\xff\x04\x00\x0 0\x00\x00\x00\x00\x00', 1.0), ('123.45.67.89', 1.0), ('123.45.67.90', 1.0), ('123.45.67.92', 2.0)]
22. Links • https://github.com/RedisLabsModules/topk
23. CountMinSketch Count the frequency of items
24. 1 2 3 4 h1 0 0 0 0 h2 0 0 0 0 h3 0 0 0 0
25. 1 2 3 4 h1 1 0 0 0 h2 0 1 0 0 h3 0 0 1 0 h1(s1) = 1; h2(s1) = 2; h3(s1) = 3
26. 1 2 3 4 h1 1 0 0 1 h2 0 1 0 1 h3 0 0 1 1 h1(s2) = 4; h2(s2) = 4; h3(s2) = 4
27. 1 2 3 4 h1 2 1 1 1 h2 0 1 0 1 h3 0 0 1 1 h1(s3) = 1; h2(s3) = 1; h3(s3) = 1
28. User Pageview counter >>> r.execute_command('CMS.INCRBY u:pv:20170831 123 1 456 3 789 2 234 1 567 1') 'OK' >>> r.execute_command('CMS.QUERY u:pv:20170831 123 456 789 234 567') [1L, 3L, 2L, 1L, 1L]
29. Merge Counters >>> r.execute_command('CMS.MERGE u:pv:201708 3 u:pv:20170829 u:pv:20170830 u:pv:20170831') 'OK'
30. Links • https://github.com/RedisLabsModules/countminsketch • https://redislabs.com/blog/count-min-sketch-the-art-and-science- of-estimating-stuff/
31. Bloom Filters Test Membership in a Set
32. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Empty Bit Array
33. 0 0 1 0 0 1 0 0 1 0 0 0 0 0 0 0 h1(item1) = 2; h2(item1) = 5; h3(item1) = 8 Insert Item 1
34. 0 0 1 0 0 1 0 1 1 0 1 0 0 0 0 0 h1(item2) = 7; h2(item2) = 8; h3(item2) = 10 Insert Item 2
35. 0 0 1 0 0 1 0 1 1 0 1 0 0 0 0 0 h1(item3) = 2; h2(item3) = 11; h3(item3) = 0 Check Item3
36. 0 0 1 0 0 1 0 1 1 0 1 0 0 0 0 0 h1(item4) = 10; h2(item4) = 8; h3(item4) = 7 Check Item4
37. Bloom Filter returns What it means False Definitely not in the set True Maybe in the set
38. Check User Session >>> r.execute_command('BF.MADD u:sess:20170831 123 456 789') [1L, 1L, 1L] >>> r.execute_command('BF.EXISTS u:sess:20170831 456') 1L >>> r.execute_command('BF.EXISTS u:sess:20170831 234') 0L
39. Links • https://github.com/RedisLabsModules/rebloom • https://redislabs.com/blog/rebloom-bloom-filter-datatype-redis/ • https://github.com/kristoff-it/redis-cuckoofilter - Better than bloom filters
40. “An 80% solution today is much better than an 100% solution tomorrow.”
41. Thank You https://cnu.name/talks/redisconf-2018/