Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Probabilistic Data Structures
Search
cnu
September 03, 2017
Programming
0
600
Probabilistic Data Structures
Learn how to use Probabilistic Data Structures and modules in Redis v4 to analyse logs.
cnu
September 03, 2017
Tweet
Share
More Decks by cnu
See All by cnu
Redisconf 2018: Probabilistic Data Structures
cnu
1
950
The Rocky Road from Monolithic to Microservices Architecture
cnu
0
1k
AWS Lambda - Pycon India 2016
cnu
0
480
ZeroMQ - PyCon India 2013
cnu
2
1.5k
Other Decks in Programming
See All in Programming
事業KPIを基に価値の解像度を上げる
nealle
0
190
漸進。
ssssota
0
450
知識0からカンファレンスやってみたらこうなった!
syossan27
5
320
Use Perl as Better Shell Script
karupanerura
0
600
OpenNext + Hono on Cloudflare でイマドキWeb開発スタックを実現する
rokuosan
0
100
當開發遇上包裝:AI 如何讓產品從想法變成商品
clonn
0
2.3k
Interface vs Types ~型推論が過多推論~
hirokiomote
1
220
Building an Application with TDD, DDD and Hexagonal Architecture - Isn't it a bit too much?
mufrid
0
370
TypeScript エンジニアが Android 開発の世界に飛び込んだ話
yuisakamoto
6
900
List Unfolding - 'unfold' as the Computational Dual of 'fold', and how 'unfold' relates to 'iterate'"
philipschwarz
PRO
0
110
「MCPを使ってる人」が より詳しくなるための解説
yamaguchidesu
0
520
AIにコードを生成するコードを作らせて、再現性を担保しよう! / Let AI generate code to ensure reproducibility
yamachu
7
6k
Featured
See All Featured
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
The Straight Up "How To Draw Better" Workshop
denniskardys
233
140k
StorybookのUI Testing Handbookを読んだ
zakiyama
30
5.8k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
30
2.4k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.3k
How to Ace a Technical Interview
jacobian
276
23k
Why You Should Never Use an ORM
jnunemaker
PRO
56
9.4k
Docker and Python
trallard
44
3.4k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
32
5.8k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
252
21k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
15
880
Transcript
Probabilistic Data Structures in Redis
Srinivasan Rangarajan Head of Engineering
Srinivasan Rangarajan @cnu https://cnu.name
Agenda • Log Analysis • Redis V4 • Probabilistic Data
Structures
Log Analysis
Challenges • 100s of Millions of events processed every day
• Peak of ~10 Million events in an hour • Needs Realtime processing • Memory/Storage Requirements
Cost Accuracy Scale
None
Sample Event Data { "ip": "123.123.123.123", "client_id": 232, "user_id": "35827",
"email": "foo@example.com", "product_id": "ABC-12345", "image_id": 3, "action": "pageview", "datetime": "2017-06-29T12:42:53Z", }
None
Redis Version 4 • Module system • Better Replication •
Cache eviction Improvements • Non-Blocking DEL and FLUSH* commands • Mixed RDB-AOF persistence format • MEMORY DOCTOR
Modules mikicon NounProject
Loading Modules • ./redis-server --loadmodule /path/to/module.so • redis.conf loadmodule /path/to/module.so
• MODULE LOAD /path/to/module.so
Execute a custom command
Probabilistic Data Structures
There are three kinds of people in the world. 1.
Those who can count. 2. Those who can’t count.
There are three kinds of people in the world. data
structures 1. Those who can count. 2. Those who can’t count. 3. Those who count approximately.
None
Advantage: Huge Memory Savings
3 Data Structures
HyperLogLog Count the Cardinality of a Set http://antirez.com/news/75
Count Unique Visitor / hour
Merge Hourly into Daily
TopK Get Top k Elements in a set https://github.com/RedisLabsModules/topk
Top k IP Addresses
None
CountMinSketch Count the frequency of items https://github.com/RedisLabsModules/countminsketch
User Pageview counter
Bloom Filters Test membership in a set https://github.com/RedisLabsModules/rebloom
Bloom Filters False Positives False Negatives
User Session checking
~3 Data Structures • HyperLogLog • TopK • CountMinSketch •
BloomFilter
Thank you
Follow me @cnu