Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Probabilistic Data Structures
Search
cnu
September 03, 2017
Programming
0
520
Probabilistic Data Structures
Learn how to use Probabilistic Data Structures and modules in Redis v4 to analyse logs.
cnu
September 03, 2017
Tweet
Share
More Decks by cnu
See All by cnu
Redisconf 2018: Probabilistic Data Structures
cnu
1
850
The Rocky Road from Monolithic to Microservices Architecture
cnu
0
920
AWS Lambda - Pycon India 2016
cnu
0
420
ZeroMQ - PyCon India 2013
cnu
2
1.5k
Other Decks in Programming
See All in Programming
大規模マルチテナントを解決するYugabyteDBという選択肢
nnaka2992
1
250
20240706_CDKConf
takuyay0ne
0
1.2k
Trial
cairolibrary720
1
130
実用的かつリーズナブルな 「Azure × Gemini × LINE」~キャラクターBot 実装ライブデモ~
tomodo_ysys
1
170
Rubyのパフォーマンスプロファイリングの改善 / Enhancing performance profiling for Ruby
osyoyu
1
410
英語
s_shimotori
1
220
OpenAI/Gemini APIを使って EPUBを翻訳するCLIツールをつくってみた
tomiyan
0
790
なぜ宣言的 UI は壊れにくいのか / Why declarative UI is less fragile
uenitty
29
13k
CSC307 Lecture 06
javiergs
PRO
0
360
How to use Macrobenchmark
veronikapj
0
160
CSC307 Lecture 13
javiergs
PRO
0
150
Ruby メモリ管理 プログラミング
megmogmog1965
0
130
Featured
See All Featured
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
13
430
The Illustrated Children's Guide to Kubernetes
chrisshort
39
47k
YesSQL, Process and Tooling at Scale
rocio
166
14k
Web Components: a chance to create the future
zenorocha
307
41k
A Tale of Four Properties
chriscoyier
155
22k
Rebuilding a faster, lazier Slack
samanthasiow
78
8.5k
Speed Design
sergeychernyshev
9
270
ReactJS: Keep Simple. Everything can be a component!
pedronauck
662
120k
Learning to Love Humans: Emotional Interface Design
aarron
269
39k
GraphQLとの向き合い方2022年版
quramy
36
13k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
17
1.5k
Agile that works and the tools we love
rasmusluckow
325
20k
Transcript
Probabilistic Data Structures in Redis
Srinivasan Rangarajan Head of Engineering
Srinivasan Rangarajan @cnu https://cnu.name
Agenda • Log Analysis • Redis V4 • Probabilistic Data
Structures
Log Analysis
Challenges • 100s of Millions of events processed every day
• Peak of ~10 Million events in an hour • Needs Realtime processing • Memory/Storage Requirements
Cost Accuracy Scale
None
Sample Event Data { "ip": "123.123.123.123", "client_id": 232, "user_id": "35827",
"email": "
[email protected]
", "product_id": "ABC-12345", "image_id": 3, "action": "pageview", "datetime": "2017-06-29T12:42:53Z", }
None
Redis Version 4 • Module system • Better Replication •
Cache eviction Improvements • Non-Blocking DEL and FLUSH* commands • Mixed RDB-AOF persistence format • MEMORY DOCTOR
Modules mikicon NounProject
Loading Modules • ./redis-server --loadmodule /path/to/module.so • redis.conf loadmodule /path/to/module.so
• MODULE LOAD /path/to/module.so
Execute a custom command
Probabilistic Data Structures
There are three kinds of people in the world. 1.
Those who can count. 2. Those who can’t count.
There are three kinds of people in the world. data
structures 1. Those who can count. 2. Those who can’t count. 3. Those who count approximately.
None
Advantage: Huge Memory Savings
3 Data Structures
HyperLogLog Count the Cardinality of a Set http://antirez.com/news/75
Count Unique Visitor / hour
Merge Hourly into Daily
TopK Get Top k Elements in a set https://github.com/RedisLabsModules/topk
Top k IP Addresses
None
CountMinSketch Count the frequency of items https://github.com/RedisLabsModules/countminsketch
User Pageview counter
Bloom Filters Test membership in a set https://github.com/RedisLabsModules/rebloom
Bloom Filters False Positives False Negatives
User Session checking
~3 Data Structures • HyperLogLog • TopK • CountMinSketch •
BloomFilter
Thank you
Follow me @cnu