Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
How IBLT Works
cipepser
March 11, 2018
Technology
0
110
How IBLT Works
cipepser
March 11, 2018
Tweet
Share
More Decks by cipepser
See All by cipepser
Criterion-rs
cipepser
0
90
Practical Anonify
cipepser
2
490
procedural-macros
cipepser
0
90
Move for Libra written in Rust
cipepser
2
2.1k
Other Decks in Technology
See All in Technology
Steps toward self-service operations in eureka
fukubaka0825
0
630
次期LTSに備えよ!AOS 6.1 HCI Core 編
smzksts
0
180
モダンデータスタックとかの話(データエンジニアのお仕事とは)
foursue
0
420
220521_SFN_品質文化試論と『LEADING QUALITY』/220521_SFN_Essay_of_Quality_Culture_and_LEADING_QUALITY
mkwrd
0
250
Scrum Fest Niigata 2022 開発エンジニアに聞いてみよう!
moritamasami
1
210
[SRE NEXT 2022]KaaS桶狭間の戦い 〜Yahoo! JAPANのSLI/SLOを用いた統合監視〜
srenext
0
220
暗号資産ウォレット入門(MetaMaskの入門~NFTの購入~詐欺の注意事項など)
kayato
2
190
How We Foster Reliability in Diversity
nari_ex
PRO
9
2.7k
Stripe Search APIを利用した、LINEとStripeの顧客情報連携/line-dc-202205
stripehideokamoto
0
130
SRE の歩き方・進め方 / sre-walk-through-procedure
rrreeeyyy
0
210
0->1 フェーズで E2E 自動テストを導入した私たちの、これまでとこれから
yoyakoba
0
420
街じゅうを"駅前化"する電動マイクロモビリティのシェアサービス「LUUP」のIoTとSRE
0gm
1
700
Featured
See All Featured
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
237
19k
A Tale of Four Properties
chriscoyier
149
20k
A better future with KSS
kneath
225
15k
Become a Pro
speakerdeck
PRO
3
780
Practical Orchestrator
shlominoach
178
8.6k
From Idea to $5000 a Month in 5 Months
shpigford
372
44k
The Straight Up "How To Draw Better" Workshop
denniskardys
225
120k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
236
1M
Support Driven Design
roundedbygravity
86
8.4k
Three Pipe Problems
jasonvnalue
89
8.6k
5 minutes of I Can Smell Your CMS
philhawksworth
196
18k
Adopting Sorbet at Scale
ufuk
63
7.5k
Transcript
How IBLT works @cipepser
Structure of IBLT 1 2 3 ... ... ... m
m cells each cell have 3 fields: count keySum valueSum
Supported method ・Insert(key, value) ・Get(key) ・Delete(key, value) ・ListEntries()
Insert for each hash function h i [key], i =
1, …, k (like standard BloomFilter) T: IBLT T[h i [key]].count++ T[h i [key]].sumKey += key T[h i [key]].sumValue += value
Insert ex) m = 7 cells, k = 3 hash
functions, h i (x) = (10 * i + x) mod m (simply) Insert(key=5, value=10) h 1 (key=5) = 1 0 c: 1 ks: 5 vs: 10 1 c: 1 ks: 5 vs: 10 2 3 4 c: 1 ks: 5 vs: 10 5 6 count: c keySum: ks valueSum: vs h 2 (key=5) = 4 h 3 (key=5) = 0
Insert ex) m = 7 cells, k = 3 hash
functions, h i (x) = (10 * i + x) mod m (simply) Insert(key=2, value=30) h 1 (key=2) = 5 0 c: 1 ks: 5 vs: 10 1 c: 2 ks: 7 vs: 40 2 3 4 c: 2 ks: 7 vs: 40 5 c: 1 ks: 2 vs: 30 6 count: c keySum: ks valueSum: vs h 2 (key=2) = 1 h 3 (key=2) = 4
Get T[h i [key]].count == 0? YES NO return false
T[h i [key]].count == 1? return sumValue for i = 1, …, k end for T[h i [key]].sumKey == key? YES YES return false NO NO
Get ex) m = 7 cells, k = 3 hash
functions, h i (x) = (10 * i + x) mod m (simply) (key, value) = (5, 10), (2, 30) have been inserted Get(key=2) →return 30 0 c:1 ks: 5 vs: 10 1 c:2 ks: 7 vs: 40 2 3 4 c:2 ks: 7 vs: 40 5 c:1 ks: 2 vs: 30 6 h 1 (key=2) = 5 T[5].count = 1 T[5].keySum = 2 T[5].valueSum = 30
Get ex) m = 7 cells, k = 3 hash
functions, h i (x) = (10 * i + x) mod m (simply) (key, value) = (5, 10), (2, 30) have been inserted Get(key=3) →return false 0 c:1 ks: 5 vs: 10 1 c:2 ks: 7 vs: 40 2 3 4 c:2 ks: 7 vs: 40 5 c:1 ks: 2 vs: 30 6 c:0 ks: 0 vs: 0 h 1 (key=3) = 6 T[6].count = 0 != 1
Get ex) m = 7 cells, k = 3 hash
functions, h i (x) = (10 * i + x) mod m (simply) (key, value) = (5, 10), (2, 30) have been inserted Get(key=9) →return false 0 c:1 ks: 5 vs: 10 1 c:2 ks: 7 vs: 40 2 3 4 c:2 ks: 7 vs: 40 5 c:1 ks: 2 vs: 30 6 h 1 (key=9) = 5 T[5].keySum = 2 h 2 (key=9) = 1 T[1].keySum = 7 h 3 (key=9) = 4 T[4].keySum = 7
Get ex) m = 7 cells, k = 3 hash
functions, h i (x) = (10 * i + x) mod m (simply) (key, value) = (5, 10), (2, 30), (3, 20) have been inserted Get(key=2) →return false (false-negative) 0 c:1 ks: 5 vs: 10 1 c:2 ks: 7 vs: 40 2 c:1 ks: 3 vs: 20 3 4 c:2 ks: 7 vs: 40 5 c:2 ks: 5 vs: 50 6 c:1 ks: 3 vs: 20 h 1 (key=2) = 5 T[5].keySum = 5 != 2 h 2 (key=2) = 1 T[1].keySum = 7 != 2 h 3 (key=2) = 4 T[4].keySum = 7 != 2
Delete for each hash function h i [key], i =
1, …, k T: IBLT T[h i [key]].count-- T[h i [key]].sumKey -= key T[h i [key]].sumValue -= value
Delete ex) m = 7 cells, k = 3 hash
functions, h i (x) = (10 * i + x) mod m (simply) (key, value) = (5, 10), (2, 30) have been inserted 0 c:1 ks: 5 vs: 10 1 c:2 ks: 7 vs: 40 2 3 4 c:2 ks: 7 vs: 40 5 c:1 ks: 2 vs: 30 6 Delete(key=2, value=30) h 1 (key=2) = 5 0 c: 1 ks: 5 vs: 10 1 c: 1 ks: 5 vs: 10 2 3 4 c: 1 ks: 5 vs: 10 5 c: 0 ks: 0 vs: 0 6 h 2 (key=2) = 1 h 3 (key=2) = 4
ListEntries T[i].count == 1? YES NO for i = 1,
…, m end for push( key = T[i].sumKey, value = T[i].sumValue ) Delete(key, value) START
ListEntries ex) m = 7 cells, k = 3 hash
functions, h i (x) = (10 * i + x) mod m (simply) (key, value) = (5, 10), (2, 30) have been inserted 0 c:1 ks: 5 vs: 10 1 c:2 ks: 7 vs: 40 2 3 4 c:2 ks: 7 vs: 40 5 c:1 ks: 2 vs: 30 6 T[0].count == 1 push(key=5,value=10) Delete(key=5,value=10) 0 c:0 ks: 0 vs: 0 1 c:1 ks: 2 vs: 30 2 3 4 c:1 ks: 2 vs: 30 5 c:1 ks: 2 vs: 30 6 T[1].count == 1 push(key=2,value=30) Delete(key=2,value=30) 0 c:0 ks: 0 vs: 0 1 c:0 ks: 0 vs: 0 2 3 4 c:0 ks: 0 vs: 0 5 c:0 ks: 0 vs: 0 6 pushed key-value pairs: {(5, 10), (2, 30)}