Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Distinct Query using HyperLogLog
Search
hama_du
October 05, 2018
Science
97
2
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Distinct Query using HyperLogLog
Distinct Queryを例にHyperLogLogのお気持ちを理解する
hama_du
October 05, 2018
Transcript
29 slides
HyperLogLogͰ লϝϞϦͳDistinct Query SDDษڧձ@r-n-i 2018/10/05
Distinct Query
ΫΤϦͷྫ • Distinct([‘A’, ‘B’]) = 2 • Distinct([‘A’, ‘B’, ‘C’,
‘A’, ‘C’]) = 3
[࣮ํ๏] SetʹಥͬࠐΜͰେ͖͞ΛऔΔ ;ͭ͏ͷ
SetʹಥͬࠐΉࡍͷ • σʔλྻͷαΠζ͕େ͖͍ͱਏ͍ • ͞ͷ͚ͩϝϞϦ৯͏
ΫΤϦͷྫ - ۩ମྫ • ϢʔβIDͷྻʹରͯ͠ɺϢχʔΫϢʔβ
ϢχʔΫϢʔβ… 35915ਓͰͨ͠ʂʂ
ϢχʔΫϢʔβ… 35915ਓͰͨ͠ʂʂ
ϢχʔΫϢʔβ… 35915ਓͰͨ͠ʂʂ ͜Ε͍Δʁ
ਖ਼֬ͳ ͦΜͳʹେࣄ͡Όͳ͍ ͜ͱ͋Δ
HashΛ༻͍ͨਪఆ
ϋογϡͷܭࢉ hash(AB) = 0x36f… = 0011 0110 1111 … hash(CD)
= 0xc90… = 1100 1001 0000 … hash(EF) = 0x01e… = 0000 0001 1110 …
ઌ಄ʹ͍ͭ͘ 0 ͕͍ͭͯΔʁ zero(hash(AB)) = zero(0011 0110 1111…) = 2
zero(hash(CD)) = zero(1100 1001 0000…) = 0 zero(hash(EF)) = zero(0000 0001 1110…) = 7
͜ΕΒͷ࠷େΛऔΔ D = max( zero(hash(AB)), zero(hash(CD)), zero(hash(EF)) ) = max(2,
0, 7) = 7
ٯʹ…
࠷େ͚ͩΘ͔ͬͯΔͱ͢Δ D = 7
ͭ·Γ… D = max(?, ?, …, 7, …, ?, ?)
zero(hash(?)) = zero(0000 0001 …) = 7
ͭ·Γ… D = max(?, ?, …, 7, …, ?, ?)
zero(hash(?)) = zero(0000 0001 …) = 7 ݁ߏϨΞʂ
ͲͷఔϨΞʁ D = max(?, ?, …, 7, …, ?, ?)
zero(hash(?)) = zero(0000 0001 …) = 7 1/2^7 = 1/128
ϢχʔΫͳHashΛ͍ͭ͘ݟͨʁ D = max(?, ?, …, 7, …, ?, ?)
1/2^7 = 1/128 ฏۉ128ݸʁ
Distinct ͳཁૉ(Hash)Λ େࡶʹ༧Ͱ͖Δ
HyperLogLog
HashͷඌͰৼΓ͚ hash(AB) = 0x36f… = 0011 0110 … 1010 D:
0 1 9 10 11 14 15 1 1 0 2 0 0 0 … …
େ͖͍Ͱߋ৽ʂ hash(AB) = 0x36f… = 0011 0110 … 1010 D:
0 1 9 10 11 14 15 2 1 0 2 0 0 0 … …
ཁૉͷਪఆ • Dͷঢ়گ͕ฏۉͲͷఔϨΞ͔ʁ • ௐฏۉʂ
ཁૉͷਪఆ 1 1 2 4 C × 4 × 4
1 22 + 1 21 + 1 21 + 1 24 ശ1ͭ͋ͨΓͷೱ
ن͕খ͍͞ͱޡࠩଟΊ
ۭؒܭࢉྔ(༻ϝϞϦ) • ༻ϝϞϦ: ܕͷྻ͚ͩʂ
ࢀߟจݙ • HyperLogLog in Practice: Algorithmic Engineering of a State
of The Art Cardinality Estimation Algorithm
More Decks by hama_du
See All by hama_du
Google File System
hamadu
0
100
木の上を歩こう
hamadu
1
1.1k
linear-algebra-in-n-minutes
hamadu
0
260
Other Decks in Science
See All in Science
知能とはなにか -ヒトとAIのあいだ-
tagtag
PRO
0
280
[Relay webinaire eBIS actu n°16] Sélection pour la vitalité des jeunes bovins allaitants et laitiers
institutdelelevage
PRO
0
370
サンプル対応のない複数遺伝子発現プロファイルに対するテンソル分解型統合解析の要約
tagtag
PRO
0
290
Wet Active Matter
rajeshrinet
0
280
データベース05: SQL(2/3) 結合質問
trycycle
PRO
0
1.4k
データベース14: B+木 & ハッシュ索引
trycycle
PRO
0
1k
Build your own LLM, Live, with MicroGPT
ianozsvald
0
180
機械学習 - K近傍法 & 機械学習のお作法
trycycle
PRO
1
1.7k
医療 LLM ベンチマークの現在地:多面的評価 と日本ローカライズ
analokmaus
1
800
JPA2026_NetworkSymposium_Omizu
junkashihara
0
120
Note agro-climatique et prairies - Numéro 7
institutdelelevage
PRO
0
750
Leitner Inauguration Lecture Chalmers University of Technology
xleitix
0
410
Featured
See All Featured
Build The Right Thing And Hit Your Dates
maggiecrowley
39
3.5k
Optimising Largest Contentful Paint
csswizardry
37
4k
Everyday Curiosity
cassininazir
0
400
Leading Effective Engineering Teams in the AI Era
addyosmani
9
2.9k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
32
3.1k
Keith and Marios Guide to Fast Websites
keithpitt
413
23k
Typedesign – Prime Four
hannesfritz
42
3.2k
Ten Tips & Tricks for a 🌱 transition
stuffmc
1
290
Collaborative Software Design: How to facilitate domain modelling decisions
baasie
1
380
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
52
6.2k
Technical Leadership for Architectural Decision Making
baasie
4
650
Git: the NoSQL Database
bkeepers
PRO
433
67k