Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Riak 2.0における全文検索 と整合性の問題
Search
UENISHI Kota
November 19, 2014
Technology
2
780
Riak 2.0における全文検索 と整合性の問題
WebDB Forum 2014 のBashoジャパンの発表スライドです
UENISHI Kota
November 19, 2014
Tweet
Share
More Decks by UENISHI Kota
See All by UENISHI Kota
Metadata Management in Distributed File Systems
kuenishi
2
480
Behind The Scenes: Cloud Native Storage System for AI
kuenishi
2
360
Apache Ozone behind Simulation and AI Industries
kuenishi
0
330
Distributed Deep Learning with Chainer and Hadoop
kuenishi
3
1.2k
A Few Ways to Accelerate Deep Learning
kuenishi
0
1k
Introducing Retz
kuenishi
5
1.1k
Introducing Retz and how to develop practical frameworks
kuenishi
3
700
Formalization and Proof of Distributed Systems (ja)
kuenishi
10
6.3k
Mesos Frameworkの作り方 (How to Make Mesos Framework)
kuenishi
7
2.3k
Other Decks in Technology
See All in Technology
The Future of SEO: The Impact of AI on Search
badams
0
250
Swiftの “private” を テストする / Testing Swift "private"
yutailang0119
0
140
OpenID BizDay#17 KYC WG活動報告(法人) / 20250219-BizDay17-KYC-legalidentity
oidfj
0
420
明日からできる!技術的負債の返済を加速するための実践ガイド~『ホットペッパービューティー』の事例をもとに~
recruitengineers
PRO
3
550
2024.02.19 W&B AIエージェントLT会 / AIエージェントが業務を代行するための計画と実行 / Algomatic 宮脇
smiyawaki0820
15
4.2k
SA Night #2 FinatextのSA思想/SA Night #2 Finatext session
satoshiimai
1
150
OpenID BizDay#17 みんなの銀行による身元確認結果の活用 / 20250219-BizDay17-KYC-minna-no-ginko
oidfj
0
180
生成 AI プロダクトを育てる技術 〜データ品質向上による継続的な価値創出の実践〜
icoxfog417
PRO
5
1.8k
Reading Code Is Harder Than Writing It
trishagee
2
110
OpenID Connect for Identity Assurance の概要と翻訳版のご紹介 / 20250219-BizDay17-OIDC4IDA-Intro
oidfj
0
440
Developers Summit 2025 浅野卓也(13-B-7 LegalOn Technologies)
legalontechnologies
PRO
1
1.4k
抽象化をするということ - 具体と抽象の往復を身につける / Abstraction and concretization
soudai
27
14k
Featured
See All Featured
Build your cross-platform service in a week with App Engine
jlugia
229
18k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
7
640
Adopting Sorbet at Scale
ufuk
74
9.2k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
129
19k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
27
1.6k
Raft: Consensus for Rubyists
vanstee
137
6.8k
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
Unsuck your backbone
ammeep
669
57k
Code Reviewing Like a Champion
maltzj
521
39k
The Cult of Friendly URLs
andyhume
78
6.2k
Designing Experiences People Love
moore
140
23k
Mobile First: as difficult as doing things right
swwweet
223
9.4k
Transcript
Riak 2.0ʹ͓͚Δશจݕࡧ ͱ߹ੑͷ WebDB Forum 2014 Basho Japan ্ 2014/11/19
ࣗݾհ • ࢄγεςϜྺ6 • Bashoδϟύϯͷํ͔Βདྷ·ͨ͠ • Riak CSͷ։ൃ • ͦͷଞຊͷ͜ͱ
BashoͱRiak •ࢄσʔλϕʔεʁ •RiakΛ͍ͬͯΔʁ •BashoΛ͍ͬͯΔʁ
We are hiring •࣮ੈքͷࢄγεςϜͷ ʹڵຯ͋Δਓʂ •@BashoJapan •
[email protected]
•APࢦͷσʔλϕʔεʢ݁Ռ߹ੑʣ •҆ఆੑɺ༧ଌՄೳੑ •ʮσʔλΛઈରʹͳ͘͞ͳ͍ʯ •৽͍͠ݕࡧΤϯδϯͱͯ͠SolrΛόϯυϧ
͜Μͳͱ͜ΖͰ ಈ͍͍ͯ·͢Riak •Rovio (Angry Birds) •Yahoo!JAPAN ͷΫϥυετϨʔδ •NHS (ΠΪϦε ࠃຽอݥαʔϏε)
•League of Legends (MMORPG) •ۜߦɺήʔϜɺখചɺηϯαʔɺetc…
CAPఆཧͱཧͷDB •ͲΜͳނোʹରͯ͠ (partition tolerance) •σʔλৗʹ߹͓ͯ͠Γ (consistency) •γεςϜ͕ࢭ·Δ͜ͱͳ͍ (availability) ͜ͷ3ͭΛಉ࣌ʹຬͨ͢γεςϜଘࡏ͠ͳ͍
CAP Theorem • C: ෳͷAtomic Objectʹର͢Δ࿈ ଓͨ͠ૢ࡞ (w1, w3, w4,
….) ͕શ ͯಉҰͰ͋Δ͜ͱ (linearizable) • A: Atomic Objectʹૢ࡞ w1, w2, …Λ࣮ߦͰ͖Δ͜ͱ • P: ωοτϫʔΫ͕ΕΔͱAtomic Objectͷϝοηʔδ͕౸ୡ͠ͳ͍
Is the network reliable?
None
None
How Riak Works (mostly about search)
run Riak with Search $ which java $ sudo apt-get
install riak $ echo “search = on” >> /etc/riak/riak.conf $ riak start $ riak-admin bucket-type create tt $ riak-admin bucket-type activate tt $ curl -XPUT http://localhost:8098/search/index/t $ curl -XPUT -H ‘content-type: application/json’ \ http://localhost:8098/types/tt/props \ -d ‘{“props”:{“search_index”:”t”}}’
Consistent Hashing • 160-bit Ωʔۭؒ • ۭؒΛ͢Δ • ύʔςΟγϣϯϊʔ υ͕ݸผཧ
• ϨϓϦΧNݸͷύʔ ςΟγϣϯʹίϐʔ͞ ΕΔ OPEF OPEF OPEF OPEF hash(“meetups/spamham”) N=3
σʔλϕʔεͷݕࡧ • DBࠐܕ • groongaͳͲ • “LIKE” • Pros •
߹ੑΛҡ͍࣋͢͠ • Cons • ີ݁߹ʹͳΔ • ֎෦࿈ܞܕ • Solr • Elasticsearch • Pros • ૄ݁߹ʹͰ͖Δ • Cons • ߹ੑΛҡ࣋͠ʹ͍͘
DBͱΠϯσοΫεͷ߹ੑ • DBͷσʔλຊମͱɺΠϯσοΫεΛ࣌ʹ҆શʹߋ ৽͢ΔͨΊʹτϥϯβΫγϣϯ͕ඞཁ • ີ݁߹ͷγεςϜͰ͋ΕϩοΫͳͲΛͬͯ߹ੑ Λอো͍͢͠
ૄ݁߹ʹ͢ΔϝϦοτ • ಉ͡ݕࡧΤϯδϯͰ͍ΖΜͳDBʹରԠͰ͖Δ • ӡ༻ϊϋπʔϧɺ։ൃΛूதͰ͖Δ • όʔδϣϯ߹ͷ੍͕؇͞ΕΔ • ยํ͕ނো͍ͯͯ͠͏ยํಈ࡞Ͱ͖Δ •
֦ு͘͢͠ͳΔ
྆ํͱ ͍͍ͱ͜ͲΓ͍ͨ͠
Yokozuna
as Riak Search 2.0 = + σʔλͷӬଓԽ Ϋϥελͷཧ γϯϓϧͳૢ࡞ RVFSZ
JOEFYJOH
Riak Search 2.0 • RiakʹSolrΛόϯυϧ • RiakͷKVΛߋ৽͢Δͱσʔλ͕ࣗಈతʹΠϯσο ΫεԽ͞ΕΔ • SolrͷϓϥάΠϯεΩʔϚɺΫΤϦΛͦͷ··
͏͜ͱ͕Ͱ͖Δ • ԽɺϊʔυՃআRiakͷ୲
ૄ݁߹ʁີ݁߹ʁ • ύοέʔδಉࠝ • ϓϩηεͱͯ͠ૄ݁߹ • ΠϯσοΫεͱσʔλؒ݁Ռ߹ੑΛอো • ΠϯσοΫεͷԽ݁Ռ߹ੑΛอো
Solrͱૄ݁߹ • Riak ϓϩηε͕ Solr ͷϓϩηεΛ fork ͢Δ • ཧɺࢹશͯRiak͕ߦ͏
• ผϓϩηεɺϝϞϦۭؒผ
ΠϯσοΫεԽͷྲྀΕ • Riakͷߋ৽ͱҰ৺ಉମʢdocument-based indexingʣ • ΠϯσοΫεԽʹࣦഊ͢ΔͱPUT͕ࣦഊ
ݕࡧΫΤϦͷॲཧ • Riakͷ2iMapReduceͱಉछͷΫΤϦ͕Δ • 3͓͖ͭͷ vnode ʹΞΫηε͢ΔͷͰɺॏෳͳ͍ ू
߹ੑͷ ͱҟৗܥ ߹ੑҡ࣋ͷͨΊʹߋ৽ϩάΛอ࣋ͯ͠ ϦΧόϦ͢Δͱ͍͏ํ๏ͰɺՄ༻ੑͱ அੑΛ୲อͰ͖ͳ͍ɻRiakͰଞͷ ํ๏Ͱσʔλͷ߹ੑΛҡ࣋͢Δɻ
Α͋͘Δҟৗܥ #1 •ϊʔυނোͰෳΛ3ͭ࡞Εͳ͍ •෮چͯ͠ɺ2ͭͷ··ʹͳͬͯ͠·͏
Α͋͘Δҟৗܥ #1 • ෮چͨ͋͠ͱʹRead Repair͢Δ • GETͨ͠ͱ͖ʹɺσʔλʹ͕ܽؕ͋ͬͨΒ RiakଆͰPUT͢͠ v2 v2
not found v2
Read Repair͕ ݺΕͳ͍ͱ σʔλ͕Βͳ͍ʁ YES
ΫϥΠΞϯτ͔Β GET͕ݺΕͳ͍ͱ σʔλ͕Βͳ͍ʁ NO
AAE (Active Anti Entropy) https://www.flickr.com/photos/51pct/7507525118/
Active Anti Entropy • APࢦͷDBͷσʔλྼԽΛ͙ ͨΊͷόοΫάϥϯυॲཧ • Merkle-TreeΛͬͯύʔςΟγϣ ϯຖͷʮνΣοΫαϜʯΛܭࢉ •
ࠩΛݟ͚ͭͨΒͦ͜ΛRead Repair͢Δ hash(vnode=0, pid=0) hash(vnode=1, pid=0) hash(vnode=2, pid=0)
Α͋͘Δҟৗܥ #2 • Solr͕ΠϯσοΫεԽ͢Δલʹམͪͨ • Key, Valueอଘ͞Ε͕ͨΠϯσοΫεͳ͍
Read Repair͕ ݺΕΕΠϯσο Ϋε͕म෮͞ΕΔʁ NO
AAE (Active Anti Entropy) https://www.flickr.com/photos/51pct/7507525118/
YZ Active Anti Entropy • ݕࡧΠϯσοΫεʹ͍ͭͯ Merkle TreeΛ࡞ • ύʔςΟγϣϯຖʹKey-Valueͷ
Treeͱൺֱ͠ɺ͕ࠩ͋ΕΠϯ σοΫεΛमਖ਼ hash(vnode=0, pid=0, kv) hash(vnode=0, pid=0, yz)
Α͋͘Δҟৗܥ #2 •Searchઐ༻AAE͕ಈ࡞͍ͯ͠Δ •ΠϯσοΫεͱσʔλͷෆ߹Λݟ͚ͭͯ म෮͢Δ AAE
Α͋͘Δҟৗܥ #3 • ωοτϫʔΫ͕Ε͍ͯͨͱ͖ʹ྆ଆͰߋ৽͕ى ͖ͨʢSplit Brainʣ • Hinted HandoffʹΑΓͲͪΒॻ͖ࠐΈޭ
Α͋͘Δҟৗܥ #3 • ෮چ࣌ʹHandoffͰฦ͞Εɺ྆ํͷΛอ࣋ • GET·ͨAAEʹΑͬͯRead Repair͞ΕΔ
ҟৗܥ #1, #2, #3 •Ͳͷҟৗɺಛʹۓٸͷରॲඞཁͳ͍ •߹ੑݕࠪͱम෮͕ࣗಈͰߦΘΕΔ •ϧʔνϯϫʔΫͷӡ༻࡞ۀ͕΄ͱΜͲͳ͍
·ͱΊ • σʔλϕʔεͱશจݕࡧͷ • ߹ੑΛͱΔ͔ɺૄ݁߹ΛͱΔ͔ͷ • ৽͍͠Riak Search 2.0 •
ਫฏࢄͯ͠εέʔϧΞτͰ͖Δݕࡧ • ݁Ռ߹ੑΛબͿ͜ͱʹΑΓૄ݁߹ԽͱࢄԽ ʹޭʢ͔͠APܕʹʂʣ • ݁Ռ߹ੑΛબͨ͠ઃܭʹΑΓӡ༻͕؆୯ʹ
Questions?
FAQ •ΦϯϥΠϯͷεΩʔϚมߋʁ •μΠφϛοΫεΩʔϚʁ
outline • σʔλϕʔεʹೖ͍ͬͯΔσʔλΛશจݕࡧ͍ͨ͠→ࢁͷΠϯσοΫε͕Ͱ͖ΔɺFKeyͰ ΠϯσοΫε…ͱ͍͏͚ͩͰ • શจݕࡧΤϯδϯͷछྨ • ֎෦ܕʢ߹ੑΛҡ࣋͠ʹ͍͕͘Ԡ༻͕ޮ͘ɺރΕͨιϑτΣΞϓϥάΠϯ͕ଟ͍ʣ • ΈࠐΈܕʢ߹ੑΛҡ͍͕࣋͢͠ɺϝϯςφϯε͕͍͠ʣ
• Riakͦͷதؒɹόϯυϧͯ͠ΔͷͰָɺ͚ͩͲ࣮ମSolrɺ߹ੑΛࣗಈతʹҡ࣋͢Δ • όϯυϧͷํ๏ɺ߹ੑɺࢄͷํ๏ • ݕࡧΠϯσοΫε͚ͩΛࢄͤ͞Δͷ͍͠ • DBͱΠϯσοΫεͷ߹ɺϨϓϦΧͱϨϓϦΧͷ߹ɺJepsenͷͳ͠