Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Riak 2.0における全文検索 と整合性の問題
Search
UENISHI Kota
November 19, 2014
Technology
2
740
Riak 2.0における全文検索 と整合性の問題
WebDB Forum 2014 のBashoジャパンの発表スライドです
UENISHI Kota
November 19, 2014
Tweet
Share
More Decks by UENISHI Kota
See All by UENISHI Kota
Metadata Management in Distributed File Systems
kuenishi
2
460
Behind The Scenes: Cloud Native Storage System for AI
kuenishi
2
320
Apache Ozone behind Simulation and AI Industries
kuenishi
0
280
Distributed Deep Learning with Chainer and Hadoop
kuenishi
3
1.1k
A Few Ways to Accelerate Deep Learning
kuenishi
0
980
Introducing Retz
kuenishi
5
1k
Introducing Retz and how to develop practical frameworks
kuenishi
3
660
Formalization and Proof of Distributed Systems (ja)
kuenishi
10
6.2k
Mesos Frameworkの作り方 (How to Make Mesos Framework)
kuenishi
7
2.3k
Other Decks in Technology
See All in Technology
令和最新版 Perlコーディングガイド
anatofuz
4
3.8k
Grafana エコシステムの活用事例 on ABEMA
tetsuya28
4
280
KubeVirt Networking ONIC 2024
orimanabu
4
630
リスクから学ぶKubernetesコンテナセキュリティ/k8s-risk-and-security
mochizuki875
1
330
スタサプ ForSCHOOLアプリのシンプルな設計
recruitengineers
PRO
3
630
「ばん・さく・つき・たー!」にならないためにSHIROBAKOから 学んだこと
ysknsid25
3
690
ADRを運用して3年経った僕らの現在地
onk
PRO
13
5.8k
Oracle Database 23ai 新機能#4 Application Continuity
oracle4engineer
PRO
0
120
テクニカルライターのチームで「目標」をどう決めたか / MVV for a Team of Technical Writers
lycorptech_jp
PRO
3
100
Oracle Database 23ai 新機能#4 Real Application Clusters
oracle4engineer
PRO
0
160
寒冷地稲作の歴史にみるコミュニティ
miu_crescent
2
110
Develop to Survive - YAPC::Hakodate 2024 Keynote
moznion
8
2.8k
Featured
See All Featured
Gamification - CAS2011
davidbonilla
80
5k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
31
2.6k
Statistics for Hackers
jakevdp
796
220k
Pencils Down: Stop Designing & Start Developing
hursman
119
11k
Imperfection Machines: The Place of Print at Facebook
scottboms
264
13k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
131
33k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
167
49k
Music & Morning Musume
bryan
46
6.1k
RailsConf 2023
tenderlove
28
840
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
1
300
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
159
15k
Fantastic passwords and where to find them - at NoRuKo
philnash
50
2.8k
Transcript
Riak 2.0ʹ͓͚Δશจݕࡧ ͱ߹ੑͷ WebDB Forum 2014 Basho Japan ্ 2014/11/19
ࣗݾհ • ࢄγεςϜྺ6 • Bashoδϟύϯͷํ͔Βདྷ·ͨ͠ • Riak CSͷ։ൃ • ͦͷଞຊͷ͜ͱ
BashoͱRiak •ࢄσʔλϕʔεʁ •RiakΛ͍ͬͯΔʁ •BashoΛ͍ͬͯΔʁ
We are hiring •࣮ੈքͷࢄγεςϜͷ ʹڵຯ͋Δਓʂ •@BashoJapan •
[email protected]
•APࢦͷσʔλϕʔεʢ݁Ռ߹ੑʣ •҆ఆੑɺ༧ଌՄೳੑ •ʮσʔλΛઈରʹͳ͘͞ͳ͍ʯ •৽͍͠ݕࡧΤϯδϯͱͯ͠SolrΛόϯυϧ
͜Μͳͱ͜ΖͰ ಈ͍͍ͯ·͢Riak •Rovio (Angry Birds) •Yahoo!JAPAN ͷΫϥυετϨʔδ •NHS (ΠΪϦε ࠃຽอݥαʔϏε)
•League of Legends (MMORPG) •ۜߦɺήʔϜɺখചɺηϯαʔɺetc…
CAPఆཧͱཧͷDB •ͲΜͳނোʹରͯ͠ (partition tolerance) •σʔλৗʹ߹͓ͯ͠Γ (consistency) •γεςϜ͕ࢭ·Δ͜ͱͳ͍ (availability) ͜ͷ3ͭΛಉ࣌ʹຬͨ͢γεςϜଘࡏ͠ͳ͍
CAP Theorem • C: ෳͷAtomic Objectʹର͢Δ࿈ ଓͨ͠ૢ࡞ (w1, w3, w4,
….) ͕શ ͯಉҰͰ͋Δ͜ͱ (linearizable) • A: Atomic Objectʹૢ࡞ w1, w2, …Λ࣮ߦͰ͖Δ͜ͱ • P: ωοτϫʔΫ͕ΕΔͱAtomic Objectͷϝοηʔδ͕౸ୡ͠ͳ͍
Is the network reliable?
None
None
How Riak Works (mostly about search)
run Riak with Search $ which java $ sudo apt-get
install riak $ echo “search = on” >> /etc/riak/riak.conf $ riak start $ riak-admin bucket-type create tt $ riak-admin bucket-type activate tt $ curl -XPUT http://localhost:8098/search/index/t $ curl -XPUT -H ‘content-type: application/json’ \ http://localhost:8098/types/tt/props \ -d ‘{“props”:{“search_index”:”t”}}’
Consistent Hashing • 160-bit Ωʔۭؒ • ۭؒΛ͢Δ • ύʔςΟγϣϯϊʔ υ͕ݸผཧ
• ϨϓϦΧNݸͷύʔ ςΟγϣϯʹίϐʔ͞ ΕΔ OPEF OPEF OPEF OPEF hash(“meetups/spamham”) N=3
σʔλϕʔεͷݕࡧ • DBࠐܕ • groongaͳͲ • “LIKE” • Pros •
߹ੑΛҡ͍࣋͢͠ • Cons • ີ݁߹ʹͳΔ • ֎෦࿈ܞܕ • Solr • Elasticsearch • Pros • ૄ݁߹ʹͰ͖Δ • Cons • ߹ੑΛҡ࣋͠ʹ͍͘
DBͱΠϯσοΫεͷ߹ੑ • DBͷσʔλຊମͱɺΠϯσοΫεΛ࣌ʹ҆શʹߋ ৽͢ΔͨΊʹτϥϯβΫγϣϯ͕ඞཁ • ີ݁߹ͷγεςϜͰ͋ΕϩοΫͳͲΛͬͯ߹ੑ Λอো͍͢͠
ૄ݁߹ʹ͢ΔϝϦοτ • ಉ͡ݕࡧΤϯδϯͰ͍ΖΜͳDBʹରԠͰ͖Δ • ӡ༻ϊϋπʔϧɺ։ൃΛूதͰ͖Δ • όʔδϣϯ߹ͷ੍͕؇͞ΕΔ • ยํ͕ނো͍ͯͯ͠͏ยํಈ࡞Ͱ͖Δ •
֦ு͘͢͠ͳΔ
྆ํͱ ͍͍ͱ͜ͲΓ͍ͨ͠
Yokozuna
as Riak Search 2.0 = + σʔλͷӬଓԽ Ϋϥελͷཧ γϯϓϧͳૢ࡞ RVFSZ
JOEFYJOH
Riak Search 2.0 • RiakʹSolrΛόϯυϧ • RiakͷKVΛߋ৽͢Δͱσʔλ͕ࣗಈతʹΠϯσο ΫεԽ͞ΕΔ • SolrͷϓϥάΠϯεΩʔϚɺΫΤϦΛͦͷ··
͏͜ͱ͕Ͱ͖Δ • ԽɺϊʔυՃআRiakͷ୲
ૄ݁߹ʁີ݁߹ʁ • ύοέʔδಉࠝ • ϓϩηεͱͯ͠ૄ݁߹ • ΠϯσοΫεͱσʔλؒ݁Ռ߹ੑΛอো • ΠϯσοΫεͷԽ݁Ռ߹ੑΛอো
Solrͱૄ݁߹ • Riak ϓϩηε͕ Solr ͷϓϩηεΛ fork ͢Δ • ཧɺࢹશͯRiak͕ߦ͏
• ผϓϩηεɺϝϞϦۭؒผ
ΠϯσοΫεԽͷྲྀΕ • Riakͷߋ৽ͱҰ৺ಉମʢdocument-based indexingʣ • ΠϯσοΫεԽʹࣦഊ͢ΔͱPUT͕ࣦഊ
ݕࡧΫΤϦͷॲཧ • Riakͷ2iMapReduceͱಉछͷΫΤϦ͕Δ • 3͓͖ͭͷ vnode ʹΞΫηε͢ΔͷͰɺॏෳͳ͍ ू
߹ੑͷ ͱҟৗܥ ߹ੑҡ࣋ͷͨΊʹߋ৽ϩάΛอ࣋ͯ͠ ϦΧόϦ͢Δͱ͍͏ํ๏ͰɺՄ༻ੑͱ அੑΛ୲อͰ͖ͳ͍ɻRiakͰଞͷ ํ๏Ͱσʔλͷ߹ੑΛҡ࣋͢Δɻ
Α͋͘Δҟৗܥ #1 •ϊʔυނোͰෳΛ3ͭ࡞Εͳ͍ •෮چͯ͠ɺ2ͭͷ··ʹͳͬͯ͠·͏
Α͋͘Δҟৗܥ #1 • ෮چͨ͋͠ͱʹRead Repair͢Δ • GETͨ͠ͱ͖ʹɺσʔλʹ͕ܽؕ͋ͬͨΒ RiakଆͰPUT͢͠ v2 v2
not found v2
Read Repair͕ ݺΕͳ͍ͱ σʔλ͕Βͳ͍ʁ YES
ΫϥΠΞϯτ͔Β GET͕ݺΕͳ͍ͱ σʔλ͕Βͳ͍ʁ NO
AAE (Active Anti Entropy) https://www.flickr.com/photos/51pct/7507525118/
Active Anti Entropy • APࢦͷDBͷσʔλྼԽΛ͙ ͨΊͷόοΫάϥϯυॲཧ • Merkle-TreeΛͬͯύʔςΟγϣ ϯຖͷʮνΣοΫαϜʯΛܭࢉ •
ࠩΛݟ͚ͭͨΒͦ͜ΛRead Repair͢Δ hash(vnode=0, pid=0) hash(vnode=1, pid=0) hash(vnode=2, pid=0)
Α͋͘Δҟৗܥ #2 • Solr͕ΠϯσοΫεԽ͢Δલʹམͪͨ • Key, Valueอଘ͞Ε͕ͨΠϯσοΫεͳ͍
Read Repair͕ ݺΕΕΠϯσο Ϋε͕म෮͞ΕΔʁ NO
AAE (Active Anti Entropy) https://www.flickr.com/photos/51pct/7507525118/
YZ Active Anti Entropy • ݕࡧΠϯσοΫεʹ͍ͭͯ Merkle TreeΛ࡞ • ύʔςΟγϣϯຖʹKey-Valueͷ
Treeͱൺֱ͠ɺ͕ࠩ͋ΕΠϯ σοΫεΛमਖ਼ hash(vnode=0, pid=0, kv) hash(vnode=0, pid=0, yz)
Α͋͘Δҟৗܥ #2 •Searchઐ༻AAE͕ಈ࡞͍ͯ͠Δ •ΠϯσοΫεͱσʔλͷෆ߹Λݟ͚ͭͯ म෮͢Δ AAE
Α͋͘Δҟৗܥ #3 • ωοτϫʔΫ͕Ε͍ͯͨͱ͖ʹ྆ଆͰߋ৽͕ى ͖ͨʢSplit Brainʣ • Hinted HandoffʹΑΓͲͪΒॻ͖ࠐΈޭ
Α͋͘Δҟৗܥ #3 • ෮چ࣌ʹHandoffͰฦ͞Εɺ྆ํͷΛอ࣋ • GET·ͨAAEʹΑͬͯRead Repair͞ΕΔ
ҟৗܥ #1, #2, #3 •Ͳͷҟৗɺಛʹۓٸͷରॲඞཁͳ͍ •߹ੑݕࠪͱम෮͕ࣗಈͰߦΘΕΔ •ϧʔνϯϫʔΫͷӡ༻࡞ۀ͕΄ͱΜͲͳ͍
·ͱΊ • σʔλϕʔεͱશจݕࡧͷ • ߹ੑΛͱΔ͔ɺૄ݁߹ΛͱΔ͔ͷ • ৽͍͠Riak Search 2.0 •
ਫฏࢄͯ͠εέʔϧΞτͰ͖Δݕࡧ • ݁Ռ߹ੑΛબͿ͜ͱʹΑΓૄ݁߹ԽͱࢄԽ ʹޭʢ͔͠APܕʹʂʣ • ݁Ռ߹ੑΛબͨ͠ઃܭʹΑΓӡ༻͕؆୯ʹ
Questions?
FAQ •ΦϯϥΠϯͷεΩʔϚมߋʁ •μΠφϛοΫεΩʔϚʁ
outline • σʔλϕʔεʹೖ͍ͬͯΔσʔλΛશจݕࡧ͍ͨ͠→ࢁͷΠϯσοΫε͕Ͱ͖ΔɺFKeyͰ ΠϯσοΫε…ͱ͍͏͚ͩͰ • શจݕࡧΤϯδϯͷछྨ • ֎෦ܕʢ߹ੑΛҡ࣋͠ʹ͍͕͘Ԡ༻͕ޮ͘ɺރΕͨιϑτΣΞϓϥάΠϯ͕ଟ͍ʣ • ΈࠐΈܕʢ߹ੑΛҡ͍͕࣋͢͠ɺϝϯςφϯε͕͍͠ʣ
• Riakͦͷதؒɹόϯυϧͯ͠ΔͷͰָɺ͚ͩͲ࣮ମSolrɺ߹ੑΛࣗಈతʹҡ࣋͢Δ • όϯυϧͷํ๏ɺ߹ੑɺࢄͷํ๏ • ݕࡧΠϯσοΫε͚ͩΛࢄͤ͞Δͷ͍͠ • DBͱΠϯσοΫεͷ߹ɺϨϓϦΧͱϨϓϦΧͷ߹ɺJepsenͷͳ͠