Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Riak 2.0における全文検索 と整合性の問題
Search
UENISHI Kota
November 19, 2014
Technology
2
820
Riak 2.0における全文検索 と整合性の問題
WebDB Forum 2014 のBashoジャパンの発表スライドです
UENISHI Kota
November 19, 2014
Tweet
Share
More Decks by UENISHI Kota
See All by UENISHI Kota
Storage Systems in Preferred Networks
kuenishi
0
31
Metadata Management in Distributed File Systems
kuenishi
2
510
Behind The Scenes: Cloud Native Storage System for AI
kuenishi
2
400
Apache Ozone behind Simulation and AI Industries
kuenishi
0
370
Distributed Deep Learning with Chainer and Hadoop
kuenishi
3
1.2k
A Few Ways to Accelerate Deep Learning
kuenishi
0
1.1k
Introducing Retz
kuenishi
5
1.1k
Introducing Retz and how to develop practical frameworks
kuenishi
3
730
Formalization and Proof of Distributed Systems (ja)
kuenishi
10
6.4k
Other Decks in Technology
See All in Technology
クラウド開発の舞台裏とSRE文化の醸成 / SRE NEXT 2025 Lunch Session
kazeburo
1
590
サービスを止めるな! DDoS攻撃へのスマートな備えと最前線の事例
coconala_engineer
1
180
How Do I Contact Jetblue Airlines® Reservation Number: Fast Support Guide
thejetblueairhelpsupport
0
150
ABEMAの本番環境負荷試験への挑戦
mk2taiga
5
1.3k
shake-upを科学する
rsakata
7
1k
セキュアなAI活用のためのLiteLLMの可能性
tk3fftk
1
340
サイバーエージェントグループのSRE10年の歩みとAI時代の生存戦略
shotatsuge
4
1k
Maintainer Meetupで「生の声」を聞く ~講演だけじゃないKubeCon
logica0419
0
110
ソフトウェアQAがハードウェアの人になったの
mineo_matsuya
3
200
ClaudeCode_vs_GeminiCLI_Terraformで比較してみた
tkikuchi
1
1.2k
ビジネス職が分析も担う事業部制組織でのデータ活用の仕組みづくり / Enabling Data Analytics in Business-Led Divisional Organizations
zaimy
1
400
モニタリング統一への道のり - 分散モニタリングツール統合のためのオブザーバビリティプロジェクト
niftycorp
PRO
1
520
Featured
See All Featured
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
830
Bash Introduction
62gerente
613
210k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2.2k
Optimizing for Happiness
mojombo
379
70k
It's Worth the Effort
3n
185
28k
Designing Experiences People Love
moore
142
24k
Docker and Python
trallard
45
3.5k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
138
34k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
18
990
4 Signs Your Business is Dying
shpigford
184
22k
Git: the NoSQL Database
bkeepers
PRO
430
65k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Transcript
Riak 2.0ʹ͓͚Δશจݕࡧ ͱ߹ੑͷ WebDB Forum 2014 Basho Japan ্ 2014/11/19
ࣗݾհ • ࢄγεςϜྺ6 • Bashoδϟύϯͷํ͔Βདྷ·ͨ͠ • Riak CSͷ։ൃ • ͦͷଞຊͷ͜ͱ
BashoͱRiak •ࢄσʔλϕʔεʁ •RiakΛ͍ͬͯΔʁ •BashoΛ͍ͬͯΔʁ
We are hiring •࣮ੈքͷࢄγεςϜͷ ʹڵຯ͋Δਓʂ •@BashoJapan •
[email protected]
•APࢦͷσʔλϕʔεʢ݁Ռ߹ੑʣ •҆ఆੑɺ༧ଌՄೳੑ •ʮσʔλΛઈରʹͳ͘͞ͳ͍ʯ •৽͍͠ݕࡧΤϯδϯͱͯ͠SolrΛόϯυϧ
͜Μͳͱ͜ΖͰ ಈ͍͍ͯ·͢Riak •Rovio (Angry Birds) •Yahoo!JAPAN ͷΫϥυετϨʔδ •NHS (ΠΪϦε ࠃຽอݥαʔϏε)
•League of Legends (MMORPG) •ۜߦɺήʔϜɺখചɺηϯαʔɺetc…
CAPఆཧͱཧͷDB •ͲΜͳނোʹରͯ͠ (partition tolerance) •σʔλৗʹ߹͓ͯ͠Γ (consistency) •γεςϜ͕ࢭ·Δ͜ͱͳ͍ (availability) ͜ͷ3ͭΛಉ࣌ʹຬͨ͢γεςϜଘࡏ͠ͳ͍
CAP Theorem • C: ෳͷAtomic Objectʹର͢Δ࿈ ଓͨ͠ૢ࡞ (w1, w3, w4,
….) ͕શ ͯಉҰͰ͋Δ͜ͱ (linearizable) • A: Atomic Objectʹૢ࡞ w1, w2, …Λ࣮ߦͰ͖Δ͜ͱ • P: ωοτϫʔΫ͕ΕΔͱAtomic Objectͷϝοηʔδ͕౸ୡ͠ͳ͍
Is the network reliable?
None
None
How Riak Works (mostly about search)
run Riak with Search $ which java $ sudo apt-get
install riak $ echo “search = on” >> /etc/riak/riak.conf $ riak start $ riak-admin bucket-type create tt $ riak-admin bucket-type activate tt $ curl -XPUT http://localhost:8098/search/index/t $ curl -XPUT -H ‘content-type: application/json’ \ http://localhost:8098/types/tt/props \ -d ‘{“props”:{“search_index”:”t”}}’
Consistent Hashing • 160-bit Ωʔۭؒ • ۭؒΛ͢Δ • ύʔςΟγϣϯϊʔ υ͕ݸผཧ
• ϨϓϦΧNݸͷύʔ ςΟγϣϯʹίϐʔ͞ ΕΔ OPEF OPEF OPEF OPEF hash(“meetups/spamham”) N=3
σʔλϕʔεͷݕࡧ • DBࠐܕ • groongaͳͲ • “LIKE” • Pros •
߹ੑΛҡ͍࣋͢͠ • Cons • ີ݁߹ʹͳΔ • ֎෦࿈ܞܕ • Solr • Elasticsearch • Pros • ૄ݁߹ʹͰ͖Δ • Cons • ߹ੑΛҡ࣋͠ʹ͍͘
DBͱΠϯσοΫεͷ߹ੑ • DBͷσʔλຊମͱɺΠϯσοΫεΛ࣌ʹ҆શʹߋ ৽͢ΔͨΊʹτϥϯβΫγϣϯ͕ඞཁ • ີ݁߹ͷγεςϜͰ͋ΕϩοΫͳͲΛͬͯ߹ੑ Λอো͍͢͠
ૄ݁߹ʹ͢ΔϝϦοτ • ಉ͡ݕࡧΤϯδϯͰ͍ΖΜͳDBʹରԠͰ͖Δ • ӡ༻ϊϋπʔϧɺ։ൃΛूதͰ͖Δ • όʔδϣϯ߹ͷ੍͕؇͞ΕΔ • ยํ͕ނো͍ͯͯ͠͏ยํಈ࡞Ͱ͖Δ •
֦ு͘͢͠ͳΔ
྆ํͱ ͍͍ͱ͜ͲΓ͍ͨ͠
Yokozuna
as Riak Search 2.0 = + σʔλͷӬଓԽ Ϋϥελͷཧ γϯϓϧͳૢ࡞ RVFSZ
JOEFYJOH
Riak Search 2.0 • RiakʹSolrΛόϯυϧ • RiakͷKVΛߋ৽͢Δͱσʔλ͕ࣗಈతʹΠϯσο ΫεԽ͞ΕΔ • SolrͷϓϥάΠϯεΩʔϚɺΫΤϦΛͦͷ··
͏͜ͱ͕Ͱ͖Δ • ԽɺϊʔυՃআRiakͷ୲
ૄ݁߹ʁີ݁߹ʁ • ύοέʔδಉࠝ • ϓϩηεͱͯ͠ૄ݁߹ • ΠϯσοΫεͱσʔλؒ݁Ռ߹ੑΛอো • ΠϯσοΫεͷԽ݁Ռ߹ੑΛอো
Solrͱૄ݁߹ • Riak ϓϩηε͕ Solr ͷϓϩηεΛ fork ͢Δ • ཧɺࢹશͯRiak͕ߦ͏
• ผϓϩηεɺϝϞϦۭؒผ
ΠϯσοΫεԽͷྲྀΕ • Riakͷߋ৽ͱҰ৺ಉମʢdocument-based indexingʣ • ΠϯσοΫεԽʹࣦഊ͢ΔͱPUT͕ࣦഊ
ݕࡧΫΤϦͷॲཧ • Riakͷ2iMapReduceͱಉछͷΫΤϦ͕Δ • 3͓͖ͭͷ vnode ʹΞΫηε͢ΔͷͰɺॏෳͳ͍ ू
߹ੑͷ ͱҟৗܥ ߹ੑҡ࣋ͷͨΊʹߋ৽ϩάΛอ࣋ͯ͠ ϦΧόϦ͢Δͱ͍͏ํ๏ͰɺՄ༻ੑͱ அੑΛ୲อͰ͖ͳ͍ɻRiakͰଞͷ ํ๏Ͱσʔλͷ߹ੑΛҡ࣋͢Δɻ
Α͋͘Δҟৗܥ #1 •ϊʔυނোͰෳΛ3ͭ࡞Εͳ͍ •෮چͯ͠ɺ2ͭͷ··ʹͳͬͯ͠·͏
Α͋͘Δҟৗܥ #1 • ෮چͨ͋͠ͱʹRead Repair͢Δ • GETͨ͠ͱ͖ʹɺσʔλʹ͕ܽؕ͋ͬͨΒ RiakଆͰPUT͢͠ v2 v2
not found v2
Read Repair͕ ݺΕͳ͍ͱ σʔλ͕Βͳ͍ʁ YES
ΫϥΠΞϯτ͔Β GET͕ݺΕͳ͍ͱ σʔλ͕Βͳ͍ʁ NO
AAE (Active Anti Entropy) https://www.flickr.com/photos/51pct/7507525118/
Active Anti Entropy • APࢦͷDBͷσʔλྼԽΛ͙ ͨΊͷόοΫάϥϯυॲཧ • Merkle-TreeΛͬͯύʔςΟγϣ ϯຖͷʮνΣοΫαϜʯΛܭࢉ •
ࠩΛݟ͚ͭͨΒͦ͜ΛRead Repair͢Δ hash(vnode=0, pid=0) hash(vnode=1, pid=0) hash(vnode=2, pid=0)
Α͋͘Δҟৗܥ #2 • Solr͕ΠϯσοΫεԽ͢Δલʹམͪͨ • Key, Valueอଘ͞Ε͕ͨΠϯσοΫεͳ͍
Read Repair͕ ݺΕΕΠϯσο Ϋε͕म෮͞ΕΔʁ NO
AAE (Active Anti Entropy) https://www.flickr.com/photos/51pct/7507525118/
YZ Active Anti Entropy • ݕࡧΠϯσοΫεʹ͍ͭͯ Merkle TreeΛ࡞ • ύʔςΟγϣϯຖʹKey-Valueͷ
Treeͱൺֱ͠ɺ͕ࠩ͋ΕΠϯ σοΫεΛमਖ਼ hash(vnode=0, pid=0, kv) hash(vnode=0, pid=0, yz)
Α͋͘Δҟৗܥ #2 •Searchઐ༻AAE͕ಈ࡞͍ͯ͠Δ •ΠϯσοΫεͱσʔλͷෆ߹Λݟ͚ͭͯ म෮͢Δ AAE
Α͋͘Δҟৗܥ #3 • ωοτϫʔΫ͕Ε͍ͯͨͱ͖ʹ྆ଆͰߋ৽͕ى ͖ͨʢSplit Brainʣ • Hinted HandoffʹΑΓͲͪΒॻ͖ࠐΈޭ
Α͋͘Δҟৗܥ #3 • ෮چ࣌ʹHandoffͰฦ͞Εɺ྆ํͷΛอ࣋ • GET·ͨAAEʹΑͬͯRead Repair͞ΕΔ
ҟৗܥ #1, #2, #3 •Ͳͷҟৗɺಛʹۓٸͷରॲඞཁͳ͍ •߹ੑݕࠪͱम෮͕ࣗಈͰߦΘΕΔ •ϧʔνϯϫʔΫͷӡ༻࡞ۀ͕΄ͱΜͲͳ͍
·ͱΊ • σʔλϕʔεͱશจݕࡧͷ • ߹ੑΛͱΔ͔ɺૄ݁߹ΛͱΔ͔ͷ • ৽͍͠Riak Search 2.0 •
ਫฏࢄͯ͠εέʔϧΞτͰ͖Δݕࡧ • ݁Ռ߹ੑΛબͿ͜ͱʹΑΓૄ݁߹ԽͱࢄԽ ʹޭʢ͔͠APܕʹʂʣ • ݁Ռ߹ੑΛબͨ͠ઃܭʹΑΓӡ༻͕؆୯ʹ
Questions?
FAQ •ΦϯϥΠϯͷεΩʔϚมߋʁ •μΠφϛοΫεΩʔϚʁ
outline • σʔλϕʔεʹೖ͍ͬͯΔσʔλΛશจݕࡧ͍ͨ͠→ࢁͷΠϯσοΫε͕Ͱ͖ΔɺFKeyͰ ΠϯσοΫε…ͱ͍͏͚ͩͰ • શจݕࡧΤϯδϯͷछྨ • ֎෦ܕʢ߹ੑΛҡ࣋͠ʹ͍͕͘Ԡ༻͕ޮ͘ɺރΕͨιϑτΣΞϓϥάΠϯ͕ଟ͍ʣ • ΈࠐΈܕʢ߹ੑΛҡ͍͕࣋͢͠ɺϝϯςφϯε͕͍͠ʣ
• Riakͦͷதؒɹόϯυϧͯ͠ΔͷͰָɺ͚ͩͲ࣮ମSolrɺ߹ੑΛࣗಈతʹҡ࣋͢Δ • όϯυϧͷํ๏ɺ߹ੑɺࢄͷํ๏ • ݕࡧΠϯσοΫε͚ͩΛࢄͤ͞Δͷ͍͠ • DBͱΠϯσοΫεͷ߹ɺϨϓϦΧͱϨϓϦΧͷ߹ɺJepsenͷͳ͠