Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Consistency types of Riak
Search
UENISHI Kota
June 04, 2014
Technology
2
360
Consistency types of Riak
Riakの整合性は2.0からもっといろいろ調整できるようになるので、その解説(25分)
UENISHI Kota
June 04, 2014
Tweet
Share
More Decks by UENISHI Kota
See All by UENISHI Kota
Storage Systems in Preferred Networks
kuenishi
0
36
Metadata Management in Distributed File Systems
kuenishi
2
510
Behind The Scenes: Cloud Native Storage System for AI
kuenishi
2
410
Apache Ozone behind Simulation and AI Industries
kuenishi
0
380
Distributed Deep Learning with Chainer and Hadoop
kuenishi
3
1.2k
A Few Ways to Accelerate Deep Learning
kuenishi
0
1.1k
Introducing Retz
kuenishi
5
1.2k
Introducing Retz and how to develop practical frameworks
kuenishi
3
740
Formalization and Proof of Distributed Systems (ja)
kuenishi
10
6.4k
Other Decks in Technology
See All in Technology
あなたの知らない OneDrive
murachiakira
0
240
小さなチーム 大きな仕事 - 個人開発でAIをフル活用する
himaratsu
0
130
Goss: New Production-Ready Go Binding for Faiss #coefl_go_jp
bengo4com
0
1.1k
AIエージェントの開発に必須な「コンテキスト・エンジニアリング」とは何か──プロンプト・エンジニアリングとの違いを手がかりに考える
masayamoriofficial
0
430
人を動かすことについて考える
ichimichi
2
340
JavaScript 研修
recruitengineers
PRO
4
730
アジャイルテストで高品質のスプリントレビューを
takesection
0
120
【 LLMエンジニアがヒューマノイド開発に挑んでみた 】 - 第104回 Machine Learning 15minutes! Hybrid
soneo1127
0
140
夢の印税生活 / Life on Royalties
tmtms
0
290
RAID6 を楔形文字で組んで現代人を怖がらせましょう(実装編)
mimifuwa
1
320
R-SCoRe: Revisiting Scene Coordinate Regression for Robust Large-Scale Visual Localization
takmin
0
440
どこで動かすか、誰が動かすか 〜 kintoneのインフラ基盤刷新と運用体制のシフト 〜
ueokande
0
200
Featured
See All Featured
Thoughts on Productivity
jonyablonski
69
4.8k
Code Reviewing Like a Champion
maltzj
525
40k
Become a Pro
speakerdeck
PRO
29
5.5k
A better future with KSS
kneath
239
17k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
49
3k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
17k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
Documentation Writing (for coders)
carmenintech
73
5k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
900
Six Lessons from altMBA
skipperchong
28
4k
Transcript
Consistency types in Riak 2014/6/4 Riak Meetup #4 @kuenishi
ࣗݾհ •UENISHI Kota @ kuenishi •github, twitter, … •Erlangྺ5ɺࢄγεςϜྺ6 •࠷ۙ
Riak CS ͷ։ൃ
Consistencyɹ߹ੑ •ʮNoSQLίϯγεςϯγʔ͕ͳ͍ʯ •ʮSQLʹίϯγεςϯγʔ͕͋Δʯ •ʢ; °Д°ʣ
SQL => ߹ੑʁ NO
߹ੑ㲈 τϥϯβΫγϣϯ ˚
߹ੑʹෆม݅
ෆม݅ •ཧతͳ߹ੑ •ςʔϒϧؒͷ੍݅ɺΠϯσοΫε •ཧతͳ߹ੑ •ෳ͕ඞͣಉ͡Α͏ʹ؍ଌ͞ΕΔ͔
RiakͰ •Siblings •CRDT (2.0~) •Strong Consistency (2.0~)
Why Siblings?
Consistent Hashing • 160-bit Ωʔۭؒ • ۭؒΛ͢Δ • ύʔςΟγϣϯϊʔ υ͕ݸผཧ
• ϨϓϦΧNݸͷύʔ ςΟγϣϯʹίϐʔ͞ ΕΔ OPEF OPEF OPEF OPEF hash(“meetups/spamham”) N=3
CAPఆཧͱཧͷDB •ͲΜͳނোʹରͯ͠ (partition tolerance) •σʔλৗʹ߹͓ͯ͠Γ (consistency) •γεςϜ͕ࢭ·Δ͜ͱͳ͍ (availability) ͜ͷ3ͭΛಉ࣌ʹຬͨ͢γεςϜଘࡏ͠ͳ͍
Consistency͍͠ •ߋ৽ΛࢭΊΔʢAvailabilityΛԼ͛Δʣ͔ɺߋ৽ͷ্ॻ͖Λ ڐ͢ʢσʔλΛࣦ͏ʣ͔͔͠બࢶ͕ͳ͍ Server2 Server1 Server3 PUT V=42 PUT V=0
V=?
ConsistencyͷΘΓʹ •ͱΓ͋͑ͣෳͷόʔδϣϯͷڞଘΛڐ͢ •Ͳͷόʔδϣϯ͕ਖ਼͍͔͠ɺ͘͠Ϛʔδ͢Δ͔ΛRead࣌ʹܾఆ Server2 Server1 Server3 PUT V=42 PUT V=0
V=0 or 42 V=0 V=0 or 42 V=42
APΛ࣮ݱ •ωοτϫʔΫஅ͕ى͖͍ͯͯͱΓ͋͑ͣॻ͖ࠐΈΛڐ͢ Server2 Server1 Server3 PUT V=42 PUT V=0 Server4
෮چͨ͠Βॻ͖͢ ྆ํ͓࣋ͬͯ͘
γϣοϐϯάΧʔτͷྫ •UnionΛͱΕΑ͍ Server2 Server1 Server3 PUT cart=[a,b,d] PUT cart=[a,b,c] union([a,b,c],
[a,b,d]) => [a,b,c,d] [a,b,c] [a,b,c] or [a,b,d] [a,b,d]
Siblings riak_object = Riak.fetch(bucket, key)! riak_object.version! riak_object.has_siblings! for value in
riak_object.values: …! riak_object.data = new_value! riak_object.store!
Siblings ͷෆม݅ •ಛʹͳ͠…͍͋͑ͯ͏ͱϨϓϦΧͷू߹ •Data = R1 ∪ R2 ∪ R3
ෳόʔδϣϯΛ ڐ͢͜ͱͷ •ϓϩάϥϛϯά͕͍͠ʢτϥϯβΫγϣϯૉ Β͍͠ʣ •ݱ࣮ੈքγϣοϐϯάΧʔτͱΧϯλʔ͚ͩ Ͱͳ͍ •҆શͳMerge, update͕Ͱ͖ΔσʔλߏΛຖճ ߟ͑ͳ͚ΕͳΒͳ͍ •͍ͬͯΔ͏ͪʹࣅͨΑ͏ͳϥΠϒϥϦ͕͋ͪ͜
ͪͰग़དྷ্͕Δ
ͳ͍ͥ͠ͷ͔ʁ •σʔλͷWriteͱWrite͕ೖΕସΘΓ͏ ΔʹSerializableͲ͜Ζ͔WriteҰ؏ ͨ͠ঢ়ଶʹͰ͖ͳ͍ Server2 Server1 Server3 w1 w2 w1
w2 w2 (w1 lost)
͑: CRDT •ʮෳՄೳͳՄσʔλܕʯ •Conflict-Free Replicated Data Types •Commutative Replicated Data
Types •… •(Going to be included in Riak 2.0) ) CRDTͷ࡞ऀLogical Monotinicy ͱ͍͏ݴ༿͍ͬͯͳ͍
CRDTͷෆม݅ •σʔλʹର͢ΔՄͳૢ࡞ͷΈΛڐ͢ʂ Data = update(w2, update(w1, Data0)) = update(w1, update(w2,
Data0)) Data = merge(update(w2, Data0), Data)
CRDT in Riak 2.0 •KVSͷVʹʮܕʯΛ࣋ͨͤͯɺܕʹΑͬͯ UpdateͱMergeͷϩδοΫΛܾΊΔ •Read࣌ʹMerge͕αʔόʔଆͰࣗಈతʹ࣮ ߦ͞ΕΔ •ΞϓϦέʔγϣϯܕΛࢦఆ͢Δ͚ͩͰΑ͘ɺ ෳόʔδϣϯͷϋϯυϦϯά͕ෆཁʹͳΔ
CRDT riak_object = Riak.fetch(bucket, key)! riak_object.type => counter|set|…! riak_object.set <<
element! riak_object.set.delete(old_element)! riak_object.store!
CRDT example •PN-Counter •Set •OR-sets •LWW-register •Graph…
PN-Counter • merge • {a: {1,-1}, b: {1,0}, c: {2,0}}
• {a: {0,0}, b: {2, 0}, c: {0, -2}} • => {a: {1,-1}, b:{2,0}, c:{2,-2}} => 2 • update • a͕ {increment, 3} Λड͚͚Δͱ • {a: {4,-1}, b: {1,0}, c: {2,0}}
OR-Sets • merge • {a:{“foo”:true}, b:{“bar”:false}} • + {a:{“foo”:true}, b:{“foo”:false,
“bar”:false}} • => {a:{“foo”:true}, b:{“foo”:false, “bar”:true}} • => [“bar”] • update • add: {a:{}} => +”foo” => {a:{“foo”:false}} • remove: {a: {“foo”:false}} => {a: {“foo”:true}}
Ϣʔεέʔε •ΫϦοΫͷΧϯτ (G-counter) • riak-server/types/counters/buckets/likes/datatypes/basho.com -d 1 •γϣοϐϯάΧʔτ (OR-sets) •ϩάΠϯϢʔβʔ
(PN-counter) •͜ΕΒͷΈ߹Θͤ (map & LWW-register, boolean) •{ name : “basho.com”, likes: 20000, users: 3000, links: [ “basho.co.jp”, “basho.co.uk” ], cool: true }
Ͱ͖ͳ͍͜ͱ •ʮ0Ҏ্ʯͷPN-counter •ϢχʔΫͳIDൃߦ •ͦͷଞCAS͕ඞཁͳσʔλߏͱૢ࡞ •RESTfulͳૢ࡞
Strong Consistency •ෆม݅: t(w1) > t(w2) or t(w2) > t(w1)
•Sequencial Consistencyʹ͍ۙ Riak c1 c2 get get v1 v1 w1(v1) ok w2(v1) fail
Strong Consistency •͍ͪͲGETͯ͠ɺͦͷόʔδϣϯʹର͢ Δૢ࡞ΛૹΔ •ͦͷؒʹߋ৽͞Ε͍ͯͨΒࣦഊɺޭ •MultiPaxos
Strong Consistency do {! riak_object = Riak.fetch(bucket, key)! riak_object.data =
new_value! } while (riak_object.store != ok)
·ͱΊ •Riak 1.x Մ༻ੑͷ͋Δࢄσʔλϕʔε •RESTfulͳઃܭΛ͢ΔͳΒSiblings •2.0 ͔Βෳͷ߹ੑϞσϧΛબͰ͖ΔΑ ͏ʹͳͬͨ •ΞϓϦΛ؆୯ʹ࡞Γ͍ͨͳΒCRDT •CASతߋ৽Λ͍ͨ͠ͳΒStrong
Consistency
Questions? •Riak 2.0 Λָ͠Έʹ͍ͯͩ͘͠͞ •Web: http://basho.co.jp •Twitter: @BashoJapan •Me:
[email protected]
•ML:
[email protected]