Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scaling Your Data Safely with Riak
Search
Alex Moore
January 10, 2014
Programming
3
140
Scaling Your Data Safely with Riak
CodeMash 2014 Presentation.
Alex Moore
January 10, 2014
Tweet
Share
Other Decks in Programming
See All in Programming
AI時代の『改訂新版 良いコード/悪いコードで学ぶ設計入門』 / ai-good-code-bad-code
minodriven
20
8.1k
코딩 에이전트 체크리스트: Claude Code ver.
nacyot
0
680
Is Xcode slowly dying out in 2025?
uetyo
1
280
Rails Frontend Evolution: It Was a Setup All Along
skryukov
0
160
RailsGirls IZUMO スポンサーLT
16bitidol
0
190
git worktree × Claude Code × MCP ~生成AI時代の並列開発フロー~
hisuzuya
1
590
Hack Claude Code with Claude Code
choplin
4
2.3k
Deep Dive into ~/.claude/projects
hiragram
14
2.6k
たった 1 枚の PHP ファイルで実装する MCP サーバ / MCP Server with Vanilla PHP
okashoi
1
270
Goで作る、開発・CI環境
sin392
0
240
AIともっと楽するE2Eテスト
myohei
7
2.8k
ニーリーにおけるプロダクトエンジニア
nealle
0
870
Featured
See All Featured
Scaling GitHub
holman
460
140k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
107
19k
Navigating Team Friction
lara
187
15k
Intergalactic Javascript Robots from Outer Space
tanoku
271
27k
Git: the NoSQL Database
bkeepers
PRO
430
65k
Code Reviewing Like a Champion
maltzj
524
40k
Faster Mobile Websites
deanohume
307
31k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.7k
How to Think Like a Performance Engineer
csswizardry
25
1.7k
Optimizing for Happiness
mojombo
379
70k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
48
2.9k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
44
2.4k
Transcript
with Riak @alexmoore Scaling Data Safely
Internet 1 (2) (3)
Internet 503 Timeout 1 (2) (3)
Internet 1 (2) 3
Internet 1 (2) 3
Internet (2) 3 2 3 1
Internet 4 2 3 (2) 1 3 3
Internet CDN 2 6 7 2 4 3 1 3
3
Internet CDN 5 6 2 4 3 2 1 3
3 7 3
Internet CDN 6 7 2 3 2 1 3 3
Internet CDN P S S 3 6 7 2 5
3 2 1 4 4 3 3 7
Internet CDN 6 8 2 3 2 1 3 3
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic Who’s on first?
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic What the hell have you built? Who’s on first?
Enter Riak
Riak is a distributed, fault-tolerant, highly available database
None
Distributed
Distributed a cluster of nodes each node has an identical
role
Fault- tolerant
Fault- Tolerant symmetrical no Single Point of Failure
Highly Available
Highly Available replicas & redundancy
Scalable
Scalable add or remove nodes linear scalability
Scalable add or remove nodes linear scalability
Multiple Platforms
Multiple Platforms
Multiple Platforms
Predictable Performance
None
Internet CDN 5 6 2 3 2 1 3 3
Internet CDN 5 6 2 3 2 1 3 3
3 3 4 4
1 2 3 4 5
6 7 8 9 10 11 12 Building A Cluster # All Nodes $ riak start ! # Nodes 2..n.each $ riak-‐admin join riak@node1.internal.company.com ! # Any Node $ riak-‐admin cluster plan $ riak-‐admin cluster commit !
None
None
Architecture
Riak is the ops-friendly database
Riak is distributed
Consistent Hashing hash(“bucket/key”)
hash ring
tokenize it
node 0 node 1 node 2
node 0 node 1 node 2 hash(key)
node 0 node 1 node 2 Replicas are stored to
the N - 1 contiguous partitions
node 0 node 1 node 2 hash(“companies/GE”) Replicas are stored
to the N - 1 contiguous partitions
node 0 node 1 node 2 hash(“companies/GE”) Replicas are stored
to the N - 1 contiguous partitions
node 0 node 1 node 2
node 0 node 1 node 2 node 3 + Scaling
out
Quorum requests N R W PR/PW DW
get(“users/clay-davis”)
get(“users/clay-davis”) client Riak
get(“users/clay-davis”) Get Handler (FSM) client Riak
get(“users/clay-davis”) Get Handler (FSM) client Riak hash(“users/clay-davis”) == 10, 11,
12
get(“users/clay-davis”) Get Handler (FSM) client Riak hash(“users/clay-davis”) == 10, 11,
12 Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring
get(“users/clay-davis”) Get Handler (FSM) client Riak get(“users/clay-davis”) Coordinating node Cluster
6 7 8 9 10 11 12 13 14 15 16 The Ring
get(“users/clay-davis”) Get Handler (FSM) client Riak Coordinating node Cluster 6
7 8 9 10 11 12 13 14 15 16 The Ring R=2
get(“users/clay-davis”) Get Handler (FSM) client Riak Coordinating node Cluster 6
7 8 9 10 11 12 13 14 15 16 The Ring R=2 obj
get(“users/clay-davis”) Get Handler (FSM) client Riak R=2 obj obj
get(“users/clay-davis”) Get Handler (FSM) client Riak R=2 obj obj
get(“users/clay-davis”) obj
Read Repair (Anti-Entropy)
replica replica replica
replica replica replica X
replica replica replica replica replica replica
Active Anti-Entropy (self healing clusters)
real-time updates persistent non-blocking disk-based
merkle tree to track changes coordinated at the vnode level
runs as a background process exchange with neighbor vnodes for inconsistencies resolution semantics: trigger read-repair
= hashes marked “dirty”
None
None
None
None
= keys to read-repair
Using Riak
Riak is the ops-friendly database
Riak is friendly to Developers, too!
{“key”: “value”}
key value key value key value key value key value
key value key value key value key value Keys are namespaced into Buckets
Buckets aren’t “real”
✘ ✘ ✘ Buckets aren’t “real”
Metadata Bucket Properties
Aitch Tee Tee Pee GET /buckets/bucket/keys/key PUT /buckets/bucket/keys/key DEL /buckets/bucket/keys/key
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - Starting $ gem install riak-‐client ! book = { isbn: '1111979723', title: 'Moby Dick', author: 'Herman Melville', body: 'Call me Ishmael. Some years ago...', copies_owned: 3 }
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - Create # Starting Client client = Riak::Client.new(protocol: 'pbc', pb_port: 8087, host: '10.0.0.1') ! books_bucket = client.bucket('books') ! robj_moby = books_bucket.new(book[:isbn]) robj_moby.data = book robj_moby.store ! !
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - Read fetched_book = books_bucket.get('1111979723') ! puts fetched_book.raw_data ! => {"isbn":"1111979723","title":"Moby Dick”, "author":"Herman Melville", "body":"Call me Ishmael. Some years ago...","copies_owned":3} ! fetched_book.data['isbn'] ! => "1111979723"
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - Delete fetched_book.delete ! # or ! books_bucket.delete('1111979723')
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - 2I ! fetched_book.indexes['author_bin'] = [] fetched_book.indexes['author_bin'] << fetched_book.data['author'] ! fetched_book.store ! books_bucket.get_index('author_bin','Herman Melville') ! => ["1111979723"]
etc...
JUST ONE MORE THING
Internet CDN 5 6 2 3 2 1 3 3
3 3 4 4
Internet CDN 5 6 2 3 2 1 3 3
3 4
Internet CDN 5 6 2 3 2 1 3 3
3 4
in riak.conf:
search = off in riak.conf:
search = off search = on in riak.conf:
1 2 3 4 5
6 7 8 9 10 11 12 client.create_search_index(“tweets_idx”) ! bucket = Riak::Bucket.new(client, "tweets") bucket.props = {"search_index" => "tweets_idx"} ! # load some tweets into riak… ! results = client.search("tweets", "text:who") ! puts results[‘docs'].first
1 2 3 4 5
6 7 8 9 10 11 12 pp results[‘docs'].first ! {"score"=>"2.50804900000000019489e+00", "_yz_rb"=>"tweets", "_yz_rk"=>"421672897250226176", ...}
More resources: docs.basho.com “Taste of Riak”
Multi-Datacenter Replication (E.G. - Nuke proofing)
Full Sync
Real-Time
None
put(“users/jimmy”)
put(“users/jimmy”)
put(“users/jimmy”)
put(“users/jimmy”)
put(“users/jimmy”)
Internet CDN 5 6 2 3 2 1 3 3
3 4
Internet CDN 5 6 2 3 2 1 3 3
3 4 ?
Internet CDN 5 6 2 3 2 1 3 3
3 4 ?
Q & A Alex Moore - Client Services Engineer, Basho
@alexmoore amoore@basho.com http://alexmoore.io