Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scaling Your Data Safely with Riak
Search
Alex Moore
January 10, 2014
Programming
3
140
Scaling Your Data Safely with Riak
CodeMash 2014 Presentation.
Alex Moore
January 10, 2014
Tweet
Share
Other Decks in Programming
See All in Programming
RailsのPull requestsのレビューの時に私が考えていること
yahonda
5
2.8k
C#/.NETのこれまでのふりかえり
tomokusaba
1
180
【Kaigi on Rails 2024】YOUTRUST スポンサーLT
krpk1900
1
310
現場で役立つモデリング 超入門
masuda220
PRO
15
3.1k
Amazon Qを使ってIaCを触ろう!
maruto
0
370
cXML という電子商取引の トランザクションを支える プロトコルと向きあっている話
phigasui
3
2.3k
みんなでプロポーザルを書いてみた
yuriko1211
0
190
Dev ContainersとGitHub Codespacesの素敵な関係
ymd65536
1
140
ふかぼれ!CSSセレクターモジュール / Fukabore! CSS Selectors Module
petamoriken
0
120
タクシーアプリ『GO』のリアルタイムデータ分析基盤における機械学習サービスの活用
mot_techtalk
4
450
AWS Lambdaから始まった Serverlessの「熱」とキャリアパス / It started with AWS Lambda Serverless “fever” and career path
seike460
PRO
0
140
Content Security Policy入門 セキュリティ設定と 違反レポートのはじめ方 / Introduction to Content Security Policy Getting Started with Security Configuration and Violation Reporting
uskey512
1
480
Featured
See All Featured
Making Projects Easy
brettharned
115
5.9k
Rebuilding a faster, lazier Slack
samanthasiow
79
8.7k
Embracing the Ebb and Flow
colly
84
4.5k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
15
2k
Happy Clients
brianwarren
97
6.7k
Speed Design
sergeychernyshev
24
600
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
169
50k
The Language of Interfaces
destraynor
154
24k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5k
Product Roadmaps are Hard
iamctodd
PRO
49
11k
Build your cross-platform service in a week with App Engine
jlugia
229
18k
Building a Scalable Design System with Sketch
lauravandoore
459
33k
Transcript
with Riak @alexmoore Scaling Data Safely
Internet 1 (2) (3)
Internet 503 Timeout 1 (2) (3)
Internet 1 (2) 3
Internet 1 (2) 3
Internet (2) 3 2 3 1
Internet 4 2 3 (2) 1 3 3
Internet CDN 2 6 7 2 4 3 1 3
3
Internet CDN 5 6 2 4 3 2 1 3
3 7 3
Internet CDN 6 7 2 3 2 1 3 3
Internet CDN P S S 3 6 7 2 5
3 2 1 4 4 3 3 7
Internet CDN 6 8 2 3 2 1 3 3
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic Who’s on first?
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic What the hell have you built? Who’s on first?
Enter Riak
Riak is a distributed, fault-tolerant, highly available database
None
Distributed
Distributed a cluster of nodes each node has an identical
role
Fault- tolerant
Fault- Tolerant symmetrical no Single Point of Failure
Highly Available
Highly Available replicas & redundancy
Scalable
Scalable add or remove nodes linear scalability
Scalable add or remove nodes linear scalability
Multiple Platforms
Multiple Platforms
Multiple Platforms
Predictable Performance
None
Internet CDN 5 6 2 3 2 1 3 3
Internet CDN 5 6 2 3 2 1 3 3
3 3 4 4
1 2 3 4 5
6 7 8 9 10 11 12 Building A Cluster # All Nodes $ riak start ! # Nodes 2..n.each $ riak-‐admin join
[email protected]
! # Any Node $ riak-‐admin cluster plan $ riak-‐admin cluster commit !
None
None
Architecture
Riak is the ops-friendly database
Riak is distributed
Consistent Hashing hash(“bucket/key”)
hash ring
tokenize it
node 0 node 1 node 2
node 0 node 1 node 2 hash(key)
node 0 node 1 node 2 Replicas are stored to
the N - 1 contiguous partitions
node 0 node 1 node 2 hash(“companies/GE”) Replicas are stored
to the N - 1 contiguous partitions
node 0 node 1 node 2 hash(“companies/GE”) Replicas are stored
to the N - 1 contiguous partitions
node 0 node 1 node 2
node 0 node 1 node 2 node 3 + Scaling
out
Quorum requests N R W PR/PW DW
get(“users/clay-davis”)
get(“users/clay-davis”) client Riak
get(“users/clay-davis”) Get Handler (FSM) client Riak
get(“users/clay-davis”) Get Handler (FSM) client Riak hash(“users/clay-davis”) == 10, 11,
12
get(“users/clay-davis”) Get Handler (FSM) client Riak hash(“users/clay-davis”) == 10, 11,
12 Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring
get(“users/clay-davis”) Get Handler (FSM) client Riak get(“users/clay-davis”) Coordinating node Cluster
6 7 8 9 10 11 12 13 14 15 16 The Ring
get(“users/clay-davis”) Get Handler (FSM) client Riak Coordinating node Cluster 6
7 8 9 10 11 12 13 14 15 16 The Ring R=2
get(“users/clay-davis”) Get Handler (FSM) client Riak Coordinating node Cluster 6
7 8 9 10 11 12 13 14 15 16 The Ring R=2 obj
get(“users/clay-davis”) Get Handler (FSM) client Riak R=2 obj obj
get(“users/clay-davis”) Get Handler (FSM) client Riak R=2 obj obj
get(“users/clay-davis”) obj
Read Repair (Anti-Entropy)
replica replica replica
replica replica replica X
replica replica replica replica replica replica
Active Anti-Entropy (self healing clusters)
real-time updates persistent non-blocking disk-based
merkle tree to track changes coordinated at the vnode level
runs as a background process exchange with neighbor vnodes for inconsistencies resolution semantics: trigger read-repair
= hashes marked “dirty”
None
None
None
None
= keys to read-repair
Using Riak
Riak is the ops-friendly database
Riak is friendly to Developers, too!
{“key”: “value”}
key value key value key value key value key value
key value key value key value key value Keys are namespaced into Buckets
Buckets aren’t “real”
✘ ✘ ✘ Buckets aren’t “real”
Metadata Bucket Properties
Aitch Tee Tee Pee GET /buckets/bucket/keys/key PUT /buckets/bucket/keys/key DEL /buckets/bucket/keys/key
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - Starting $ gem install riak-‐client ! book = { isbn: '1111979723', title: 'Moby Dick', author: 'Herman Melville', body: 'Call me Ishmael. Some years ago...', copies_owned: 3 }
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - Create # Starting Client client = Riak::Client.new(protocol: 'pbc', pb_port: 8087, host: '10.0.0.1') ! books_bucket = client.bucket('books') ! robj_moby = books_bucket.new(book[:isbn]) robj_moby.data = book robj_moby.store ! !
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - Read fetched_book = books_bucket.get('1111979723') ! puts fetched_book.raw_data ! => {"isbn":"1111979723","title":"Moby Dick”, "author":"Herman Melville", "body":"Call me Ishmael. Some years ago...","copies_owned":3} ! fetched_book.data['isbn'] ! => "1111979723"
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - Delete fetched_book.delete ! # or ! books_bucket.delete('1111979723')
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - 2I ! fetched_book.indexes['author_bin'] = [] fetched_book.indexes['author_bin'] << fetched_book.data['author'] ! fetched_book.store ! books_bucket.get_index('author_bin','Herman Melville') ! => ["1111979723"]
etc...
JUST ONE MORE THING
Internet CDN 5 6 2 3 2 1 3 3
3 3 4 4
Internet CDN 5 6 2 3 2 1 3 3
3 4
Internet CDN 5 6 2 3 2 1 3 3
3 4
in riak.conf:
search = off in riak.conf:
search = off search = on in riak.conf:
1 2 3 4 5
6 7 8 9 10 11 12 client.create_search_index(“tweets_idx”) ! bucket = Riak::Bucket.new(client, "tweets") bucket.props = {"search_index" => "tweets_idx"} ! # load some tweets into riak… ! results = client.search("tweets", "text:who") ! puts results[‘docs'].first
1 2 3 4 5
6 7 8 9 10 11 12 pp results[‘docs'].first ! {"score"=>"2.50804900000000019489e+00", "_yz_rb"=>"tweets", "_yz_rk"=>"421672897250226176", ...}
More resources: docs.basho.com “Taste of Riak”
Multi-Datacenter Replication (E.G. - Nuke proofing)
Full Sync
Real-Time
None
put(“users/jimmy”)
put(“users/jimmy”)
put(“users/jimmy”)
put(“users/jimmy”)
put(“users/jimmy”)
Internet CDN 5 6 2 3 2 1 3 3
3 4
Internet CDN 5 6 2 3 2 1 3 3
3 4 ?
Internet CDN 5 6 2 3 2 1 3 3
3 4 ?
Q & A Alex Moore - Client Services Engineer, Basho
@alexmoore
[email protected]
http://alexmoore.io