Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
CRDTs - The science behind Phoenix Presence
Search
Maciej Kaszubowski
May 25, 2017
Programming
2
250
CRDTs - The science behind Phoenix Presence
Maciej Kaszubowski
May 25, 2017
Tweet
Share
More Decks by Maciej Kaszubowski
See All by Maciej Kaszubowski
Error-free Elixir
mkaszubowski
0
290
Modular Design in Elixir (ElixirConf EU 2019)
mkaszubowski
2
670
The Big Ball of Nouns
mkaszubowski
0
88
Modular Design in Elixir
mkaszubowski
1
370
Our three years with Elixir
mkaszubowski
0
210
Concurrency Basics for Elixir
mkaszubowski
0
100
Distributed Elixir
mkaszubowski
0
120
Software Architecture
mkaszubowski
0
120
Let it crash - fault tolerance in Elixir/OTP
mkaszubowski
0
410
Other Decks in Programming
See All in Programming
CloudRun, Spanner に対する負荷試験の反省と オブザーバビリティによるアプローチ
oyasumipants
1
160
もう僕は OpenAPI を書きたくない
sgash708
6
1.9k
From the Wild into the Clouds - Laravel Meetup Talk
neverything
0
180
自力でTTSモデルを作った話
zgock999
0
120
仕様変更に耐えるための"今の"DRY原則を考える
mkmk884
9
3.3k
Google Cloudとo11yで実現するアプリケーション開発者主体のDB改善
nnaka2992
1
110
PEPCは何を変えようとしていたのか
ken7253
3
310
コードを読んで理解するko build
bells17
1
120
Jakarta EE meets AI
ivargrimstad
0
630
PHPのバージョンアップ時にも役立ったAST
matsuo_atsushi
0
230
color-scheme: light dark; を完全に理解する
uhyo
7
510
CloudNativePGを布教したい
nnaka2992
0
120
Featured
See All Featured
Docker and Python
trallard
44
3.3k
Build The Right Thing And Hit Your Dates
maggiecrowley
34
2.5k
Embracing the Ebb and Flow
colly
84
4.6k
The Power of CSS Pseudo Elements
geoffreycrofte
75
5.5k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5.2k
Making the Leap to Tech Lead
cromwellryan
133
9.1k
Being A Developer After 40
akosma
89
590k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
11
540
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.3k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
The Language of Interfaces
destraynor
156
24k
Transcript
The Problem
None
Server Node 1 Server Node 2
Node A Node B [User] [User] [User] [] [] [User]
User connects User disconnects
There's no global time
Node A Node B [User] [User] [User] [] [] [User]
User connects User disconnects
Node A Node B [User] [User] [User] [] [] [User]
User connects User disconnects
Node A Node B
P.track(self, "users", "U1", %{})
P.track(self, "users", "U1", %{}) P.list("users") %{"U1" %{metas: [%{phx_ref: "…"}]}}
P.track(self, "users", "U1", %{}) P.list("users") P.track(self, "users", "U2", %{}) %{"U1"
%{metas: [%{phx_ref: "…"}]}}
P.track(self, "users", "U1", %{}) P.list("users") P.track(self, "users", "U2", %{}) P.list("users")
%{"U1" %{metas: [%{phx_ref: "…"}]}, "U2" %{metas: [%{phx_ref: "…"}]}} %{"U1" %{metas: [%{phx_ref: "…"}]}}
P.track(self, "users", "U1", %{}) P.list("users") P.track(self, "users", "U2", %{}) P.list("users")
%{"U1" %{metas: [%{phx_ref: "…"}]}, "U2" %{metas: [%{phx_ref: "…"}]}} P.track(self, "users", "U1", %{}) P.untrack(self, "users", "U1") %{"U1" %{metas: [%{phx_ref: "…"}]}}
P.track(self, "users", "U1", %{}) P.list("users") P.track(self, "users", "U2", %{}) P.list("users")
%{"U1" %{metas: [%{phx_ref: "…"}]}, "U2" %{metas: [%{phx_ref: "…"}]}} P.track(self, "users", "U1", %{}) P.untrack(self, "users", "U1") P.list("users") %{"U1" %{metas: [%{phx_ref: "…"}]}, "U2" %{metas: [%{phx_ref: "…"}]}} P.list("users") %{"U1" %{metas: [%{phx_ref: "…"}]}, "U2" %{metas: [%{phx_ref: "…"}]}} %{"U1" %{metas: [%{phx_ref: "…"}]}}
CRDTs The science behind Phoenix Presence Maciej Kaszubowski
Conflict-free Replicated Data Type
Alternatives
• Single source of truth (DB) • Consensus algorithm •
Resolving conflicts manually
Why CRDTs?
Eventually consistent Highly available Easy to use
Eventually consistent Highly available Easy to use Hard to create
:(
Features
1. Commutative 2. Associative 3. Idempotent x y = y
x (x y) z = x (y z) x x = x
Server Node 1 Server Node 2 Server Node 3
Client Client Client Client Client Client Client Client Client Server
Examples
Counters
Node A Node B +5 5 8 -2 3 +5
User connects User disconnects 0 0 5 3 8
Node A Node B +5 5 8 -2 8 +5
User connects User disconnects 0 0 5 3 13 10
G-Counter Grow-only counter
Node 2: 3 Value=8 Node 1: 5 Node 3: 1
Merge
Node 2: 3 Value=9 Node 1: 6 Node 3: 1
+1 Merge
PN-Counter Positive-Negative Counter
Node 2: P=2 N=2 Value=7 Node 1: P=5 N=2 Node
3: P=4 N=0 3 0 4 Merge
Node 2: P=2 N=2 Value=6 Node 1: P=5 N=3 Node
3: P=4 N=0 2 0 4 +1 Merge
Node 2: P=2 N=2 Value=8 Node 1: P=5 N=3 Node
3: P=6 N=0 2 0 6 +2 +1 Merge
Sets
Node A Node B [User] [User] [User] [] [] [User]
User connects User disconnects
G-Set Grow-only set
Node 2: [1,2] Value=[1,2,3,4] Node 1: [1] Node 3: [3,4]
Merge
Node 2: [1,2] Value=[1,2,3,4,5] Node 1: [1,5] Node 3: [3,4]
Merge
2P-Set Two-phase set
Node 2: [1,2],[] Value=[1,2,3,4] Node 1: [1],[] Node 3: [3,4],[]
Merge
Node 2: [1,2],[] Value=[1,2,3,4] Node 1: [1],[] Node 3: [3,4],[]
Merge G-Set for adds
Node 2: [1,2],[] Value=[1,2,4] Node 1: [1],[] Node 3: [3,4],[3]
Merge G-Set for removals
Elements cannot be re-added
Removes win
Node A Node B [ ] Add Remove [1] [
] [1]
Node A Node B [ ] [1] Add Remove [1]
[1] [ ] [1] [ ] [1]
Node A Node B [ ] [1] Add Remove [1]
[1] [ ] [1] [ ] [1]
Node A Node B [ ] [1] [1] Add Remove
[1] [1] [1] [ ] [1] [1] [1] [ ] [1]
OR-Set Observed-remove set
Node A Node B [ ] Add Remove [{A,1}] [
] [{A,1}]
Node A Node B [ ] [ ] Add Remove
[{A,1}] [{A,1}] [ ] [ ] [{A,1}] [{A,1}, ]
Node A Node B [ ] [ 1 ] Add
Remove [{A,1}] [{A,1}] [ ] [ ] [{A,1}] [{A,1},{A,2}]
Node A Node B [ ] [ 1 ] Add
Remove [{A,1}] [{A,1}] [ ] [ ] [{A,1}] [{A,1},{A,2}]
Node A Node B [ ] [ 1 ] Add
Remove [{A,1}] [{A,1}] [ ] [ ] [{A,1}] [{A,1},{A,2}] [ 1 ] [{A,1},{A,2}] [{A,1},{A,2}] [ 1 ]
Node A Node B [ ] [ 1 ] Add
Remove [{A,1}] [{A,1}] [ ] [ ] [{A,1}] [{A,1},{A,2}] [ 1 ] [{A,1},{A,2}] [{A,1},{A,2}] [ 1 ] [A] [A]
Add 1000 Remove 1000 Add Remove …
ORSWOT Observed-remove set without tombstones
None
None
Use Cases
None
None
None
None
• Load balancing / routing • Mobile clients synchronisation •
Temporary data on the servers • Avoiding work duplication • Collaborative editing
None
lasp-lang.readme.io
Summary
You're (almost) always designing a distributed system
Think about failures
Choose the correct tool for the job
References • https://medium.com/@istanbul_techie/a-look-at-conflict-free-replicated-data- types-crdt-221a5f629e7e • http://basho.com/posts/technical/distributed-data-types-riak-2-0/ • http://highscalability.com/blog/2014/10/13/how-league-of-legends-scaled- chat-to-70-million-players-it-t.html •
https://hal.inria.fr/inria-00609399v1/document • https://developers.soundcloud.com/blog/roshi-a-crdt-system-for- timestamped-events
Thanks!