Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Road to masterless multi-node distributed system in Elixir
Search
udit
December 12, 2018
Technology
0
43
Road to masterless multi-node distributed system in Elixir
udit
December 12, 2018
Tweet
Share
More Decks by udit
See All by udit
Scalable dist-sys from grounds up
yudistrange
0
130
Other Decks in Technology
See All in Technology
家族アルバム みてねにおけるGrafana活用術 / Grafana Meetup Japan Vol.1 LT
isaoshimizu
1
770
データベース02: データベースの概念
trycycle
0
160
エンジニアのキャリアをちょっと楽しくする3本の軸/Three Pillars to Make an Engineer's Career More Enjoyable
kwappa
0
2.7k
Cloud Native Java with Spring Boot (CNCF Aarhus, April 2024)
thomasvitale
1
170
Postman v10リリース後を振り返る / Looking back at Postman v10 after release
yokawasa
1
160
LLM開発・活用の舞台裏@2024.04.25
yushin_n
1
350
Tellus の衛星データを見てみよう #mf_fukuoka
kongmingstrap
0
210
EMとして2023年度に頑張ったこと / What we did well in FY2023 as a EM
pauli
1
170
Meta Quest 3 で動く桜マシマシ WebXR アプリを IBM Cloud Code Engine と Babylon.js で作った話
1ftseabass
PRO
0
120
Gitlab本から学んだこと - そーだいなるプレイバック / gitlab-book
soudai
4
440
web-application-security
matsuihidetoshi
0
170
GraphQL 成熟度モデルの紹介と、プロダクトに当てはめた事例 / GraphQL maturity model
mh4gf
7
1.3k
Featured
See All Featured
A designer walks into a library…
pauljervisheath
200
23k
Gamification - CAS2011
davidbonilla
76
4.6k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
187
16k
Facilitating Awesome Meetings
lara
42
5.6k
Building a Modern Day E-commerce SEO Strategy
aleyda
17
6.4k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
221
21k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
322
20k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
274
13k
Infographics Made Easy
chrislema
238
18k
In The Pink: A Labor of Love
frogandcode
138
21k
A Modern Web Designer's Workflow
chriscoyier
689
190k
How STYLIGHT went responsive
nonsquared
92
4.8k
Transcript
The Road to a in Elixir Masterless Multi-node Distributed System
BUILDING AND SERVICES SYSTEMS
•WHY THIS JOURNEY? •THE START •WHERE WE ARE •THE ROAD
AHEAD AGENDA
WHY THIS JOURNEY?
PLATFORM TO DISCOVER AND BUY THE BEST EXPERIENCES IN EVENTS,
TRAVEL AND FOOD IN CITIES ACROSS INDIA
FOUNDED IN 2014 10K+ EVENTS, 3M+ TICKETS IN 2018 ACQUIRED
BY PAYTM IN 2017
MAKE INSIDER A PLATFORM FOR HOSTING AND PARTICIPATING IN DIGITAL
EVENTS
LIVE QUIZ
A PLATFORM THAT ENABLES INTERACTIONS ON A LIVE VIDEO STREAM
interaction by Aman from the Noun Project
FLEXIBLE AND SCALABLE FUEL INTERACTIONS flexible by Anup from the
Noun Project PLATFORM TO
Elixir Functional Programming Immutable Data Structures Concurrency Fault Tolerance Battle
Tested VM Erlang Interoperability
THE START
M W W W W NODE
MASTER CANNOT HANDLE Problem PLAYERS
M W W W W NODE SM SM
UNNECESSARY CHANGE CALLS CASTS
PLAYER STATE TRANSFER GENSERVER ETS
M W W W W NODE C NODE B NODE
A
SENDING MESSAGE TO A PID OVER THE NETWORK COPIES THE
MESSAGE OVER THE NETWORK Problem
Manifold github.com/discordapp/manifold Batching of messages passed between nodes
MANIFOLD WORKS FOR FAN-OUTs WHAT ABOUT FAN-INs? Problem
SO MUCH CHATTER MESSAGE PASSING Problem BECAME EXPENSIVE
MQTT Publisher/Subscriber Messaging Protocol Built on top of TCP/IP Lightweight
and Bandwidth Efficient Quality of Service Data Agnostic Message Queueing Telemetry Transport
Distributed, Scalable and Highly Extensible MQTT message broker Written in
Erlang/OTP http://emqtt.io
E M Q W W W W M /connect /connect
/connect /connect
Problem GLOBAL LOCK ACROSS WORKERS TO HANDLE PLAYER CONNECTIONS
Workers subscribe to /connect/<slot id> Players connect to /connect/<slot id>
Slot = Hash(player_id) T P I O S C SLOTTED
E M Q W W W W M /connect/1 /connect/2
/connect/2 /connect/1
Problem CRASHES APPLICATION
SUPERVISION TREES
Master Worker Worker Worker App Supervisor Master Supervisor Worker Supervisor
CURRENT ARCHITECTURE IS TIGHTLY COUPLED TO QUIZ Problem
HOW TO SUPPORT STATES AND TRANSITIONS FOR DIFFERENT ENGAGEMENTS? Problem
FSM Event State Engagement State gen_statem in Erlang/OTP
WHERE WE ARE
M W W W W Director
WE HAVE IN PLACE BUILDING BLOCKS
HOW TO SCALE IN THE LONG RUN?
WE NEED TO THINK LONG TERM
INTERMISSION
THE ROAD AHEAD
Multi Node
M W W W W M W W W W
GM
libcluster https://github.com/bitwalker/libcluster Form clusters of Erlang nodes Static or Dynamic
Node Membership Custom Strategy to Deal with Nodes Joining/Leaving Notification when Nodes Join/Leave
Resilience
ETS Immortal https://github.com/danielberkompas/immortal Keep ETS alive using a heir process
Keep ETS alive when owner dies Give ownership back to owner after it reboots
NODE A NODE B MASTER MA WA WB MB WA
WA WA WB WB WB MB WB MA WB WB WB WA WA WA WA AND WORKER REPLICA
Masterless
M W W W W GM M W W W
W GM M W W W W GM NODE A NODE C NODE B
CRDT Conflict-free Coordination-free Commutative or Convergent
CRDT Flags Maps Counters Sets
https://kubernetes.io/ SETUP/TEARDOWN ON-DEMAND FOR EVENTS INFRASTRUCTURE
RECAP
• https://elixirschool.com/en/ • https://ferd.ca/the-zen-of-erlang.html • http://blog.plataformatec.com.br/ • http://basho.com/tag/riak/ • https://www.theerlangelist.com/
Resources
Books Litte Elixir and OTP https://www.manning.com/books/the-little-elixir-and-otp-guidebook Learn you some Erlang
http://learnyousomeerlang.com Designing for Scalability with Erlang/OTP http://shop.oreilly.com/product/0636920024149.do
DISTRIBUTED SYSTEMS ARE HARD ELIXIR MAKES THINGS EASIER
Thanks! @kirang89 @yudistrange https://bit.ly/2QVT7CB