The Road to a Masterless Multi-node Distributed System in Elixir

In this talk, we will cover our journey towards building a masterless multi-node distributed system in Elixir. We’ll talk about the challenges faced, decisions made and how they shaped/evolved the architecture.

Presented by Udit Kumar(https://speakerdeck.com/yudistrange) and Myself.

Transcript

1. The Road to a in Elixir Masterless Multi-node Distributed System
2. BUILDING AND SERVICES SYSTEMS
3. •WHY THIS JOURNEY? •THE START •WHERE WE ARE •THE ROAD AHEAD AGENDA
4. WHY THIS JOURNEY?
5. PLATFORM TO DISCOVER AND BUY THE BEST EXPERIENCES IN EVENTS, TRAVEL AND FOOD IN CITIES ACROSS INDIA
6. FOUNDED IN 2014 10K+ EVENTS, 3M+ TICKETS IN 2018 ACQUIRED BY PAYTM IN 2017
7. MAKE INSIDER A PLATFORM FOR HOSTING AND PARTICIPATING IN DIGITAL EVENTS
8. LIVE QUIZ
9. A PLATFORM THAT ENABLES INTERACTIONS ON A LIVE VIDEO STREAM interaction by Aman from the Noun Project
10. FLEXIBLE AND SCALABLE FUEL INTERACTIONS flexible by Anup from the Noun Project PLATFORM TO
11. Elixir Functional Programming Immutable Data Structures Concurrency Fault Tolerance Battle Tested VM Erlang Interoperability
12. THE START
13. M W W W W NODE
14. MASTER CANNOT HANDLE Problem PLAYERS
15. M W W W W NODE SM SM
16. UNNECESSARY CHANGE CALLS CASTS
17. PLAYER STATE TRANSFER GENSERVER ETS
18. M W W W W NODE C NODE B NODE A
19. SENDING MESSAGE TO A PID OVER THE NETWORK COPIES THE MESSAGE OVER THE NETWORK Problem
20. Manifold github.com/discordapp/manifold Batching of messages passed between nodes
21. MANIFOLD WORKS FOR FAN-OUTs WHAT ABOUT FAN-INs? Problem
22. SO MUCH CHATTER MESSAGE PASSING Problem BECAME EXPENSIVE
23. MQTT Publisher/Subscriber Messaging Protocol Built on top of TCP/IP Lightweight and Bandwidth Efficient Quality of Service Data Agnostic Message Queueing Telemetry Transport
24. Distributed, Scalable and Highly Extensible MQTT message broker Written in Erlang/OTP http://emqtt.io
25. E M Q W W W W M /connect /connect /connect /connect
26. Problem GLOBAL LOCK ACROSS WORKERS TO HANDLE PLAYER CONNECTIONS
27. Workers subscribe to /connect/<slot id> Players connect to /connect/<slot id> Slot = Hash(player_id) T P I O S C SLOTTED
28. E M Q W W W W M /connect/1 /connect/2 /connect/2 /connect/1
29. Problem CRASHES APPLICATION
30. SUPERVISION TREES
31. Master Worker Worker Worker App Supervisor Master
 Supervisor Worker
 Supervisor
32. CURRENT ARCHITECTURE IS TIGHTLY COUPLED TO QUIZ Problem
33. HOW TO SUPPORT STATES AND TRANSITIONS FOR DIFFERENT ENGAGEMENTS? Problem
34. FSM Event State Engagement State gen_statem in Erlang/OTP
35. WHERE WE ARE
36. M W W W W Director
37. WE HAVE IN PLACE BUILDING BLOCKS
38. HOW TO SCALE IN THE LONG RUN?
39. WE NEED TO THINK LONG TERM
40. INTERMISSION
41. THE ROAD AHEAD
42. Multi Node
43. M W W W W M W W W W GM
44. libcluster https://github.com/bitwalker/libcluster Form clusters of Erlang nodes Static or Dynamic Node Membership Custom Strategy to Deal with Nodes Joining/Leaving Notification when Nodes Join/Leave
45. Resilience
46. ETS Immortal https://github.com/danielberkompas/immortal Keep ETS alive using a heir process Keep ETS alive when owner dies Give ownership back to owner after it reboots
47. NODE A NODE B MASTER MA WA WB MB WA WA WA WB WB WB MB WB MA WB WB WB WA WA WA WA AND WORKER REPLICA
48. Masterless
49. M W W W W GM M W W W W GM M W W W W GM NODE A NODE C NODE B
50. CRDT Conflict-free Coordination-free Commutative or Convergent
51. CRDT Flags Maps Counters Sets
52. https://kubernetes.io/ SETUP/TEARDOWN ON-DEMAND FOR EVENTS INFRASTRUCTURE
53. RECAP
54. • https://elixirschool.com/en/ • https://ferd.ca/the-zen-of-erlang.html • http://blog.plataformatec.com.br/ • http://basho.com/tag/riak/ • https://www.theerlangelist.com/ Resources
55. Books Litte Elixir and OTP https://www.manning.com/books/the-little-elixir-and-otp-guidebook Learn you some Erlang http://learnyousomeerlang.com Designing for Scalability with Erlang/OTP http://shop.oreilly.com/product/0636920024149.do
56. DISTRIBUTED SYSTEMS ARE HARD ELIXIR MAKES THINGS EASIER
57. Thanks! @kirang89 @yudistrange https://bit.ly/2QVT7CB