The Road to a Masterless Multi-node Distributed System in Elixir

by Kiran

Slide 1

Slide 1 text

The Road to a in Elixir Masterless Multi-node Distributed System

Slide 2

Slide 2 text

BUILDING AND SERVICES SYSTEMS

Slide 3

Slide 3 text

•WHY THIS JOURNEY? •THE START •WHERE WE ARE •THE ROAD AHEAD AGENDA

Slide 4

Slide 4 text

WHY THIS JOURNEY?

Slide 5

Slide 5 text

PLATFORM TO DISCOVER AND BUY THE BEST EXPERIENCES IN EVENTS, TRAVEL AND FOOD IN CITIES ACROSS INDIA

Slide 6

Slide 6 text

FOUNDED IN 2014 10K+ EVENTS, 3M+ TICKETS IN 2018 ACQUIRED BY PAYTM IN 2017

Slide 7

Slide 7 text

MAKE INSIDER A PLATFORM FOR HOSTING AND PARTICIPATING IN DIGITAL EVENTS

Slide 8

Slide 8 text

LIVE QUIZ

Slide 9

Slide 9 text

A PLATFORM THAT ENABLES INTERACTIONS ON A LIVE VIDEO STREAM interaction by Aman from the Noun Project

Slide 10

Slide 10 text

FLEXIBLE AND SCALABLE FUEL INTERACTIONS flexible by Anup from the Noun Project PLATFORM TO

Slide 11

Slide 11 text

Elixir Functional Programming Immutable Data Structures Concurrency Fault Tolerance Battle Tested VM Erlang Interoperability

Slide 12

Slide 12 text

THE START

Slide 13

Slide 13 text

M W W W W NODE

Slide 14

Slide 14 text

MASTER CANNOT HANDLE Problem PLAYERS

Slide 15

Slide 15 text

M W W W W NODE SM SM

Slide 16

Slide 16 text

UNNECESSARY CHANGE CALLS CASTS

Slide 17

Slide 17 text

PLAYER STATE TRANSFER GENSERVER ETS

Slide 18

Slide 18 text

M W W W W NODE C NODE B NODE A

Slide 19

Slide 19 text

SENDING MESSAGE TO A PID OVER THE NETWORK COPIES THE MESSAGE OVER THE NETWORK Problem

Slide 20

Slide 20 text

Manifold github.com/discordapp/manifold Batching of messages passed between nodes

Slide 21

Slide 21 text

MANIFOLD WORKS FOR FAN-OUTs WHAT ABOUT FAN-INs? Problem

Slide 22

Slide 22 text

SO MUCH CHATTER MESSAGE PASSING Problem BECAME EXPENSIVE

Slide 23

Slide 23 text

MQTT Publisher/Subscriber Messaging Protocol Built on top of TCP/IP Lightweight and Bandwidth Efficient Quality of Service Data Agnostic Message Queueing Telemetry Transport

Slide 24

Slide 24 text

Distributed, Scalable and Highly Extensible MQTT message broker Written in Erlang/OTP http://emqtt.io

Slide 25

Slide 25 text

E M Q W W W W M /connect /connect /connect /connect

Slide 26

Slide 26 text

Problem GLOBAL LOCK ACROSS WORKERS TO HANDLE PLAYER CONNECTIONS

Slide 27

Slide 27 text

Workers subscribe to /connect/ Players connect to /connect/ Slot = Hash(player_id) T P I O S C SLOTTED

Slide 28

Slide 28 text

E M Q W W W W M /connect/1 /connect/2 /connect/2 /connect/1

Slide 29

Slide 29 text

Problem CRASHES APPLICATION

Slide 30

Slide 30 text

SUPERVISION TREES

Slide 31

Slide 31 text

Master Worker Worker Worker App Supervisor Master  Supervisor Worker  Supervisor

Slide 32

Slide 32 text

CURRENT ARCHITECTURE IS TIGHTLY COUPLED TO QUIZ Problem

Slide 33

Slide 33 text

HOW TO SUPPORT STATES AND TRANSITIONS FOR DIFFERENT ENGAGEMENTS? Problem

Slide 34

Slide 34 text

FSM Event State Engagement State gen_statem in Erlang/OTP

Slide 35

Slide 35 text

WHERE WE ARE

Slide 36

Slide 36 text

M W W W W Director

Slide 37

Slide 37 text

WE HAVE IN PLACE BUILDING BLOCKS

Slide 38

Slide 38 text

HOW TO SCALE IN THE LONG RUN?

Slide 39

Slide 39 text

WE NEED TO THINK LONG TERM

Slide 40

Slide 40 text

INTERMISSION

Slide 41

Slide 41 text

THE ROAD AHEAD

Slide 42

Slide 42 text

Multi Node

Slide 43

Slide 43 text

M W W W W M W W W W GM

Slide 44

Slide 44 text

libcluster https://github.com/bitwalker/libcluster Form clusters of Erlang nodes Static or Dynamic Node Membership Custom Strategy to Deal with Nodes Joining/Leaving Notification when Nodes Join/Leave

Slide 45

Slide 45 text

Resilience

Slide 46

Slide 46 text

ETS Immortal https://github.com/danielberkompas/immortal Keep ETS alive using a heir process Keep ETS alive when owner dies Give ownership back to owner after it reboots

Slide 47

Slide 47 text

NODE A NODE B MASTER MA WA WB MB WA WA WA WB WB WB MB WB MA WB WB WB WA WA WA WA AND WORKER REPLICA

Slide 48

Slide 48 text

Masterless

Slide 49

Slide 49 text

M W W W W GM M W W W W GM M W W W W GM NODE A NODE C NODE B

Slide 50

Slide 50 text

CRDT Conflict-free Coordination-free Commutative or Convergent

Slide 51

Slide 51 text

CRDT Flags Maps Counters Sets

Slide 52

Slide 52 text

https://kubernetes.io/ SETUP/TEARDOWN ON-DEMAND FOR EVENTS INFRASTRUCTURE

Slide 53

Slide 53 text

RECAP

Slide 54

Slide 54 text

• https://elixirschool.com/en/ • https://ferd.ca/the-zen-of-erlang.html • http://blog.plataformatec.com.br/ • http://basho.com/tag/riak/ • https://www.theerlangelist.com/ Resources

Slide 55

Slide 55 text

Books Litte Elixir and OTP https://www.manning.com/books/the-little-elixir-and-otp-guidebook Learn you some Erlang http://learnyousomeerlang.com Designing for Scalability with Erlang/OTP http://shop.oreilly.com/product/0636920024149.do