$30 off During Our Annual Pro Sale. View Details »

The Road to a Masterless Multi-node Distributed System in Elixir

Kiran
December 13, 2018

The Road to a Masterless Multi-node Distributed System in Elixir

In this talk, we will cover our journey towards building a masterless multi-node distributed system in Elixir. We’ll talk about the challenges faced, decisions made and how they shaped/evolved the architecture.

Presented by Udit Kumar(https://speakerdeck.com/yudistrange) and Myself.

Kiran

December 13, 2018
Tweet

More Decks by Kiran

Other Decks in Programming

Transcript

  1. The Road to a
    in Elixir
    Masterless
    Multi-node
    Distributed System

    View Slide

  2. BUILDING
    AND
    SERVICES
    SYSTEMS

    View Slide

  3. •WHY THIS JOURNEY?
    •THE START
    •WHERE WE ARE
    •THE ROAD AHEAD
    AGENDA

    View Slide

  4. WHY THIS
    JOURNEY?

    View Slide

  5. PLATFORM TO
    DISCOVER AND BUY THE BEST EXPERIENCES IN
    EVENTS, TRAVEL AND FOOD IN CITIES ACROSS INDIA

    View Slide

  6. FOUNDED IN 2014
    10K+ EVENTS, 3M+ TICKETS IN 2018
    ACQUIRED BY PAYTM IN 2017

    View Slide

  7. MAKE INSIDER A PLATFORM FOR
    HOSTING AND PARTICIPATING IN
    DIGITAL EVENTS

    View Slide

  8. LIVE
    QUIZ

    View Slide

  9. A PLATFORM THAT ENABLES
    INTERACTIONS ON A
    LIVE VIDEO STREAM
    interaction by Aman from the Noun Project

    View Slide

  10. FLEXIBLE AND SCALABLE
    FUEL INTERACTIONS
    flexible by Anup from the Noun Project
    PLATFORM TO

    View Slide

  11. Elixir
    Functional Programming
    Immutable Data Structures
    Concurrency
    Fault Tolerance
    Battle Tested VM
    Erlang Interoperability

    View Slide

  12. THE START

    View Slide

  13. M
    W
    W
    W
    W
    NODE

    View Slide

  14. MASTER CANNOT HANDLE Problem
    PLAYERS

    View Slide

  15. M
    W
    W
    W
    W
    NODE
    SM
    SM

    View Slide

  16. UNNECESSARY
    CHANGE
    CALLS
    CASTS

    View Slide

  17. PLAYER STATE
    TRANSFER
    GENSERVER
    ETS

    View Slide

  18. M
    W
    W
    W
    W
    NODE C
    NODE B
    NODE A

    View Slide

  19. SENDING MESSAGE TO A PID
    OVER THE NETWORK
    COPIES THE MESSAGE
    OVER THE NETWORK
    Problem

    View Slide

  20. Manifold
    github.com/discordapp/manifold
    Batching of messages passed between nodes

    View Slide

  21. MANIFOLD WORKS FOR
    FAN-OUTs
    WHAT ABOUT
    FAN-INs?
    Problem

    View Slide

  22. SO MUCH CHATTER
    MESSAGE PASSING
    Problem
    BECAME EXPENSIVE

    View Slide

  23. MQTT
    Publisher/Subscriber Messaging Protocol
    Built on top of TCP/IP
    Lightweight and Bandwidth Efficient
    Quality of Service
    Data Agnostic
    Message Queueing Telemetry Transport

    View Slide

  24. Distributed, Scalable and Highly
    Extensible MQTT message broker
    Written in Erlang/OTP
    http://emqtt.io

    View Slide

  25. E
    M
    Q
    W
    W
    W
    W
    M
    /connect
    /connect
    /connect
    /connect

    View Slide

  26. Problem
    GLOBAL LOCK ACROSS
    WORKERS TO HANDLE
    PLAYER CONNECTIONS

    View Slide

  27. Workers subscribe to /connect/
    Players connect to /connect/
    Slot = Hash(player_id)
    T P I
    O S
    C
    SLOTTED

    View Slide

  28. E
    M
    Q
    W
    W
    W
    W
    M
    /connect/1
    /connect/2
    /connect/2
    /connect/1

    View Slide

  29. Problem
    CRASHES
    APPLICATION

    View Slide

  30. SUPERVISION
    TREES

    View Slide

  31. Master Worker Worker Worker
    App
    Supervisor
    Master

    Supervisor
    Worker

    Supervisor

    View Slide

  32. CURRENT ARCHITECTURE
    IS TIGHTLY COUPLED TO
    QUIZ
    Problem

    View Slide

  33. HOW TO SUPPORT STATES
    AND TRANSITIONS FOR
    DIFFERENT ENGAGEMENTS?
    Problem

    View Slide

  34. FSM Event State
    Engagement State
    gen_statem in Erlang/OTP

    View Slide

  35. WHERE WE ARE

    View Slide

  36. M
    W W
    W
    W
    Director

    View Slide

  37. WE HAVE
    IN PLACE
    BUILDING BLOCKS

    View Slide

  38. HOW TO SCALE
    IN THE
    LONG RUN?

    View Slide

  39. WE NEED TO THINK
    LONG TERM

    View Slide

  40. INTERMISSION

    View Slide

  41. THE ROAD AHEAD

    View Slide

  42. Multi Node

    View Slide

  43. M
    W W
    W
    W
    M
    W W
    W
    W
    GM

    View Slide

  44. libcluster
    https://github.com/bitwalker/libcluster
    Form clusters of Erlang nodes
    Static or Dynamic Node Membership
    Custom Strategy to Deal with Nodes Joining/Leaving
    Notification when Nodes Join/Leave

    View Slide

  45. Resilience

    View Slide

  46. ETS Immortal
    https://github.com/danielberkompas/immortal
    Keep ETS alive using a heir process
    Keep ETS alive when owner dies
    Give ownership back to owner after it reboots

    View Slide

  47. NODE A
    NODE B
    MASTER MA
    WA
    WB
    MB
    WA WA WA
    WB WB WB
    MB
    WB
    MA
    WB WB WB
    WA WA WA WA
    AND
    WORKER
    REPLICA

    View Slide

  48. Masterless

    View Slide

  49. M
    W W
    W
    W
    GM
    M
    W W
    W
    W
    GM
    M
    W W
    W
    W
    GM
    NODE A
    NODE C
    NODE B

    View Slide

  50. CRDT
    Conflict-free
    Coordination-free
    Commutative or Convergent

    View Slide

  51. CRDT
    Flags
    Maps
    Counters
    Sets

    View Slide

  52. https://kubernetes.io/
    SETUP/TEARDOWN
    ON-DEMAND
    FOR EVENTS
    INFRASTRUCTURE

    View Slide

  53. RECAP

    View Slide

  54. • https://elixirschool.com/en/
    • https://ferd.ca/the-zen-of-erlang.html
    • http://blog.plataformatec.com.br/
    • http://basho.com/tag/riak/
    • https://www.theerlangelist.com/
    Resources

    View Slide

  55. Books
    Litte Elixir and OTP
    https://www.manning.com/books/the-little-elixir-and-otp-guidebook
    Learn you some Erlang
    http://learnyousomeerlang.com
    Designing for Scalability with Erlang/OTP
    http://shop.oreilly.com/product/0636920024149.do

    View Slide

  56. DISTRIBUTED SYSTEMS ARE HARD
    ELIXIR MAKES THINGS EASIER

    View Slide

  57. Thanks!
    @kirang89
    @yudistrange
    https://bit.ly/2QVT7CB

    View Slide