Exploring BEAM-based systems with erlang.pl

Exploring BEAM-based systems with erlang.pl

Processes running on the BEAM virtual machine share data through message passing. If certain processes receive more messages than they can handle, their inbox queue starts growing. Over time it can lead to bottlenecks. Similarly distributed systems share data through network packets sent between nodes and in some cases it can lead to network congestions. Gaining visibility into message passing and network traffic is of great importance as it helps to understand the behaviour of the entire system.

During this talk we will explore the behaviour of two systems under load - one based on Erlang (MongooseIM) and one based on Elixir (Phoenix Channels). We will demonstrate the erlang.pl tool and observe if its graphical representation of processes and clustering can help us learn something about the characteristics of the two systems.

Videos used in the presentation:
Phoenix Channels cluster view - https://www.youtube.com/watch?v=ybnQ0rre1Jc
Phoenix Channels node view - https://www.youtube.com/watch?v=ZD5WwGsBA7Q
MongooseIM cluster view - https://www.youtube.com/watch?v=BIbLOVNKZYw

69639a461e92e590acdc1b554934bd8d?s=128

Michał Ślaski

June 28, 2017
Tweet

Transcript

  1. ERLANG.PL new coding perspective Erlang User Group Krakow, 28 June

    2017
  2. DO YOU KNOW BEAM ?

  3. WHY BEAM

  4. WHY BEAM • BEAM benefits relevant today:

  5. WHY BEAM • BEAM benefits relevant today: • concise code

  6. WHY BEAM • BEAM benefits relevant today: • concise code

    • reliable system
  7. WHY BEAM • BEAM benefits relevant today: • concise code

    • reliable system • equal latency for all users
  8. WHY BEAM • BEAM benefits relevant today: • concise code

    • reliable system • equal latency for all users • utilisation of multi core CPUs
  9. WHY BEAM • BEAM benefits relevant today: • concise code

    • reliable system • equal latency for all users • utilisation of multi core CPUs • traceability
  10. WHAT IS ERLANG.PL

  11. WHAT IS ERLANG.PL • Tool for the BEAM platform (a.k.a.

    Erlang VM)
  12. WHAT IS ERLANG.PL • Tool for the BEAM platform (a.k.a.

    Erlang VM) • Helps with system exploration
  13. WHAT IS ERLANG.PL • Tool for the BEAM platform (a.k.a.

    Erlang VM) • Helps with system exploration • Exploits benefits of traceability
  14. WHAT IS ERLANG.PL • Tool for the BEAM platform (a.k.a.

    Erlang VM) • Helps with system exploration • Exploits benefits of traceability • Aims at facilitating developer's work
  15. HISTORY

  16. HISTORY • Proof of concept in Jun 2013

  17. HISTORY • Proof of concept in Jun 2013 • Open

    source since Feb 2017
  18. HISTORY • Proof of concept in Jun 2013 • Open

    source since Feb 2017 • Presented in Mar 2017 at EEF SF
  19. HISTORY • Proof of concept in Jun 2013 • Open

    source since Feb 2017 • Presented in Mar 2017 at EEF SF • Accepted for Google Summer of Code'2017
  20. WORKFLOW - STEP 1

  21. WORKFLOW - STEP 1 • observe dashboard

  22. WORKFLOW - STEP 1 • observe dashboard • look for

    trends
  23. WORKFLOW - STEP 1 • observe dashboard • look for

    trends • observe that
 message traffic
 has increased
  24. WORKFLOW - STEP 1 • observe dashboard • look for

    trends • observe that
 message traffic
 has increased • what is the root cause?
  25. Dashboard - mnesia executing transactions

  26. Dashboard - mnesia executing transactions

  27. WORKFLOW - STEP 2

  28. WORKFLOW - STEP 2 if the root cause is external,

  29. WORKFLOW - STEP 2 if the root cause is external,

    we observe increased
 traffic from external nodes
  30. CLUSTER TRAFFIC Cluster traffic - generate load on Phoenix channels

    https://github.com/arkgil/phx_load_gen
  31. CLUSTER TRAFFIC Cluster traffic - generate load on Phoenix channels

    https://github.com/arkgil/phx_load_gen
  32. CLUSTER TRAFFIC Cluster traffic - generate load on MongooseIM XMPP

    https://github.com/esl/amoc
  33. CLUSTER TRAFFIC Cluster traffic - generate load on MongooseIM XMPP

    https://github.com/esl/amoc
  34. WORKFLOW - STEP 3

  35. WORKFLOW - STEP 3 if the root cause is internal,

  36. WORKFLOW - STEP 3 if the root cause is internal,

    we observe increased traffic
 between internal processes
  37. Message passing - WebSocket and Phoenix Channel

  38. Message passing - WebSocket and Phoenix Channel

  39. WORKFLOW - STEP 4

  40. WORKFLOW - STEP 4 • view the supervision tree

  41. WORKFLOW - STEP 4 • view the supervision tree •

    find the process generating
 a lot of internal traffic
  42. WORKFLOW - STEP 4 • view the supervision tree •

    find the process generating
 a lot of internal traffic • get to know its process_info/1
  43. WORKFLOW - STEP 5

  44. WORKFLOW - STEP 5 • analyse the process' mailbox

  45. WORKFLOW - STEP 5 • analyse the process' mailbox •

    observe the ordering of messages
  46. WORKFLOW - STEP 5 • analyse the process' mailbox •

    observe the ordering of messages inspect how processing a message changed the state
  47. WORKFLOW - STEP 6

  48. WORKFLOW - STEP 6 • inspect a flame graph

  49. WORKFLOW - STEP 6 • inspect a flame graph •

    what keeps it busy?
  50. WORKFLOW - STEP 6 • inspect a flame graph •

    what keeps it busy? • where most of the time is spent?
  51. DESIGN PRINCIPLES

  52. DESIGN PRINCIPLES • Intuitive navigation and visualisations

  53. DESIGN PRINCIPLES • Intuitive navigation and visualisations • Simplify complex

    sets of trace events
  54. DESIGN PRINCIPLES • Intuitive navigation and visualisations • Simplify complex

    sets of trace events • Don't worry about production systems
  55. DESIGN PRINCIPLES • Intuitive navigation and visualisations • Simplify complex

    sets of trace events • Don't worry about production systems • Leverage WebGL as much as possible
  56. WHAT IS AVAILABLE

  57. WHAT IS AVAILABLE • Dashboard, cluster, message passing and supervision

    tree views already implemented
  58. WHAT IS AVAILABLE • Dashboard, cluster, message passing and supervision

    tree views already implemented • Timeline tracking is work in progress
  59. HOW IT WORKS

  60. HOW IT WORKS • spawn remote fun()
 
 


  61. HOW IT WORKS • spawn remote fun()
 
 


  62. HOW IT WORKS • spawn remote fun()
 
 
 •

    turn on tracer for all processes
  63. HOW IT WORKS • spawn remote fun()
 
 
 •

    turn on tracer for all processes
  64. ERLANG.PL

  65. ERLANG.PL • www.erlang.pl • github.com/erlanglab • twitter @erlanglab

  66. QUESTIONS ? info@erlang.pl