Exploring BEAM-based systems with erlang.pl

Processes running on the BEAM virtual machine share data through message passing. If certain processes receive more messages than they can handle, their inbox queue starts growing. Over time it can lead to bottlenecks. Similarly distributed systems share data through network packets sent between nodes and in some cases it can lead to network congestions. Gaining visibility into message passing and network traffic is of great importance as it helps to understand the behaviour of the entire system.

During this talk we will explore the behaviour of two systems under load - one based on Erlang (MongooseIM) and one based on Elixir (Phoenix Channels). We will demonstrate the erlang.pl tool and observe if its graphical representation of processes and clustering can help us learn something about the characteristics of the two systems.

Videos used in the presentation:
Phoenix Channels cluster view - https://www.youtube.com/watch?v=ybnQ0rre1Jc
Phoenix Channels node view - https://www.youtube.com/watch?v=ZD5WwGsBA7Q
MongooseIM cluster view - https://www.youtube.com/watch?v=BIbLOVNKZYw

Transcript

1. ERLANG.PL new coding perspective Erlang User Group Krakow, 28 June 2017
2. DO YOU KNOW BEAM ?
3. WHY BEAM
4. WHY BEAM • BEAM benefits relevant today:
5. WHY BEAM • BEAM benefits relevant today: • concise code
6. WHY BEAM • BEAM benefits relevant today: • concise code • reliable system
7. WHY BEAM • BEAM benefits relevant today: • concise code • reliable system • equal latency for all users
8. WHY BEAM • BEAM benefits relevant today: • concise code • reliable system • equal latency for all users • utilisation of multi core CPUs
9. WHY BEAM • BEAM benefits relevant today: • concise code • reliable system • equal latency for all users • utilisation of multi core CPUs • traceability
10. WHAT IS ERLANG.PL
11. WHAT IS ERLANG.PL • Tool for the BEAM platform (a.k.a. Erlang VM)
12. WHAT IS ERLANG.PL • Tool for the BEAM platform (a.k.a. Erlang VM) • Helps with system exploration
13. WHAT IS ERLANG.PL • Tool for the BEAM platform (a.k.a. Erlang VM) • Helps with system exploration • Exploits benefits of traceability
14. WHAT IS ERLANG.PL • Tool for the BEAM platform (a.k.a. Erlang VM) • Helps with system exploration • Exploits benefits of traceability • Aims at facilitating developer's work
15. HISTORY
16. HISTORY • Proof of concept in Jun 2013
17. HISTORY • Proof of concept in Jun 2013 • Open source since Feb 2017
18. HISTORY • Proof of concept in Jun 2013 • Open source since Feb 2017 • Presented in Mar 2017 at EEF SF
19. HISTORY • Proof of concept in Jun 2013 • Open source since Feb 2017 • Presented in Mar 2017 at EEF SF • Accepted for Google Summer of Code'2017
20. WORKFLOW - STEP 1
21. WORKFLOW - STEP 1 • observe dashboard
22. WORKFLOW - STEP 1 • observe dashboard • look for trends
23. WORKFLOW - STEP 1 • observe dashboard • look for trends • observe that
 message traffic
 has increased
24. WORKFLOW - STEP 1 • observe dashboard • look for trends • observe that
 message traffic
 has increased • what is the root cause?
25. Dashboard - mnesia executing transactions
26. Dashboard - mnesia executing transactions
27. WORKFLOW - STEP 2
28. WORKFLOW - STEP 2 if the root cause is external,
29. WORKFLOW - STEP 2 if the root cause is external, we observe increased
 traffic from external nodes
30. CLUSTER TRAFFIC Cluster traffic - generate load on Phoenix channels https://github.com/arkgil/phx_load_gen
31. CLUSTER TRAFFIC Cluster traffic - generate load on Phoenix channels https://github.com/arkgil/phx_load_gen
32. CLUSTER TRAFFIC Cluster traffic - generate load on MongooseIM XMPP https://github.com/esl/amoc
33. CLUSTER TRAFFIC Cluster traffic - generate load on MongooseIM XMPP https://github.com/esl/amoc
34. WORKFLOW - STEP 3
35. WORKFLOW - STEP 3 if the root cause is internal,
36. WORKFLOW - STEP 3 if the root cause is internal, we observe increased traffic
 between internal processes
37. Message passing - WebSocket and Phoenix Channel
38. Message passing - WebSocket and Phoenix Channel
39. WORKFLOW - STEP 4
40. WORKFLOW - STEP 4 • view the supervision tree
41. WORKFLOW - STEP 4 • view the supervision tree • find the process generating
 a lot of internal traffic
42. WORKFLOW - STEP 4 • view the supervision tree • find the process generating
 a lot of internal traffic • get to know its process_info/1
43. WORKFLOW - STEP 5
44. WORKFLOW - STEP 5 • analyse the process' mailbox
45. WORKFLOW - STEP 5 • analyse the process' mailbox • observe the ordering of messages
46. WORKFLOW - STEP 5 • analyse the process' mailbox • observe the ordering of messages inspect how processing a message changed the state
47. WORKFLOW - STEP 6
48. WORKFLOW - STEP 6 • inspect a flame graph
49. WORKFLOW - STEP 6 • inspect a flame graph • what keeps it busy?
50. WORKFLOW - STEP 6 • inspect a flame graph • what keeps it busy? • where most of the time is spent?
51. DESIGN PRINCIPLES
52. DESIGN PRINCIPLES • Intuitive navigation and visualisations
53. DESIGN PRINCIPLES • Intuitive navigation and visualisations • Simplify complex sets of trace events
54. DESIGN PRINCIPLES • Intuitive navigation and visualisations • Simplify complex sets of trace events • Don't worry about production systems
55. DESIGN PRINCIPLES • Intuitive navigation and visualisations • Simplify complex sets of trace events • Don't worry about production systems • Leverage WebGL as much as possible
56. WHAT IS AVAILABLE
57. WHAT IS AVAILABLE • Dashboard, cluster, message passing and supervision tree views already implemented
58. WHAT IS AVAILABLE • Dashboard, cluster, message passing and supervision tree views already implemented • Timeline tracking is work in progress
59. HOW IT WORKS
60. HOW IT WORKS • spawn remote fun()
 
 

61. HOW IT WORKS • spawn remote fun()
 
 

62. HOW IT WORKS • spawn remote fun()
 
 
 • turn on tracer for all processes
63. HOW IT WORKS • spawn remote fun()
 
 
 • turn on tracer for all processes
64. ERLANG.PL
65. ERLANG.PL • www.erlang.pl • github.com/erlanglab • twitter @erlanglab
66. QUESTIONS ? info@erlang.pl