Erlang Performance Lab

For a few years we've been playing with an idea of visualising Erlang nodes and processes in a way that would help developers quickly grasp the behaviour of a cluster or a node. In 2016 Netflix released open source components visualising traffic in computer networks, which helped build a new generation of the tool.

In this talk we present examples of visualisations of inter-node traffic within an Erlang cluster, a supervision tree busy with after-crash restarts, a set of processes involved in message passing between them. Animations from the presentations were posted on YouTube channel https://www.youtube.com/channel/UCGkcbu799cC1rtMaQtAajpg

Transcript

1. ERLANG LAB new coding perspective Michal Slaski @michalslaski
2. ABOUT ME
3. ABOUT ME • AGH'2005
4. ABOUT ME • AGH'2005 • Tech Lead'2012
 @ Erlang Solutions
5. ABOUT ME • AGH'2005 • Tech Lead'2012
 @ Erlang Solutions • MongooseIM dev
6. ABOUT ME • AGH'2005 • Tech Lead'2012
 @ Erlang Solutions • MongooseIM dev • EUC'2013
7. ABOUT ME • AGH'2005 • Tech Lead'2012
 @ Erlang Solutions • MongooseIM dev • EUC'2013
8. ABOUT ME • AGH'2005 • Tech Lead'2012
 @ Erlang Solutions • MongooseIM dev • EUC'2013
9. ABOUT ME • AGH'2005 • Tech Lead'2012
 @ Erlang Solutions • MongooseIM dev • EUC'2013
10. ABOUT ME • AGH'2005 • Tech Lead'2012
 @ Erlang Solutions • MongooseIM dev • EUC'2013
11. DISTRIBUTED SYSTEM Vizceral - intuition engineering @ Netflix
12. DISTRIBUTED SYSTEM Vizceral - intuition engineering @ Netflix
13. PROOF OF CONCEPT Elixir's getting started app
14. PROOF OF CONCEPT Elixir's getting started app
15. AGENDA
16. AGENDA • Design Principles
17. AGENDA • Design Principles • What is available
18. AGENDA • Design Principles • What is available • How it works
19. AGENDA • Design Principles • What is available • How it works • Ideas for future
20. AGENDA • Design Principles • What is available • How it works • Ideas for future • Questions
21. DESIGN PRINCIPLES
22. DESIGN PRINCIPLES • Intuitive navigation and visualisations
23. DESIGN PRINCIPLES • Intuitive navigation and visualisations • Simplify complex sets of trace events
24. DESIGN PRINCIPLES • Intuitive navigation and visualisations • Simplify complex sets of trace events • Don't worry about production systems
25. DESIGN PRINCIPLES • Intuitive navigation and visualisations • Simplify complex sets of trace events • Don't worry about production systems • Leverage WebGL as much as possible
26. WHAT IS AVAILABLE • command line arguments
27. WHAT IS AVAILABLE • plugins providing views
28. WHAT IS AVAILABLE
29. WHAT IS AVAILABLE
30. WHAT IS AVAILABLE
31. WHAT IS AVAILABLE • Dashboard with 7 metrics and 3 charts
 - throughput (msgs and bytes)
 - number of processes
 - spawns
 - EXITs and abnormal EXITs
 - Memory
32. Dashboard - mnesia executing transactions
33. Dashboard - mnesia executing transactions
34. WHAT IS AVAILABLE
35. WHAT IS AVAILABLE • Supervision tree
 - hide / show applications
 - zoom in
 - zoom out
 - process_info(Pid)
36. Supervision tree - mnesia crashing and restarting
37. Supervision tree - mnesia crashing and restarting
38. WHAT IS AVAILABLE
39. WHAT IS AVAILABLE • Cluster traffic
 - number of Erlang distributed messages
 - ratio between ingress and egress traffic
 - distribution of load across nodes (RPS %)
40. Cluster traffic - mnesia executing transaction
41. Cluster traffic - mnesia executing transaction
42. Cluster traffic - mnesia executing sync_dirty
43. Cluster traffic - mnesia executing sync_dirty
44. Cluster traffic - mnesia executing dirty_write
45. Cluster traffic - mnesia executing dirty_write
46. WHAT IS AVAILABLE
47. WHAT IS AVAILABLE • Message passing
 - ingress traffic to TCP ports
 - messages between processes
48. Message passing - mnesia executing transaction
49. Message passing - mnesia executing transaction
50. HOW IT WORKS
51. HOW IT WORKS • spawn remote fun()
 
 

52. HOW IT WORKS • spawn remote fun()
 
 

53. HOW IT WORKS • spawn remote fun()
 
 
 • turn on tracer for all processes
54. HOW IT WORKS • spawn remote fun()
 
 
 • turn on tracer for all processes
55. IDEAS FOR FUTURE • Continuous Load Testing
56. CONTINUOUS INTEGRATION
57. CONTINUOUS INTEGRATION • Travis CI - MongooseIM on github.com
58. CONTINUOUS INTEGRATION • Travis CI - MongooseIM on github.com
59. LOAD TESTING A Murder Of Crows
60. LOAD TESTING http://www.danhudson.ca/winter-crows/ A Murder Of Crows
61. CONTINUOUS LOAD TESTS
62. CONTINUOUS LOAD TESTS • Concept similar to Travis CI
63. CONTINUOUS LOAD TESTS • Concept similar to Travis CI • Implemented Tide + amoc
64. CONTINUOUS LOAD TESTS • Concept similar to Travis CI • Implemented Tide + amoc • Based on docker-swarm for environment setup
65. TIDE
66. TIDE
67. TIDE
68. IDEAS FOR FUTURE • Continuous Load Testing • Continuous Chaos
69. CHAOS MONKEY
70. CHAOS MONKEY • Netflix runs CM on live systems
71. CHAOS MONKEY • Netflix runs CM on live systems • It helped increase service availability because engineers embrace fallback scenarios
72. CHAOS MONKEY • Netflix runs CM on live systems • It helped increase service availability because engineers embrace fallback scenarios • Erlang Lab could help observe effects of CM
73. IDEAS FOR FUTURE • Continuous Load Testing • Continuous Chaos • BEAM Modelling Language (BML)
74. BEAM MODELLING LANG courtesy of Torben Hoffmann
75. BEAM MODELLING LANG • P1 monitors P2 courtesy of Torben Hoffmann
76. BEAM MODELLING LANG • P1 monitors P2 • P1 and P2 are linked courtesy of Torben Hoffmann
77. BEAM MODELLING LANG • P1 monitors P2 • P1 and P2 are linked • P1 spawns P2 courtesy of Torben Hoffmann
78. BEAM MODELLING LANG • P1 monitors P2 • P1 and P2 are linked • P1 spawns P2 • P1 sends M to P2 courtesy of Torben Hoffmann
79. IDEAS FOR FUTURE • Continuous Load Testing • Continuous Chaos • BEAM Modelling Language (BML) • Timeline tracking
80. OPEN SOURCE COMMUNITY • React + Elm + UX
 • Supervision Tree
 • You can join too!
81. ERLANG LAB
82. ERLANG LAB • www.erlang.pl • github.com/erlanglab • twitter @erlanglab
83. QUESTIONS ? Michal Slaski @michalslaski