Erlang Performance Lab

Erlang Performance Lab

For a few years we've been playing with an idea of visualising Erlang nodes and processes in a way that would help developers quickly grasp the behaviour of a cluster or a node. In 2016 Netflix released open source components visualising traffic in computer networks, which helped build a new generation of the tool.

In this talk we present examples of visualisations of inter-node traffic within an Erlang cluster, a supervision tree busy with after-crash restarts, a set of processes involved in message passing between them. Animations from the presentations were posted on YouTube channel https://www.youtube.com/channel/UCGkcbu799cC1rtMaQtAajpg

69639a461e92e590acdc1b554934bd8d?s=128

Michał Ślaski

March 23, 2017
Tweet

Transcript

  1. ERLANG LAB new coding perspective Michal Slaski @michalslaski

  2. ABOUT ME

  3. ABOUT ME • AGH'2005

  4. ABOUT ME • AGH'2005 • Tech Lead'2012
 @ Erlang Solutions

  5. ABOUT ME • AGH'2005 • Tech Lead'2012
 @ Erlang Solutions

    • MongooseIM dev
  6. ABOUT ME • AGH'2005 • Tech Lead'2012
 @ Erlang Solutions

    • MongooseIM dev • EUC'2013
  7. ABOUT ME • AGH'2005 • Tech Lead'2012
 @ Erlang Solutions

    • MongooseIM dev • EUC'2013
  8. ABOUT ME • AGH'2005 • Tech Lead'2012
 @ Erlang Solutions

    • MongooseIM dev • EUC'2013
  9. ABOUT ME • AGH'2005 • Tech Lead'2012
 @ Erlang Solutions

    • MongooseIM dev • EUC'2013
  10. ABOUT ME • AGH'2005 • Tech Lead'2012
 @ Erlang Solutions

    • MongooseIM dev • EUC'2013
  11. DISTRIBUTED SYSTEM Vizceral - intuition engineering @ Netflix

  12. DISTRIBUTED SYSTEM Vizceral - intuition engineering @ Netflix

  13. PROOF OF CONCEPT Elixir's getting started app

  14. PROOF OF CONCEPT Elixir's getting started app

  15. AGENDA

  16. AGENDA • Design Principles

  17. AGENDA • Design Principles • What is available

  18. AGENDA • Design Principles • What is available • How

    it works
  19. AGENDA • Design Principles • What is available • How

    it works • Ideas for future
  20. AGENDA • Design Principles • What is available • How

    it works • Ideas for future • Questions
  21. DESIGN PRINCIPLES

  22. DESIGN PRINCIPLES • Intuitive navigation and visualisations

  23. DESIGN PRINCIPLES • Intuitive navigation and visualisations • Simplify complex

    sets of trace events
  24. DESIGN PRINCIPLES • Intuitive navigation and visualisations • Simplify complex

    sets of trace events • Don't worry about production systems
  25. DESIGN PRINCIPLES • Intuitive navigation and visualisations • Simplify complex

    sets of trace events • Don't worry about production systems • Leverage WebGL as much as possible
  26. WHAT IS AVAILABLE • command line arguments

  27. WHAT IS AVAILABLE • plugins providing views

  28. WHAT IS AVAILABLE

  29. WHAT IS AVAILABLE

  30. WHAT IS AVAILABLE

  31. WHAT IS AVAILABLE • Dashboard with 7 metrics and 3

    charts
 - throughput (msgs and bytes)
 - number of processes
 - spawns
 - EXITs and abnormal EXITs
 - Memory
  32. Dashboard - mnesia executing transactions

  33. Dashboard - mnesia executing transactions

  34. WHAT IS AVAILABLE

  35. WHAT IS AVAILABLE • Supervision tree
 - hide / show

    applications
 - zoom in
 - zoom out
 - process_info(Pid)
  36. Supervision tree - mnesia crashing and restarting

  37. Supervision tree - mnesia crashing and restarting

  38. WHAT IS AVAILABLE

  39. WHAT IS AVAILABLE • Cluster traffic
 - number of Erlang

    distributed messages
 - ratio between ingress and egress traffic
 - distribution of load across nodes (RPS %)
  40. Cluster traffic - mnesia executing transaction

  41. Cluster traffic - mnesia executing transaction

  42. Cluster traffic - mnesia executing sync_dirty

  43. Cluster traffic - mnesia executing sync_dirty

  44. Cluster traffic - mnesia executing dirty_write

  45. Cluster traffic - mnesia executing dirty_write

  46. WHAT IS AVAILABLE

  47. WHAT IS AVAILABLE • Message passing
 - ingress traffic to

    TCP ports
 - messages between processes
  48. Message passing - mnesia executing transaction

  49. Message passing - mnesia executing transaction

  50. HOW IT WORKS

  51. HOW IT WORKS • spawn remote fun()
 
 


  52. HOW IT WORKS • spawn remote fun()
 
 


  53. HOW IT WORKS • spawn remote fun()
 
 
 •

    turn on tracer for all processes
  54. HOW IT WORKS • spawn remote fun()
 
 
 •

    turn on tracer for all processes
  55. IDEAS FOR FUTURE • Continuous Load Testing

  56. CONTINUOUS INTEGRATION

  57. CONTINUOUS INTEGRATION • Travis CI - MongooseIM on github.com

  58. CONTINUOUS INTEGRATION • Travis CI - MongooseIM on github.com

  59. LOAD TESTING A Murder Of Crows

  60. LOAD TESTING http://www.danhudson.ca/winter-crows/ A Murder Of Crows

  61. CONTINUOUS LOAD TESTS

  62. CONTINUOUS LOAD TESTS • Concept similar to Travis CI

  63. CONTINUOUS LOAD TESTS • Concept similar to Travis CI •

    Implemented Tide + amoc
  64. CONTINUOUS LOAD TESTS • Concept similar to Travis CI •

    Implemented Tide + amoc • Based on docker-swarm for environment setup
  65. TIDE

  66. TIDE

  67. TIDE

  68. IDEAS FOR FUTURE • Continuous Load Testing • Continuous Chaos

  69. CHAOS MONKEY

  70. CHAOS MONKEY • Netflix runs CM on live systems

  71. CHAOS MONKEY • Netflix runs CM on live systems •

    It helped increase service availability because engineers embrace fallback scenarios
  72. CHAOS MONKEY • Netflix runs CM on live systems •

    It helped increase service availability because engineers embrace fallback scenarios • Erlang Lab could help observe effects of CM
  73. IDEAS FOR FUTURE • Continuous Load Testing • Continuous Chaos

    • BEAM Modelling Language (BML)
  74. BEAM MODELLING LANG courtesy of Torben Hoffmann

  75. BEAM MODELLING LANG • P1 monitors P2 courtesy of Torben

    Hoffmann
  76. BEAM MODELLING LANG • P1 monitors P2 • P1 and

    P2 are linked courtesy of Torben Hoffmann
  77. BEAM MODELLING LANG • P1 monitors P2 • P1 and

    P2 are linked • P1 spawns P2 courtesy of Torben Hoffmann
  78. BEAM MODELLING LANG • P1 monitors P2 • P1 and

    P2 are linked • P1 spawns P2 • P1 sends M to P2 courtesy of Torben Hoffmann
  79. IDEAS FOR FUTURE • Continuous Load Testing • Continuous Chaos

    • BEAM Modelling Language (BML) • Timeline tracking
  80. OPEN SOURCE COMMUNITY • React + Elm + UX
 •

    Supervision Tree
 • You can join too!
  81. ERLANG LAB

  82. ERLANG LAB • www.erlang.pl • github.com/erlanglab • twitter @erlanglab

  83. QUESTIONS ? Michal Slaski @michalslaski