Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Erlang Performance Lab

Erlang Performance Lab

For a few years we've been playing with an idea of visualising Erlang nodes and processes in a way that would help developers quickly grasp the behaviour of a cluster or a node. In 2016 Netflix released open source components visualising traffic in computer networks, which helped build a new generation of the tool.

In this talk we present examples of visualisations of inter-node traffic within an Erlang cluster, a supervision tree busy with after-crash restarts, a set of processes involved in message passing between them. Animations from the presentations were posted on YouTube channel https://www.youtube.com/channel/UCGkcbu799cC1rtMaQtAajpg

Michał Ślaski

March 23, 2017
Tweet

More Decks by Michał Ślaski

Other Decks in Programming

Transcript

  1. AGENDA • Design Principles • What is available • How

    it works • Ideas for future • Questions
  2. DESIGN PRINCIPLES • Intuitive navigation and visualisations • Simplify complex

    sets of trace events • Don't worry about production systems
  3. DESIGN PRINCIPLES • Intuitive navigation and visualisations • Simplify complex

    sets of trace events • Don't worry about production systems • Leverage WebGL as much as possible
  4. WHAT IS AVAILABLE • Dashboard with 7 metrics and 3

    charts
 - throughput (msgs and bytes)
 - number of processes
 - spawns
 - EXITs and abnormal EXITs
 - Memory
  5. WHAT IS AVAILABLE • Supervision tree
 - hide / show

    applications
 - zoom in
 - zoom out
 - process_info(Pid)
  6. WHAT IS AVAILABLE • Cluster traffic
 - number of Erlang

    distributed messages
 - ratio between ingress and egress traffic
 - distribution of load across nodes (RPS %)
  7. WHAT IS AVAILABLE • Message passing
 - ingress traffic to

    TCP ports
 - messages between processes
  8. HOW IT WORKS • spawn remote fun()
 
 
 •

    turn on tracer for all processes
  9. HOW IT WORKS • spawn remote fun()
 
 
 •

    turn on tracer for all processes
  10. CONTINUOUS LOAD TESTS • Concept similar to Travis CI •

    Implemented Tide + amoc • Based on docker-swarm for environment setup
  11. CHAOS MONKEY • Netflix runs CM on live systems •

    It helped increase service availability because engineers embrace fallback scenarios
  12. CHAOS MONKEY • Netflix runs CM on live systems •

    It helped increase service availability because engineers embrace fallback scenarios • Erlang Lab could help observe effects of CM
  13. BEAM MODELLING LANG • P1 monitors P2 • P1 and

    P2 are linked courtesy of Torben Hoffmann
  14. BEAM MODELLING LANG • P1 monitors P2 • P1 and

    P2 are linked • P1 spawns P2 courtesy of Torben Hoffmann
  15. BEAM MODELLING LANG • P1 monitors P2 • P1 and

    P2 are linked • P1 spawns P2 • P1 sends M to P2 courtesy of Torben Hoffmann
  16. IDEAS FOR FUTURE • Continuous Load Testing • Continuous Chaos

    • BEAM Modelling Language (BML) • Timeline tracking
  17. OPEN SOURCE COMMUNITY • React + Elm + UX
 •

    Supervision Tree
 • You can join too!