Load-testing 101 & Anatomy Of A Load-Generator in Node.js

Slides for a talk given at London Node.js User Group on 23/09/2015

NOTE: Minigun has since been renamed to Artillery :) - https://artillery.io

Transcript

4. https://yld.io
6. “Load-­‐‑testing  101, Anatomy  of  a  load-­‐‑generator  in  Node.js, and The  joys  of  load-­‐‑testing,  or When  you  have  a  great  hammer...” A  Talk  In  3  Acts Hassy  Veldstra  <h@veldstra.org> @hveldstra LNUG,  September  2015
7. Episode  1:  Load-­‐Testing  101
8. Act  1:  Load-­‐Testing  101 “Load-­‐testing  is  the  process  of  putting  demand  on  a  software  system   and  measuring  its  response.”  -­‐ Wikipedia
9. Simple?
10. Breaking  stuff  is  enlightening
11. JMeter
15. Tsung
16. ab
17. BlazeMeter,  Blitz,  Loader
18. Minigun
19. (biased)  tldr: -­‐ab  for  raw  RPS -­‐Minigun for  everything  else
20. Load-­‐testing  activities
21. Load-­‐testing  activities 1. Exploratory  benchmarking
22. Load-­‐testing  activities 1. Exploratory  benchmarking 1. How  much  overhead  will  adding  NewRelic APM  add?
23. Load-­‐testing  activities 1. Exploratory  benchmarking 1. How  much  overhead  will  adding  NewRelic APM  add? 2. How  much  overhead  will  this  piece  of  embedded  Lua code  add  to  our   nginx?
24. Load-­‐testing  activities 1. Exploratory  benchmarking 1. How  much  overhead  will  adding  NewRelic APM  add? 2. How  much  overhead  will  this  piece  of  embedded  Lua code  add  to  our   nginx? 3. Shall  I  go  with  Heroku for  my  WS-­‐heavy  app?
25. Load-­‐testing  activities 1. Exploratory  benchmarking 1. How  much  overhead  will  adding  NewRelic APM  add? 2. How  much  overhead  will  this  piece  of  embedded  Lua code  add  to  our   nginx? 3. Shall  I  go  with  Heroku for  my  WS-­‐heavy  app? 4. External  APIs
26. Load-­‐testing  activities 1. Exploratory  benchmarking 1. How  much  overhead  will  adding  NewRelic APM  add? 2. How  much  overhead  will  this  piece  of  embedded  Lua code  add  to  our   nginx? 3. Shall  I  go  with  Heroku for  my  WS-­‐heavy  app? 2. Soak  testing
27. An  aside! “I'm  not  a  real  programmer.  I  throw  together  things   until  it  works  then  I  move  on.  The  real  programmers   will  say  "Yeah  it  works  but  you're  leaking  memory   everywhere.  Perhaps  we  should  fix  that."  I’ll  just  restart   Apache  every  10  requests.” -­‐ Rasmus Lerdorf (creator  of  PHP)
28. Load-­‐testing  activities 1. Exploratory  benchmarking 1. How  much  overhead  will  adding  NewRelic APM  add? 2. How  much  overhead  will  this  piece  of  embedded  Lua code  add  to  our   nginx? 3. Shall  I  go  with  Heroku for  my  WS-­‐heavy  app? 2. Soak  testing 3. Load  amplification
29. Load-­‐testing  activities 1. Exploratory  benchmarking 1. How  much  overhead  will  adding  NewRelic APM  add? 2. How  much  overhead  will  this  piece  of  embedded  Lua code  add  to  our   nginx? 3. Shall  I  go  with  Heroku for  my  WS-­‐heavy  app? 2. Soak  testing 3. Load  amplification 1. Add  extra  load,  in  production,  at  peak
30. Load-­‐testing  activities 1. Exploratory  benchmarking 1. How  much  overhead  will  adding  NewRelic APM  add? 2. How  much  overhead  will  this  piece  of  embedded  Lua code  add  to  our   nginx? 3. Shall  I  go  with  Heroku for  my  WS-­‐heavy  app? 2. Soak  testing 3. Load  amplification 1. Add  extra  load,  in  production,  at  peak 2. JustEat do  this  every  day
31. Load-­‐testing  activities 1. Exploratory  benchmarking 1. How  much  overhead  will  adding  NewRelic APM  add? 2. How  much  overhead  will  this  piece  of  embedded  Lua code  add  to  our   nginx? 3. Shall  I  go  with  Heroku for  my  WS-­‐heavy  app? 2. Soak  testing 3. Load  amplification 1. Add  extra  load,  in  production,  at  peak 2. JustEat do  this  every  day 3. Trellotoo
33. Why  Minigun? 1. Node.js!
34. Why  Minigun? 1. Node.js! 2. just  `npm install`
35. Why  Minigun? 1. Node.js! 2. just  `npm install` 3. HTTP  &  WS  out  of  the  box
36. Why  Minigun? 1. Node.js! 2. just  `npm install` 3. HTTP  &  WS  out  of  the  box 4. Request  chaining,  variable  payloads  =  complex  flows
37. Why  Minigun? 1. Node.js! 2. just  `npm install` 3. HTTP  &  WS  out  of  the  box 4. Request  chaining,  variable  payloads  =  complex  flows 5. JSON  reports
38. Why  Minigun? 1. Node.js! 2. just  `npm install` 3. HTTP  &  WS  out  of  the  box 4. Request  chaining,  variable  payloads  =  complex  flows 5. JSON  reports 6. Graphical  reports
39. Why  Minigun? 1. Node.js! 2. just  `npm install` 3. HTTP  &  WS  out  of  the  box 4. Request  chaining,  variable  payloads  =  complex  flows 5. JSON  reports 6. Graphical  reports 7. Plugin  support  (e.g.  StatsD output)
40. Why  Minigun? 1. Node.js! 2. just  `npm install` 3. HTTP  &  WS  out  of  the  box 4. Request  chaining,  variable  payloads  =  complex  flows 5. JSON  reports 6. Graphical  reports 7. Plugin  support  (e.g.  StatsD output) 8. CLI  or  library
41. Why  Minigun? 1. Node.js! 2. just  `npm install` 3. HTTP  &  WS  out  of  the  box 4. Request  chaining,  variable  payloads  =  complex  flows 5. JSON  reports 6. Graphical  reports 7. Plugin  support  (e.g.  StatsD output) 8. CLI  or  library 9. Decent  performance
42. https://github.com/artilleryio/minigun
43. Fin  1
44. Act  II  – Anatomy  of  a  load-­‐generator
45. • Interface
46. • Interface • Flow  control  /  concurrency
47. • Interface • Flow  control  /  concurrency • Performance
48. <3 • commander  for  CLI
49. <3 • commander  for  CLI • timers  &  event  emitters
50. <3 • commander  for  CLI • timers  &  event  emitters • async <3  <3  <3
51. <3 • commander  for  CLI • timers  &  event  emitters • async <3  <3  <3 • request  <3  and  ws <3
52. <3 • commander  for  CLI • timers  &  event  emitters • async <3  <3  <3 • request  <3  and  ws <3 • measured  for  stats
53. 1. minigun 2. minigun-­‐core
54. iTerm
55. Sending  lots  of  requests 1. Bottom  up
56. Sending  lots  of  requests 1. Bottom  up 1. request  spec  -­‐>  function
57. Sending  lots  of  requests 1. Bottom  up 1. request  spec  -­‐>  function 1. async.waterfall
58. Sending  lots  of  requests 1. Bottom  up 1. request  spec  -­‐>  function 1. async.waterfall
59. Sending  lots  of  requests 1. Bottom  up 1. request  spec  -­‐>  function 1. async.waterfall 2. What  creates  the  waterfall? 1. compileScenario()  -­‐>  function 1. async.waterfall with  a  context
60. Sending  lots  of  requests 1. Bottom  up 1. request  spec  -­‐>  function 1. async.waterfall 2. What  creates  the  waterfall? 1. compileScenario()  -­‐>  function 1. async.waterfall with  a  context
61. Sending  lots  of  requests 1. Bottom  up 1. request  spec  -­‐>  function 1. async.waterfall 2. What  creates  the  waterfall? 1. compileScenario()  -­‐>  function 1. async.waterfall with  a  context 2. One  up  to  Runner
62. Sending  lots  of  requests 1. Bottom  up 1. request  spec  -­‐>  function 1. async.waterfall 2. What  creates  the  waterfall? 1. compileScenario()  -­‐>  function 1. async.waterfall with  a  context 2. One  up  to  Runner 1. Each  phaseSpec -­‐>  function
63. Sending  lots  of  requests 1. Bottom  up 1. request  spec  -­‐>  function 1. async.waterfall 2. What  creates  the  waterfall? 1. compileScenario()  -­‐>  function 1. async.waterfall with  a  context 2. One  up  to  Runner 1. Each  phaseSpec -­‐>  function 1. interval  running  scenario  functions
64. Sending  lots  of  requests 1. Bottom  up 1. request  spec  -­‐>  function 1. async.waterfall 2. What  creates  the  waterfall? 1. compileScenario()  -­‐>  function 1. async.waterfall with  a  context 2. One  up  to  Runner 1. Each  phaseSpec -­‐>  function 1. interval  running  scenario  functions 2. phaseSpec functions  are  run  with  async.series
65. Sending  lots  of  requests 1. Bottom  up 1. request  spec  -­‐>  function 1. async.waterfall 2. What  creates  the  waterfall? 1. compileScenario()  -­‐>  function 1. async.waterfall with  a  context 2. One  up  to  Runner 1. Each  phaseSpec -­‐>  function 1. interval  running  scenario  functions 2. phaseSpec functions  are  run  with  async.series 3. Wait  to  finish  once  all  phases  launch
66. That’s  it
67. Two  more  things 1. measured
68. Two  more  things 1. measured 2. plugins
69. Two  more  things 1. measured 2. plugins 1. require  if  `config.settings`
70. Two  more  things 1. measured 2. plugins 1. require  if  `config.settings` 2. call  exported  function  with  config
71. Two  more  things 1. measured 2. plugins 1. require  if  `config.settings` 2. call  exported  function  with  config 3. and  give  it  an  EE
72. Act  II  Pt.  II  – Performance!
73. Performance 1. Unoptimized
74. Performance 1. Unoptimized 1. 500  rps on  entry  level  Digital  Ocean  droplet
75. Performance 1. Unoptimized 1. 500  rps on  entry  level  Digital  Ocean  droplet 2. `http`  is  ~2.2k
76. Performance 1. Unoptimized 1. 500  rps on  entry  level  Digital  Ocean  droplet 2. `http`  is  ~2.2k 3. ab  is  ~5.9k
77. Performance 1. Unoptimized 1. 500  rps on  entry  level  Digital  Ocean  droplet 2. `http`  is  ~2.2k 3. ab  is  ~5.9k 2. Goal:  do  as  little  as  possible
78. Performance 1. Unoptimized 1. 500  rps on  entry  level  Digital  Ocean  droplet 2. `http`  is  ~2.2k 3. ab  is  ~5.9k 2. Goal:  do  as  little  as  possible 1. Flamegraphs!
79. Performance 1. Unoptimized 1. 500  rps on  entry  level  Digital  Ocean  droplet 2. `http`  is  ~2.2k 3. ab  is  ~5.9k 2. Goal:  do  as  little  as  possible 1. Flamegraphs! 2. perf
80. Performance 1. eventemitter3?
81. Performance 1. eventemitter3? 2. neo-­‐async?
82. Performance 1. eventemitter3? 2. neo-­‐async? 3. nanotimer!
83. Act  III  – The  Joys  Of  Load-­‐Testing
84. Act  III  – The  Joys  Of  Load-­‐Testing 1.  Heroku WS
85. Act  III  – The  Joys  Of  Load-­‐Testing 1. Heroku WS 1. ~6k  open  connections
86. Act  III  – The  Joys  Of  Load-­‐Testing 1. Heroku WS 1. ~6k  open  connections 2. 160  conn/s
87. Act  III  – The  Joys  Of  Load-­‐Testing 1. Heroku WS 1. ~6k  open  connections 2. 160  conn/s 3. DigitalOcean is  ~7.2k  &  350  new  conn/sec
88. Act  III  – The  Joys  Of  Load-­‐Testing 1. New  Relic  agent 1. No  agent
89. Act  III  – The  Joys  Of  Load-­‐Testing "latency": { "min": 2.3, "max": 138.28, "median": 8.62, "p95": 30.28, "p99": 53.83 }
90. Act  III  – The  Joys  Of  Load-­‐Testing 1. New  Relic  agent 1. No  agent 2. With  an  agent
91. Act  III  – The  Joys  Of  Load-­‐Testing "latency": { "min": 3.08, "max": 298.44, "median": 23.96, "p95": 122.28, "p99": 264.9 }
92. Act  III  – The  Joys  Of  Load-­‐Testing 1. New  Relic  agent 1. No  agent 2. With  an  agent 1. at  the  top  of  the  requires
93. Act  III  – The  Joys  Of  Load-­‐Testing "latency": { "min": 9.04, "max": 8659.03, "median": 1447.43, "p95": 8227.31, "p99": 8416.88 }, "errors": { "ESOCKETTIMEDOUT": 190, "ETIMEDOUT": 805 }
94. Fin! @hveldstra https://yld.io