Bulletproof Node.js Backends (with load-testing & Artillery)

Node.js London Meetup, May 2016 - Why load test, do's and don'ts of load-testing, and why Artillery.io is a great choice for load-testing, plus a quick overview of flamegraphs and heapdumps in Node.js

Transcript

1. Bulletproof Node.js Backends Hassy Veldstra <h@veldstra.org> @hveldstra
2. # whoami • Node.js & DevOps engineer
3. # whoami • Node.js & DevOps engineer • Worked on some awesome Node.js projects
4. # whoami • Node.js & DevOps engineer • Worked on some awesome Node.js projects • Open-source: Artillery, Chaos Llama, Dino etc
5. # whoami • Node.js & DevOps engineer • Worked on some awesome Node.js projects • Open-source: Artillery, Chaos Llama, Dino etc • Stalk away! • https://github.com/hassy • https://github.com/shoreditch-ops • https://twitter.com/hveldstra
6. Node.js is everywhere <3 <3 <3
7. Node.js Backends • BFF
8. Node.js Backends • BFF • Web API (something that speaks HTTP/REST)
9. Node.js Backends • BFF • Web API (something that speaks HTTP/REST) • IoT backend (big peaks!)
10. Node.js Backends • BFF • Web API (something that speaks HTTP/REST) • IoT backend (big peaks!) • Realtime app (WebSocket / socket.io) – a chat app or a game backend
11. Node.js Backends • BFF • Web API (something that speaks HTTP/REST) • IoT backend (big peaks!) • Realtime app (WebSocket / socket.io) – a chat app or a game backend • Good old web application
12. Performance • Backend performance
13. Performance • Backend performance • Two options: • You find problems • Users find them
14. Performance • Backend performance • Two options: • You find problems • Users find them • Or: • You crash your app in development • Or it crashes under load in production
15. Performance • Backend performance • Two options: • You find problems • Users find them • Or: • You crash your app in development • Or it crashes under load in production • Or: • You don’t need to worry if you like downtime (and getting paged) • Of if your users love every extra 100ms of response time
17. Load-testing The only way to ship performant backends
18. Load-testing Load testing is the process of putting demand on a software system or computing device and measuring its response. Load testing is performed to determine a system's behavior under both normal and anticipated peak load conditions. -Wikipedia
19. Load-testing • More than just stress testing
20. Load-testing • More than just stress testing • Learning tool! • Learn about the limits of your code, your dependencies, your whole stack • Example: New Relic agent for Node can reduce the performance of your up by as much as 20% under high load scenarios
21. Load-testing • More than just stress testing • Learning tool! • Learn about the limits of your code, your dependencies, your whole stack • Example: New Relic agent for Node can reduce the performance of your up by as much as 20% under high load scenarios • Design aid • PDD: http://bit.ly/1rndEiw or https://blog.yld.io
22. Load-testing Checklist • Pick the right tool
23. Load-testing Checklist • Pick the right tool • Know what you’re testing
24. Less this
25. More this
26. Load-testing Checklist • Pick the right tool • Know what you’re testing • X-ray vision
28. Load-testing Checklist • Pick the right tool • Know what you’re testing • X-ray vision • i.e. visibility into what’s happening in your whole system, across the stack
29. Tools! • To load test, you need a load generator
30. Tools! • To load test, you need a load generator • Basically, a tool that can send a lot of requests to a server, very quickly
31. Tools! • To load test, you need a load generator • Basically, a tool that can send a lot of requests to a server, very quickly • Many options (30+ listed on Wikipedia)
32. Tools! • To load test, you need a load generator • Basically, a tool that can send a lot of requests to a server, very quickly • Many options (30+ listed on Wikipedia) • Where they differ is: • protocols they support • how flexible they are in allowing you to model the load • UI, UX, integrations
33. Artillery
34. Artillery • Supports multiple protocols (HTTP + WebSocket out of the box), extensible ”engine” interface. HTTP/2 and MQTT support coming.
35. Artillery • Supports multiple protocols (HTTP + WebSocket out of the box), extensible ”engine” interface. HTTP/2 and MQTT support coming. • Has a nice CLI. Integrates well with CI servers.
36. Artillery • Supports multiple protocols (HTTP + WebSocket out of the box), extensible ”engine” interface. HTTP/2 and MQTT support coming. • Has a nice CLI. Integrates well with CI servers. • Very flexible for modeling realistic load - user scenarios, arrival rates, loops
37. Artillery • Supports multiple protocols (HTTP + WebSocket out of the box), extensible ”engine” interface. HTTP/2 and MQTT support coming. • Has a nice CLI. Integrates well with CI servers. • Very flexible for modeling realistic load - user scenarios, arrival rates, loops • Virtual user behavior is scriptable with JS
38. Artillery • Supports multiple protocols (HTTP + WebSocket out of the box), extensible ”engine” interface. HTTP/2 and MQTT support coming. • Has a nice CLI. Integrates well with CI servers. • Very flexible for modeling realistic load - user scenarios, arrival rates, loops • Virtual user behavior is scriptable with JS • Good performance (multicore)
39. Artillery – Mental Model • Virtual users, arriving to your service
40. Artillery – Mental Model • Virtual users, arriving to your service • Phases control arrivals
41. Artillery – Mental Model • Virtual users, arriving to your service • Phases control arrivals • Each arrival is a new TCP socket (just like in the real world), picks a scenario, runs through it to completion or failure
42. Artillery – Mental Model • Virtual users, arriving to your service • Phases control arrivals • Each arrival is a new TCP socket (just like in the real world), picks a scenario, runs through it to completion or failure • Can reuse the same connection to run multiple scenarios with `loop`
43. Artillery • Demo: https://bit.ly/20mUbyf
44. When NOT to use Artillery
45. When NOT to use Artillery • You hate CLIs
46. When NOT to use Artillery • You hate CLIs • You love Java Swing GUIs from the 90s
48. http://bit.ly/1RYln1s (though we’re working on a cool React-based terminal UI)
49. When NOT to use Artillery • You hate CLIs • You love Java Swing GUIs from the 90s • You need to benchmark a web server
50. When NOT to use Artillery • You hate CLIs • You love Java Swing GUIs from the 90s • You need to benchmark a web server • You love Scala
51. When NOT to use Artillery • You hate CLIs • You love Java Swing GUIs from the 90s • You need to benchmark a web server • You love Scala • Use Gatling (but make sure other devs, ops, and QA all know & love Scala too)
52. When NOT to use Artillery • You hate CLIs • You love Java Swing GUIs from the 90s • You need to benchmark a web server • You love Scala • s/Scala/Python/ ?
53. When NOT to use Artillery • You hate CLIs • You love Java Swing GUIs from the 90s • You need to benchmark a web server • You love Scala • s/Scala/Python/ ? • Use Locust (same: make sure everyone knows Python. Also it’s HTTP-only)
54. When NOT to use Artillery • You hate CLIs • You love Java Swing GUIs from the 90s • You need to benchmark a web server • You love Scala • s/Scala/Python/ ? • Otherwise, ARTILLERY :D
55. When NOT to use Artillery • You hate CLIs • You love Java Swing GUIs from the 90s • You need to benchmark a web server • You love Scala • s/Scala/Python/ ? • Otherwise, ARTILLERY :D • Everybody know JS, and JSON/YAML
57. Load-testing: Modeling
58. Load-testing: Modeling • One of the most important things when load testing – being able to model the load accurately
59. Load-testing: Modeling • One of the most important things when load testing – being able to model the load accurately • Meaning – your load test needs to accurately simulate realistic conditions
60. Load-testing: Modeling • One of the most important things when load testing – being able to model the load accurately • Meaning – your load test needs to accurately simulate realistic conditions • If it doesn’t, you’re wasting your time
61. Load-testing: Modeling • Web server – lots of concurrent GET requests, something like wrk or loadtest is good: • wrk –c20 –d30s http://127.0.0.1/index.html • loadtest –c 20 –n 500 http://127.0.0.1/index.html
62. Load-testing: Modeling • Web server – lots of concurrent GET requests, something like wrk or loadtest is good: • wrk –c20 –d30s http://127.0.0.1/index.html • loadtest –c 20 –n 500 http://127.0.0.1/index.html • Web application or API – need scenarios, e.g.: • GET /search with some parameters • Pick one of the results • GET /products/$productId • POST /cart {productId: $productId}
63. Load-testing: Modeling • Web server – lots of concurrent GET requests, something like wrk or loadtest is good: • wrk –c20 –d30s http://127.0.0.1/index.html • loadtest –c 20 –n 500 http://127.0.0.1/index.html • Web application or API – need scenarios, e.g.: • GET /search with some parameters • Pick one of the results • GET /products/$productId • POST /cart {productId: $productId} • For this you want JMeter or Gatling or Artillery
64. Load-testing: Modeling • Web APIs / applications: • Public facing? • Behind a load-balancer?
65. Load-testing: Modeling • Web APIs / applications: • Public facing? • Behind a load-balancer? • How many TCP connections to open • Open all at once vs create new ones throughout the test
66. Load-testing: Modeling • Web APIs / applications: • Public facing? • Behind a load-balancer? • How many TCP connections to open • Open all at once vs create new ones throughout the test • Throughput / load-distribution • For HTTP – requests • For WebSocket – messages/sec, pubsub patterns, active vs idle connections
67. Load-testing: Modeling • Web APIs / applications: • Public facing? • Behind a load-balancer? • How many TCP connections to open • Open all at once vs create new ones throughout the test • Throughput / load-distribution • For HTTP – requests • For WebSocket – messages/sec, pubsub patterns, active vs idle connections • Shape of peaks and valleys
68. Load-testing: Modeling • WebSocket benchmark example https://github.com/geekforbrains/websocket- tests/blob/master/artillery.json
69. Load-testing: Modeling • WebSocket benchmark example https://github.com/geekforbrains/websocket- tests/blob/master/artillery.json Isn’t testing what it’s supposed to test! (it’s testing TCP connection establishment speed, not Websocket implementations)
70. Load-testing: Goals
71. Load-testing: Goals • Examples: • What is the p95 response time of this endpoint under heavy load, where heavy load = 25 RPS
72. Load-testing: Goals • Examples: • What is the p95 response time of this endpoint under heavy load, where heavy load = 25 RPS • How many active users can my server sustain, where a user = a WebSocket connection with 1 published message per second which is forwarded onto 75 other users on average
73. Load-testing: Goals • Examples: • What is the p95 response time of this endpoint under heavy load, where heavy load = 25 RPS • How many active users can my server sustain, where a user = a WebSocket connection with 1 published message per second which is forwarded onto 75 other users on average • How many concurrent user sessions above our current production peak can the production system sustain?
74. Metrics • Response time
75. Metrics • Response time • Changes in response time under load • And max acceptable p95 value
76. Metrics • Response time • Changes in response time under load • And max acceptable p95 value • Throughput at capacity
77. Metrics • Response time • Changes in response time under load • And max acceptable p95 value • Throughput at capacity • In Node.js: event loop lag, memory usage, GC activity
78. Examples
79. Examples • React server rendering – moving from Rails to Node.js (using Rails as an API server)
80. Rails
81. Express Rails
82. Examples – Server Rendering • How do you measure the performance of the Express handler? (to compare with Rails)
83. Examples – Server Rendering • How do you measure the performance of the Express handler? (to compare with Rails) • Without network latency between client and Node.js server
84. Examples – Server Rendering • How do you measure the performance of the Express handler? (to compare with Rails) • Without network latency between client and Node.js server • Without network latency between Node.js and Rails
85. Examples – Server Rendering • How do you measure the performance of the Express handler? (to compare with Rails) • Without network latency between client and Node.js server • Without network latency between Node.js and Rails • Without the time Rails takes to return the JSON
86. Examples – Server Rendering • Send requests to /foo/myview
87. Examples – Server Rendering • Send requests to /foo/myview • Vary them (different combos of query params) for a realistic simulation
88. Examples – Server Rendering • Send requests to /foo/myview • Vary them (different combos of query params) for a realistic simulation • Have the Node.js app record how long the request to Rails took and expose that as X- Backend-Response-Time header
89. Examples – Server Rendering • Send requests to /foo/myview • Vary them (different combos of query params) for a realistic simulation • Have the Node.js app record how long the request to Rails took and expose that as X- Backend-Response-Time header • Use responseTime() middleware to have X- Response-Time
90. Examples – Server Rendering • Send requests to /foo/myview • Vary them (different combos of query params) for a realistic simulation • Have the Node.js app record how long the request to Rails took and expose that as X-Backend- Response-Time header • Use responseTime() middleware to have X-Response- Time • For every request, the delta between the two is the time we spent purely in Express code – most of which is rendering React!
91. Examples – Server Rendering • Full script here: https://gist.github.com/hassy/05bbfbc09ff98a9a5 5706e4490d3d86f
92. Examples – Server Rendering • Results: Rails over 1s, Node.js ~220ms p95 for the same view
93. WIN!
94. Examples • React server rendering • Testing a proxy
95. Examples • React server rendering • Testing a proxy • Tested with production levels of traffic to make sure there were no memory leaks and to plan the number and type of dynos needed
96. Examples • React server rendering • Testing a proxy • Load-testing an ecommerce app in production (FTSE 500 company)
97. Examples • React server rendering • Testing a proxy • Load-testing an ecommerce app in production • Key: communication between dev, ops, and biz.
98. Examples • React server rendering • Testing a proxy • Load-testing an ecommerce app in production • Key: communication between dev, ops, and biz. • Start slowly in off-peak hours
99. Examples • React server rendering • Testing a proxy • Load-testing an ecommerce app in production • Key: communication between dev, ops, and biz. • Start slowly in off-peak hours • Ramp up synthetic load to peak level during off hours
100. Examples • React server rendering • Testing a proxy • Load-testing an ecommerce app in production • Key: communication between dev, ops, and biz. • Start slowly in off-peak hours • Ramp up synthetic load to peak level during off hours • Start adding load during busy hours
101. Examples • React server rendering • Testing a proxy • Load-testing an ecommerce app in production • Key: communication between dev, ops, and biz. • Start slowly in off-peak hours • Ramp up synthetic load to peak level during off hours • Start adding load during busy hours • Gradually increase to e.g. 2X production
102. Examples • React server rendering • Testing a proxy • Load-testing an ecommerce app in production • Key: communication between dev, ops, and biz. • Start slowly in off-peak hours • Ramp up synthetic load to peak level during off hours • Start adding load during busy hours • Gradually increase to e.g. 2X production • Separate synthetic traffic (header, cookie)
103. Examples • React server rendering • Testing a proxy • Load-testing an ecommerce app in production • Key: communication between dev, ops, and biz. • Start slowly in off-peak hours • Ramp up synthetic load to peak level during off hours • Start adding load during busy hours • Gradually increase to e.g. 2X production • Separate synthetic traffic (header, cookie) • No-op certain transactions if needed (e.g. checking out)
104. Examples • Also see: • http://clarkie.io/devops/2016/04/24/load-testing- dynamodb.html
105. X-Ray Vision • Must have visibility into your app – Heroku dashboard, New Relic, Graphana etc
106. X-Ray Vision • Must have visibility into your app – Heroku dashboard, New Relic, Graphana etc • Flamegraphs
107. X-Ray Vision • Must have visibility into your app – Heroku dashboard, New Relic, Graphana etc • Flamegraphs • Heapdumps
108. What is a flamegraph?
109. Where do call stacks come from? • Linux: perf • SmartOS / FreeBSD / OSX: dtrace • Windows: ? L
110. Flamegraphs – making one • node --perf_basic_prof myprogram.js • sudo perf record -q -F 99 -p $(pgrep node) -g - - sleep 10 • sudo chown root /tmp/perf-*.map • sudo perf script > /tmp/script • stackvis perf < /tmp/script > /tmp/flamegraph.html • (npm install -g stackvis)
111. Flamegraphs – making one #2 • Use the 0x package • https://www.npmjs.com/package/0x • Works on both Linux and OSX (wraps perf or dtrace) • Generates nice flamegraphs • http://davidmarkclements.github.io/0x-demo/
112. Let’s look at one again • http://davidmarkclements.github.io/0x-demo/
113. perf vs v8 profiler • Can sample as needed without restarting the process
114. perf vs v8 profiler • Can sample as needed without restarting the process • Extremely low overhead
115. perf vs v8 profiler • Can sample as needed without restarting the process • Extremely low overhead • Reaches into native code
116. perf vs v8 profiler • Can sample as needed without restarting the process • Extremely low overhead • Reaches into native code • BONUS: Works for other languages too
117. Flamegraphs tldr • Flamegraphs = awesome • Flamegraphs let you see what’s “hot” on the CPU in your Node.js code
118. heapdump
119. heapdump • npm install heapdump
120. heapdump • npm install heapdump • require(‘heapdump’)
121. heapdump • npm install heapdump • require(‘heapdump’) • kill -USR2 $my_node_process
123. heapdump • http://blog.yld.io/2015/08/10/debugging-memory- leaks-in-node-js-a-walkthrough/
124. Flamegraphs + Heapdump • Used on Artillery itself – compile scenarios, analyze request.js overhead, debug a memory leak in request, debug a memory leak in handlebars • IME heapdump is something you’ll reach out for more often
125. Summary • If you care about performance you must load- test • When you load-test, do it right • Model what actually happens in the app • Be methodical – hypothesis / question-driven • Learn how to instrument your apps • Profit!
126. Thanks! • https://artillery.io • Twitter: @hveldstra • http://veldstra.org