London Node.js Meetup - Artillery.io

Transcript

1. Shipping indestructible systems with Artillery.io ! Hassy Veldstra @hveldstra London Node.js Meetup - May ‘17
2. $ • DevOps contractor
3. $ • DevOps contractor • Interested in resilience testing, fault-tolerant microservice architectures, chaos testing
4. $ • DevOps contractor • Interested in resilience testing, fault-tolerant microservice architectures, chaos testing • Created Artillery
5. Artillery.io • Load testing toolkit
6. Artillery.io • Load testing toolkit • Modern
7. Artillery.io • Load testing toolkit • Modern • Batteries-included, optimised for dev happiness
8. Artillery.io • Load testing toolkit • Modern • Batteries-included, optimised for dev happiness • Simple and powerful
9. Artillery.io • Load testing toolkit • Modern • Batteries-included, optimised for dev happiness • Simple and powerful • Open source
10. Objective Help developers ship scalable, fault-tolerant, and resilient systems
11. What Problems Does Artillery Solve?
12. What Problems Does Artillery Solve? 1)I want to see how my system reacts to load
13. What Problems Does Artillery Solve? 1)I want to see how my system reacts to load 2)I want to prepare my system for high load
14. What Problems Does Artillery Solve? cont’d
15. What Problems Does Artillery Solve? cont’d •I need to do load testing but JMeter makes me sad
16. What Problems Does Artillery Solve? cont’d •I need to do load testing but JMeter makes me sad •I want something that’s written in Node.js
17. What Problems Does Artillery Solve? cont’d •I need to do load testing but JMeter makes me sad •I want something that’s written in Node.js •I need to test WebSockets or Socket.io
18. What Problems Does Artillery Solve? cont’d •I need to do load testing but JMeter makes me sad •I want something that’s written in Node.js •I need to test WebSockets or Socket.io •I want something that can be scripted with JS
19. What Problems Does Artillery Solve? cont’d •I need to do load testing but JMeter makes me sad •I want something that’s written in Node.js •I need to test WebSockets or Socket.io •I want something that can be scripted with JS •I want something that can be used by everybody: QA team, SRE team, other dev teams
20. Users Ministry Of Justice, Apigee, Autodesk, Lifx, Nordstrom, Rockwell Automation, Xero, SolarWinds and many many more (insurance, connected devices, healthcare, financial services, games)
21. Community
22. Community • 45 contributors
23. Community • 45 contributors • Community plugins for TeamCity, DataDog, InfluxDB, CloudWatch, AWS Sigv4 generation
24. Community • 45 contributors • Community plugins for TeamCity, DataDog, InfluxDB, CloudWatch, AWS Sigv4 generation • Traffic recorder with Artillery output
25. Community • 45 contributors • Community plugins for TeamCity, DataDog, InfluxDB, CloudWatch, AWS Sigv4 generation • Traffic recorder with Artillery output • Postman to Artillery converter
26. Quick Demo #1
27. How Artillery Models Load
28. How Artillery Models Load • Users arrive to use our API
29. How Artillery Models Load • Users arrive to use our API • Each user will execute a scenario, which is a sequence of steps, to interact with the API
30. How Artillery Models Load • Users arrive to use our API • Each user will execute a scenario, which is a sequence of steps, to interact with the API • Users arrive in phases
31. How Artillery Models Load • Users arrive to use our API • Each user will execute a scenario, which is a sequence of steps, to interact with the API • Users arrive in phases • Each user is completely independent of all other users
32. How Artillery Models Load • Users arrive to use our API • Each user will execute a scenario, which is a sequence of steps, to interact with the API • Users arrive in phases • Each user is completely independent of all other users • Duration of a load test is the duration of the phases + the time it takes for all users to finish their scenarios
33. Realistic Load Modeling
34. Realistic Load Modeling • Users keep arriving regardless of whether other users have finished their scenarios or not
35. Realistic Load Modeling • Users keep arriving regardless of whether other users have finished their scenarios or not • Individual users will execute their scenarios sequentially*, waiting for a response before continuing
36. Realistic Load Modeling • Users keep arriving regardless of whether other users have finished their scenarios or not • Individual users will execute their scenarios sequentially*, waiting for a response before continuing • Each user will map to a new TCP connection (HTTP, WS, Socket.io)**
37. Quick Demo #2 • Same test but with a scenario file • Reading the output • HTML report • JSON log for custom reporting and such
38. Reading The Output • Percentiles • Usually the growth is interesting • Errors (e.g. ECONNRESET) and status codes >= 400 as well
39. Features, features, features!
40. Features, features, features! • GET/POST/PUT/DELETE/PATCH
41. Features, features, features! • GET/POST/PUT/DELETE/PATCH • Custom headers
42. Features, features, features! • GET/POST/PUT/DELETE/PATCH • Custom headers • Cookies
43. Features, features, features! • GET/POST/PUT/DELETE/PATCH • Custom headers • Cookies • Forms
44. Features, features, features! • GET/POST/PUT/DELETE/PATCH • Custom headers • Cookies • Forms • Request chaining
45. Request Chaining
46. Request Chaining • Make scenarios dynamic
47. Request Chaining • Make scenarios dynamic • Example: ecommerce user journey
48. Request Chaining • Make scenarios dynamic • Example: ecommerce user journey • Let’s look at a simple example
49. Request Chaining - post: url: “/session” json: username: “hassy” password: “londonnodejsmeetup” capture: - json: “$.accessToken” as: “accessToken”
50. Request Chaining - get: url: “/protected/resource” headers: Authorization: “Bearer {{ accessToken }}”
51. Hook functions
52. Hook functions • Just a Node module loaded by Artillery at runtime
53. Hook functions • Just a Node module loaded by Artillery at runtime • Any exported function is available in the scenario
54. Hook functions • Just a Node module loaded by Artillery at runtime • Any exported function is available in the scenario • Extension points:
55. Hook functions • Just a Node module loaded by Artillery at runtime • Any exported function is available in the scenario • Extension points: • beforeScenario, afterScenario
56. Hook functions • Just a Node module loaded by Artillery at runtime • Any exported function is available in the scenario • Extension points: • beforeScenario, afterScenario • beforeRequest, afterResponse
57. Hook functions • Just a Node module loaded by Artillery at runtime • Any exported function is available in the scenario • Extension points: • beforeScenario, afterScenario • beforeRequest, afterResponse • As a step anywhere in the scenario
58. Hook functions
59. Hook functions • Generate randomised data for your tests with Faker or Chance
60. Hook functions • Generate randomised data for your tests with Faker or Chance • Abort the test early if some condition is not satisfied
61. Hook functions • Generate randomised data for your tests with Faker or Chance • Abort the test early if some condition is not satisfied • Make a call to an external service to fetch some information that the virtual user needs
62. Example • Complete walkthrough: https://artillery.io/blog/ using-fakerjs-with-artillery
63. Performance
64. Performance • Multicore
65. Performance • Multicore • 1,600 RPS on an 2.4GHz Intel Atom C2550
66. Performance • Multicore • 1,600 RPS on an 2.4GHz Intel Atom C2550 • Plenty fast for meaningful testing on cheap hardware
67. Performance • Multicore • 1,600 RPS on an 2.4GHz Intel Atom C2550 • Plenty fast for meaningful testing on cheap hardware • Easy to set up on a cluster of beefy EC2 instances for massive tests
68. Demo!
69. Preparing For High Load
70. Preparing For High Load • Set up a test environment
71. Preparing For High Load • Set up a test environment • Set up monitoring
72. Preparing For High Load • Set up a test environment • Set up monitoring • Base-level infrastructure monitoring - EC2 instance metrics, ECS metrics, ELBs, RDS databases (DataDog)
73. Preparing For High Load • Set up a test environment • Set up monitoring • Base-level infrastructure monitoring - EC2 instance metrics, ECS metrics, ELBs, RDS databases (DataDog) • APM (NewRelic)
74. Preparing For High Load • Set up a test environment • Set up monitoring • Base-level infrastructure monitoring - EC2 instance metrics, ECS metrics, ELBs, RDS databases (DataDog) • APM (NewRelic) • Connect Artillery to the monitoring solution (DataDog)
75. Preparing For High Load cont’d
76. Preparing For High Load cont’d • Formulate your objective / Know what question you’re trying to answer / Always have a hypothesis to test.
77. Preparing For High Load cont’d • Formulate your objective / Know what question you’re trying to answer / Always have a hypothesis to test. • Keep a log!
78. Preparing For High Load cont’d • Formulate your objective / Know what question you’re trying to answer / Always have a hypothesis to test. • Keep a log! • Share the findings with the rest of your team
79. Staying Prepared
80. Staying Prepared • Load test regularly
81. Staying Prepared • Load test regularly • Run load tests after deploying to a certain environment (e.g. uat)
82. Staying Prepared • Load test regularly • Run load tests after deploying to a certain environment (e.g. uat) • Run load tests on a schedule (e.g. nightly) against a staging environment
83. Advanced Mode • Get to the point where you can load test production
84. Things That Go Well With Artillery • Chrome DevTools - full-stack JS performance • Chaos Lambda - backends that laugh at failures and just keep going • Fault tolerance = code testing, load testing, chaos testing
85. The End • @hveldstra • https://artillery.io • https://github.com/shoreditch-ops/artillery • https://github.com/shoreditch-ops/chaos- lambda