When Performance Matters – Effective Performance Testing from the Ground Up

Slides from my talk at W-JAX 2018: https://jax.de/devops-continuous-delivery/when-performance-matters-effective-performance-testing-from-the-ground-up

The ever-increasing complexity of modern systems, modern DevOps and continuous delivery-centric workflows place new demands on performance and reliability testing approaches. When systems are comprised of many distributed components, each with its own performance and reliability characteristics, and when a misconfiguration that causes a cascading failure under load can be automatically deployed across environments all the way from dev to production in a matter of hours, you really need to make sure that a rigorous, well-understood, and easy-to-follow performance testing process is in place. In this talk, we will look at how an effective performance testing process can be implemented from the ground up (using the open-source Artillery.io toolkit) to be an integral part of an organization’s continuous delivery pipeline. This talk is for developers, QAs, and engineering managers who are working on a greenfield project with high performance and reliability requirements or working on production systems which are experiencing issues when under load.

Transcript

1. EFFECTIVE PERFORMANCE TESTING FROM THE GROUND UP WHEN PERFORMANCE MATTERS Hassy Veldstra h@artillery.io - @hveldstra W-JAX, Nov 2018, Munich
2. HELLO
3. • Open source load & functional testing toolkit
4. WHAT THIS TALK IS NOT ABOUT
5. WHAT THIS TALK IS NOT ABOUT • Writing performant code
6. WHAT THIS TALK IS NOT ABOUT • Writing performant code • Profiling code to make it faster
7. WHAT THIS TALK IS NOT ABOUT • Writing performant code • Profiling code to make it faster • Benchmarking code
8. WHAT THIS TALK IS ABOUT
9. WHAT THIS TALK IS ABOUT • Understanding where performance testing fits into delivery process
10. WHAT THIS TALK IS ABOUT • Understanding where performance testing fits into delivery process • The people involved
11. WHAT THIS TALK IS ABOUT • Understanding where performance testing fits into delivery process • The people involved • Understanding how an effective performance testing strategy may be implemented
12. BACKGROUND
16. DONEC QUIS NUNC
17. DONEC QUIS NUNC
18. MOST IMPORTANTLY Performance is an key requirement
19. ECOMMERCE
20. IOT
21. EVENTS/TICKETING
22. • Chat • Games • Web APIs • Payments APIs
23. HOW DO WE MEET OUR PERFORMANCE GOALS?
24. QUESTIONS QUESTIONS
25. QUESTIONS QUESTIONS Where does performance testing fit into the development, testing, and delivery process?
26. QUESTIONS QUESTIONS How do we run performance tests in our CI/CD pipeline?
27. QUESTIONS QUESTIONS How do we define performance goals & SLOs* for our services? Can we have those checked automatically in CI?
28. SLOS Service Level Objectives, e.g.: • Support up to 2000 TPS • 99% of all requests should be served in under 200ms • No more than 0.1% of requests should return a 5xx response https://landing.google.com/sre/sre-book/chapters/service-level-objectives/ For a detailed discussion, see Google SRE book:
29. QUESTIONS QUESTIONS What types of performance tests are there? Which ones do we need for our services?
30. QUESTIONS QUESTIONS How do we organize our test suites? What are the best practices around structuring those tests?
31. QUESTIONS QUESTIONS How do we pick which services to test? Do we need to test all of our microservices? What about testing different environments and configurations easily?
32. QUESTIONS QUESTIONS How do we encourage collaboration on performance testing and involve everyone: developers, testers, SREs and product managers? (performance is everyone’s responsibility!)
33. QUESTIONS QUESTIONS What’s the best way to get started with all of this?
34. MAYBE SOME ANSWERS
35. MAYBE SOME ANSWERS Understand the place of performance testing in delivering process
36. MAYBE SOME ANSWERS How to go from zero to a performance testing suite that’s:
37. MAYBE SOME ANSWERS How to go from zero to a performance testing suite that’s: • Extensible & maintainable • Integrates into your CI/CD pipelines • Helps verify SLOs automatically • Works for developers, testers and SREs • Integrates with monitoring & reporting tools
38. DEFINITIONS
39. DEFINITIONS • Testing can suffer from lack of precision in terminology
40. DEFINITIONS • Testing can suffer from lack of precision in terminology • What is “testing”? Several different dimensions to consider
41. TYPE OF THING UNDER TEST
42. WHEN A TEST TAKES PLACE https://medium.com/@copyconstruct/testing-microservices-the-sane-way-9bb31d158c16
43. https://medium.com/@copyconstruct/testing-in-production-the-safe-way-18ca102d0ef1
44. WHAT THE TEST IS TRYING TO PREVENT
45. AN ASIDE http://www.skytopia.com/project/fractal/mandelbulb.html
46. WHAT IS A TEST?
47. WHAT IS A TEST? An activity, automated or manual, which:
48. WHAT IS A TEST? An activity, automated or manual, which: 1. Increases your confidence in some Thing
49. WHAT IS A TEST? An activity, automated or manual, which: 1. Increases your confidence in some Thing 2. Confirms that your understanding of a Thing or its behavior is still correct
50. WHAT IS A TEST? An activity, automated or manual, which: 1. Increases your confidence in some Thing 2. Confirms that your understanding of a Thing or its behavior is still correct 3. Increases your understanding of a Thing or its properties
51. INCREASE CONFIDENCE • A/B tests • Canarying • Traffic replay • Load test to add traffic above base level
52. CONFIRM UNDERSTANDING • Unit tests - known good/bad input, known good/ bad output • Contract-based / property-based testing • Load test on a known configuration that verifies some metrics afterwards (max response time <500ms)
53. INCREASE UNDERSTANDING • Sprint 100m and take a heart rate reading • Try opening 10k concurrent connections and see what happens • Exploratory testing of all kinds • Chaos testing
54. PERFORMANCE TESTING? any test that tests some performance-related property of a Thing. Often used interchangeably with “load testing”.
55. LOAD TESTING
57. GEOFILTERING • Give an IP address + country code • Returns yes/no
58. AUTH TOKEN SERVICE • Give username/password • Returns signed JWT token
59. OR A COMBINED API • Authenticate and get a token • Use the token to get geofiltering info
60. ACCEPTANCE/FUNCTIONAL TESTING
61. ACCEPTANCE/FUNCTIONAL TESTING • Verify that a service (or another unit!) conforms to its contract
62. ACCEPTANCE/FUNCTIONAL TESTING • Verify that a service (or another unit!) conforms to its contract • Verify that the results it produces make sense
63. AUTH SERVICE • Produces JWTs and not just empty 2xx responses • The token contains expected fields
64. • It’s possible to re-use the same test code for both acceptance tests and load tests if your tooling supports it
65. • It’s possible to re-use the same test code for both acceptance tests and load tests if your tooling supports it • (Artillery does!)
66. SMOKE TESTS
67. SMOKE TESTS • Main characteristic is their speed
68. SMOKE TESTS • Main characteristic is their speed • A test that determines whether other tests should even run
69. SMOKE TESTS • Main characteristic is their speed • A test that determines whether other tests should even run • Is anything obviously broken? If we plug this thing in and turn it on, is there any smoke?
71. SMOKE TESTS • Main characteristic is their speed • A test that determines whether other tests should even run • Is anything obviously broken? If we plug this thing in and turn it on, is there any smoke? • Run a quick happy-path test case
72. WHERE DOES PERFORMANCE TESTING FIT INTO THE DELIVERY PROCESS?
73. WHERE DOES PERFORMANCE TESTING FIT INTO THE DELIVERY PROCESS?
75. WHERE DOES PERFORMANCE TESTING FIT INTO THE DELIVERY PROCESS?
76. Early “write code” stage: help profile code or dependencies The rest typically is on a deployed service or API
77. WHAT TYPES OF PERFORMANCE TESTS ARE THERE?
78. WHAT TYPES OF PERFORMANCE TESTS ARE THERE? • Load test in CI/CD to continuously verify SLOs (service or composite API) — pre-prod
79. WHAT TYPES OF PERFORMANCE TESTS ARE THERE? • Load test in CI/CD to continuously verify SLOs (service or composite API) — pre-prod • Load testing to help capacity planning — pre-prod, manual
80. CAPACITY PLANNING • Identify amount of resources an instance of a service needs
81. CAPACITY PLANNING • Identify amount of resources an instance of a service needs • Identify the number of instances needed to meet a performance target
82. CAPACITY PLANNING • Identify amount of resources an instance of a service needs • Identify the number of instances needed to meet a performance target • Identify whether current capacity is sufficient or some limits will need to be raised pre-prod
83. WHAT TYPES OF PERFORMANCE TESTS ARE THERE? • Load test in CI/CD to continuously verify SLOs (service or composite API) — pre-prod • Load testing to help capacity planning — pre-prod, manual • Load test a service to better understand its scaling properties and tune configs — pre-prod, manual
84. WHAT TYPES OF PERFORMANCE TESTS ARE THERE? • Load test in CI/CD to continuously verify SLOs (service or composite API) — pre-prod • Load testing to help capacity planning — pre-prod, manual • Load test a service to better understand its scaling properties and tune configs — pre-prod, manual • Stress test a service to find its limits & understand how it degrades — pre-prod, manual
85. WHAT TYPES OF PERFORMANCE TESTS ARE THERE? • Load test in CI/CD to continuously verify SLOs (service or composite API) — pre-prod • Load testing to help capacity planning — pre-prod, manual • Load test a service to better understand its scaling properties and tune configs — pre-prod, manual • Stress test a service to find its limits & understand how it degrades — pre-prod, manual • In prod, manual or automatic — add extra traffic as a safety margin; a form of chaos testing
86. HOW DO WE RUN THESE IN CI/CD?
87. HOW DO WE RUN THESE IN CI/CD? • Create a parameterized CI job that can run a load test against a $service in an $environment with a $load_profile and (optionally) verify $slos.
88. HOW DO WE RUN THESE IN CI/CD? • Create a parameterized CI job that can run a load test against a $service in an $environment with a $load_profile and (optionally) verify $slos. • This can act as a stage in other pipelines
89. HOW DO WE RUN THESE IN CI/CD? • Create a parameterized CI job that can run a load test against a $service in an $environment with a $load_profile and (optionally) verify $slos. • This can act as a stage in other pipelines • … or be used manually by a dev/tester for ad-hoc testing via a web UI or a CLI (e.g. on Jenkins or AWS CodeBuild)
90. HOW DO WE RUN THESE IN CI/CD? • The tests don’t run on the CI server itself
91. HOW DO WE RUN THESE IN CI/CD? • The tests don’t run on the CI server itself • Best to be able to run them on your own (cloud) infrastructure
92. HOW DO WE RUN THESE IN CI/CD? • The tests don’t run on the CI server itself • Best to be able to run them on your own (cloud) infrastructure • Flexibility when it comes to VPCs or regions
93. HOW DO WE RUN THESE IN CI/CD? • The tests don’t run on the CI server itself • Best to be able to run them on your own (cloud) infrastructure • Flexibility when it comes to VPCs or regions • Critical for testing internal microservices
94. HOW DO WE RUN THESE IN CI/CD? • The tests don’t run on the CI server itself • Best to be able to run them on your own (cloud) infrastructure • Flexibility when it comes to VPCs or regions • Critical for testing internal microservices • More cost-effective too
95. HOW DO WE RUN THESE IN CI/CD?
97. DONEC QUIS NUNC
98. DONEC QUIS NUNC
99. HOW DO WE RUN THESE IN CI/CD?
100. HOW DO WE RUN THESE IN CI/CD?
101. HOW DO WE RUN THESE IN CI/CD? ❌
102. HOW DO WE RUN THESE IN CI/CD? ✅ ❌
103. HOW DO WE RUN THESE IN CI/CD? • What about test frequency?
104. HOW DO WE RUN THESE IN CI/CD? • What about test frequency? • No one-size-fits-all approach
105. HOW DO WE RUN THESE IN CI/CD? • What about test frequency? • No one-size-fits-all approach • Possible to test every change for microservices on the critical path, but probably excessive for most services
106. HOW DO WE RUN THESE IN CI/CD? • What about test frequency? • No one-size-fits-all approach • Possible to test every change for microservices on the critical path, but probably excessive for most services • Run on a schedule (e.g. nightly) • Run before a final promotion of a change to prod
107. DEFINING SLOS
108. DEFINING SLOS • Setting SLOs should be part of your team’s “creating a new microservice” checklist
109. DONEC QUIS NUNC
110. DEFINING SLOS • Setting SLOs should be part of your team’s “creating a new microservice” checklist • Documented alongside API specs, design docs (typically a Confluence/wiki page that follows a template)
111. DEFINING SLOS • Setting SLOs should be part of your team’s “creating a new microservice” checklist • Documented alongside API specs, design docs (typically a Confluence/wiki page that follows a template) • Involve other teams that may rely on the service!
112. DEFINING SLOS • Setting SLOs should be part of your team’s “creating a new microservice” checklist • Documented alongside API specs, design docs (typically a Confluence/wiki page that follows a template) • Involve other teams that may rely on the service! • Better to have some SLOs (& revise) than none at all
113. WHICH SERVICES SHOULD WE TEST?
114. WHICH SERVICES SHOULD WE TEST? • What could be tested?
115. WHICH SERVICES SHOULD WE TEST? • What could be tested? Anything with an API spec.
116. WHICH SERVICES SHOULD WE TEST? • What could be tested? Anything with an API spec. • Individual services
117. WHICH SERVICES SHOULD WE TEST? • What could be tested? Anything with an API spec. • Individual services • Composite APIs (that’s a “unit” with its own properties & behavior)
118. WHICH SERVICES SHOULD WE TEST? • What could be tested? Anything with an API spec. • Individual services • Composite APIs (that’s a “unit” with its own properties & behavior) • Does a microservice have SLOs? Then it should have a performance test to verify those automatically.
119. HOW DO WE ENCOURAGE COLLABORATION?
120. HOW DO WE ENCOURAGE COLLABORATION?
121. HOW DO WE ENCOURAGE COLLABORATION? • No magical solutions
122. HOW DO WE ENCOURAGE COLLABORATION? • No magical solutions • Involve all functions in discussions about performance
123. HOW DO WE ENCOURAGE COLLABORATION? • No magical solutions • Involve all functions in discussions about performance • Remove barriers:
124. HOW DO WE ENCOURAGE COLLABORATION? • No magical solutions • Involve all functions in discussions about performance • Remove barriers: • Use tools that everyone has access to • Use tools that everyone can work with
125. HOW DO WE ENCOURAGE COLLABORATION? • Monorepos help
126. HOW DO WE ENCOURAGE COLLABORATION? • Monorepos help • Good tooling helps
127. HOW DO WE ENCOURAGE COLLABORATION? • Monorepos help • Good tooling helps • Available to everyone, ideally open source
128. HOW DO WE ENCOURAGE COLLABORATION? • Monorepos help • Good tooling helps • Available to everyone, ideally open source • Easy to install and get started with
129. HOW DO WE ENCOURAGE COLLABORATION? • Monorepos help • Good tooling helps • Available to everyone, ideally open source • Easy to install and get started with • Uses a language that everyone is familiar with
130. HOW DO WE ENCOURAGE COLLABORATION? • Monorepos help • Good tooling helps • Available to everyone, ideally open source • Easy to install and get started with • Uses a language that everyone is familiar with • Make load testing reports & findings available to everyone (e.g. on Confluence/your wiki or KB)
131. HOW DO WE GET STARTED?
132. ARTILLERY 101
133. ARTILLERY 101 • Available on npm: npm install -g artillery
134. ARTILLERY 101 • Available on npm: npm install -g artillery • The artillery CLI is used to run tests and create HTML reports
135. ARTILLERY 101 • Available on npm: npm install -g artillery • The artillery CLI is used to run tests and create HTML reports • Tests are written in YAML and can be extended with Javascript
136. ARTILLERY 101 • Supports HTTP, Socketio, WebSocket, Kinesis, HLS out of the box.
137. ARTILLERY 101 • Supports HTTP, Socketio, WebSocket, Kinesis, HLS out of the box. • Third-party plugins for SQS, Lambda, SQL etc
138. ARTILLERY 101 • Supports HTTP, Socketio, WebSocket, Kinesis, HLS out of the box. • Third-party plugins for SQS, Lambda, SQL etc • Supports plugins. Out of the box: Statsd/Datadog/ Librato integration.
139. ARTILLERY 101 • Supports HTTP, Socketio, WebSocket, Kinesis, HLS out of the box. • Third-party plugins for SQS, Lambda, SQL etc • Supports plugins. Out of the box: Statsd/Datadog/ Librato integration. • Third-party plugins for other monitoring systems
140. ARTILLERY 101 • Designed to allow for complex, multi-step virtual user behavior to be scripted
141. ARTILLERY 101 • Designed to allow for complex, multi-step virtual user behavior to be scripted • Support for randomizing requests and capturing data from responses and re-using it in other requests
142. ARTILLERY 101 • Designed to allow for complex, multi-step virtual user behavior to be scripted • Support for randomizing requests and capturing data from responses and re-using it in other requests • Supports assertions on metrics such as HTTP latency — ie automated checking of SLOs
143. ARTILLERY 101 • Runs well in Docker, easy to run on ECS or Kubernetes or in a CI/CD pipeline
144. ARTILLERY 101 • Runs well in Docker, easy to run on ECS or Kubernetes or in a CI/CD pipeline • Can generate self-contained HTML reports with charts and graphs
145. ARTILLERY 101 • Runs well in Docker, easy to run on ECS or Kubernetes or in a CI/CD pipeline • Can generate self-contained HTML reports with charts and graphs • Features for creating modular test suites
146. CONFIG config: target: "" # we don't set a target by default environments: dev: target: "https://auth-service-dev.acme-corp.internal" defaults: headers: x-api-key: "0xcoffee" local: target: "http://localhost:8080" processor: “./functions.js" plugins: datadog: {} payload: - path: "./username-password.csv" fields: - username - password
147. ONE OR MORE SCENARIOS scenarios: - name: Authenticate with valid credentials flow: - post: url: "/auth" json: username: "{{ username }}" password: "{{ password }}" expect: - statusCode: 200 - contentType: json
148. $ artillery run \ --config config.yaml \ -e dev \ scenario.yaml
149. ORGANIZING OUR TEST SUITE
150. ORGANIZING OUR TEST SUITE • Using a monorepo
151. ORGANIZING OUR TEST SUITE • Using a monorepo • Easier to get started with, extend & maintain
152. ORGANIZING OUR TEST SUITE • Using a monorepo • Easier to get started with, extend & maintain • Easier to share across teams
153. ORGANIZING OUR TEST SUITE • Using a monorepo • Easier to get started with, extend & maintain • Easier to share across teams • Helps code reuse
154. ORGANIZING OUR TEST SUITE • Using a monorepo • Easier to get started with, extend & maintain • Easier to share across teams • Helps code reuse • Easier to work with in CI/CD pipelines
155. ORGANIZING OUR TEST SUITE https://github.com/shoreditch-ops/acme-corp-api-tests
156. ORGANIZING OUR TEST SUITE acme-corp-api-tests/ - services/ - auth-service/ - scenarios/ - config.yaml - functions.js - overrides.slo-response-time.json - package.json - common-config.yaml
157. WHY THAT STRUCTURE? • Extensible: • can add a new service or API easily, or add a new scenario to an existing one • allows for service-specific config such as environment URLs or data from external CSVs • allows for service-specific custom code, e.g. to generate random data in a certain format • Can encode service-specific load phases and SLOs
158. { "config": { "phases": [ { "duration": 120, "arrivalRate": 10, "rampTo": 20, "name": "Warm up the service" }, { "duration": 240, "arrivalRate": 20, "rampTo": 100, "name": "Ramp to high load" }, { "duration": 600, "arrivalRate": 100, "name": "Sustained high load" } ], "ensure": { "maxErrorRate": 0.1, "p99": 200 } } }
159. RUNNING A TEST artillery run \ —config ./services/auth-service/config.yaml \ --overrides "$(cat ./services/auth-service/ overrides.slos.json)” \ --e dev ./services/auth-service/login.yaml
160. RUNNING A TEST artillery run \ —config ./services/auth-service/config.yaml \ --overrides "$(cat ./services/auth-service/ overrides.slos.json)” \ --e dev ./services/auth-service/login.yaml
161. RUNNING A TEST artillery run \ —config ./services/auth-service/config.yaml \ --overrides "$(cat ./services/auth-service/ overrides.slos.json)” \ --e dev ./services/auth-service/login.yaml Easy to parameterize in CI/CD - service name, load/ SLO override, environment and optionally a scenario
162. RUNNING A TEST artillery run \ —config ./services/auth-service/config.yaml \ --overrides "$(cat ./services/auth-service/ overrides.slos.json)” \ --e dev ./services/auth-service/login.yaml Easy to parameterize in CI/CD - service name, load/ SLO override, environment and optionally a scenario Generic / reusable CI job!
163. RUNNING A TEST • Reusable by other jobs / pipeline stages • Or via the UI for ad hoc testing - e.g. in Jenkins or AWS CodeBuild
164. ORGANIZING OUR TEST SUITE acme-corp-api-tests/ - services/ - auth-service/ - scenarios/ - config.yaml - functions.js - overrides.slo-response-time.json - package.json - common-config.yaml
165. WHERE TO START?
166. WHERE TO START? • Pick one service
167. WHERE TO START? • Pick one service • Write tests using the template & set up a CI job to run them
168. WHERE TO START? • Pick one service • Write tests using the template & set up a CI job to run them • Show & tell to the rest of the team
169. WHERE TO START? • A good candidate service:
170. WHERE TO START? • A good candidate service: • Small API surface
171. WHERE TO START? • A good candidate service: • Small API surface • Has experienced performance issues, or
172. WHERE TO START? • A good candidate service: • Small API surface • Has experienced performance issues, or • On the critical path for other components, or
173. WHERE TO START? • A good candidate service: • Small API surface • Has experienced performance issues, or • On the critical path for other components, or • Has high performance requirements
174. WHERE TO START? • A good candidate service: • Small API surface • Has experienced performance issues, or • On the critical path for other components, or • Has high performance requirements • For example: an authentication service
175. SO… WE’VE LOOKED AT
176. SO… WE’VE LOOKED AT • What performance testing is, and different types of performance tests
177. SO… WE’VE LOOKED AT • What performance testing is, and different types of performance tests • Where performance testing fits into the development, testing, and delivery process
178. SO… WE’VE LOOKED AT • What performance testing is, and different types of performance tests • Where performance testing fits into the development, testing, and delivery process • Running performance tests in CI/CD pipelines
179. SO… WE’VE LOOKED AT • What performance testing is, and different types of performance tests • Where performance testing fits into the development, testing, and delivery process • Running performance tests in CI/CD pipelines • Setting and verifying SLOs
180. SO… WE’VE LOOKED AT • What performance testing is, and different types of performance tests • Where performance testing fits into the development, testing, and delivery process • Running performance tests in CI/CD pipelines • Setting and verifying SLOs • The mechanics of setting up a test suite with Artillery
181. THANKS & OVER TO YOU NOW! slides: