Performance Testing WebSites vs. (RESTful) APIs @ WebPerfMUC

Lars Wolff (Founder & CEO Stormforger): Performance Testing: WebSites vs. (RESTful) APIs

A lot of you know Lars and Sebastian from StormForger already. They are a referenceable source when it comes to load testing and web performance. This talk will give you an overview of approaches how to load test websites and/or APIs. It will point out common pitfalls, parts to take care of and best practices. @larsvegas (https://twitter.com/larsvegas)

https://www.meetup.com/de-DE/preview/Munchen-Web-Performance-Group/events/241842419

Transcript

1. Performance Testing Websites
 vs.
 Performance Testing (RESTful) APIs
2. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 EHLO[1] Lars Wolff
 @larsvegas • support@stormforger.com • Co-founder & CEO of
 stormforger.com • Software Development of web based systems • Agile Hippie (Coach)
 Certified ScrumMaster®, Kanban Management Professional • Amazon Web Services User Group Cologne
 #AWSUGCGN • Web Performance User Group Cologne • Web Montag Cologne
 #WEBMONTAG #WMCGN • DevHouseFriday Cologne
 #DevHouseFriday [1] http://www.ietf.org/rfc/rfc2821.txt
3. We empower organizations to do continuous performance testing.
4. EXPLICIT EVOLVING PARENTAL ADVISORY
5. I assume you are building/running a website and/or an API.
6. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Who does NOT have performance issues? :) • Who does clearly understand and has insights
 to these issues? • Under which circumstances do they occur? • What is your traffic scenario?
7. Basics
8. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Basics Most of this part should be clear. But I learned that it’s helpful to recap it.
9. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 HTTP request - response cycle http://celineotter.azurewebsites.net/world-wide-web-http-request-response-cycle/
10. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Performance Testing • Is business critical • Is orthogonal in your organisation • Results should always be available • Should be fully automated
11. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Performance Testing
 – Prerequisites • Goal? What do you want to learn? • Non-functional requirements? • Performance Budget? • Test and traffic scenario? • Environment to test against?
 (SUT = System Under Test)
12. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 1. (Non-functional) requirements 2. Test definition 3. Test execution 4. Analysis of resulting data Flow
13. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Performance Testing
 – Types of Testing 1. Load Testing 2. Stress Testing 3. Scalability Testing 4. Spike Testing 5. Soak Testing 6. Configuration Testing 7. Availability & Resilience Testing https://stormforger.com/blog/2016/07/08/types-of-performance-testing/ https://docs.stormforger.com/guides/
14. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 User, Application and System Monitoring • Real User Monitoring (RUM) • How does the client perform for all users? • Application Performance Monitoring • What happens inside my application? • What kind of request takes how long? Why? • What kind of follow-up requests / queries does my application create?
 • System-Monitoring • What's going on in my internal network? • What's going on in my web server / app server machine? • What’s going on in my database cluster? • What's about storage and IOPS?
15. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Mind your organisation 1. Performance testing and analysis is team work! 2. Please involve: product people, developers, QA, operations 3. “Working software over comprehensive documentation” – but make your results transparent to every stage of your organization!
16. Performance Testing Websites
17. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Website characteristics • multiple types of content
 (HTML, Images, CSS, JS, ...) • “client logic”
 (JavaScript, XHR) • complex client behavior
 (client type * user behavior * context) • “heavy”
18. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Performance Testing / Load Testing ≠ #WebPerf
19. #WebPerf
20. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 – https://www.w3.org/2010/webperf/ “[...] methods to measure and improve aspects of application performance of user agent features and APIs.” #WebPerf
21. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 #WebPerf 101 • less requests! (cache stuff) • less requests! (minify, concat) • less requests! (CDN for assets) • image compression / optimization ... • … a lot more … • how long does a tcp handshake take? :) • how much tcp connections does a (certain) browser open in parallel? :)
22. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 #WebPerf 101 Know your language • Naming in general – be precise • “loads slowly” vs. “takes long until it is displayed (= rendered)”
23. #PerfTest
24. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 –Wikipedia „In software engineering, performance testing is in general, a testing practice performed to determine how a system performs in terms of responsiveness and stability under a particular workload. It can also serve to investigate, measure, validate or verify other quality attributes of the system, such as scalability, reliability and resource usage.“
25. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Things you have to take care of • XHR & clients logic (JavaScript): A lot of client logic? Single page application (SPA)? • correct HTTP (error) response codes • Assets = bandwidth => CDN • Content extractions, e.g. CSRF token to submit form data • Double opt in • Test data in general
26. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Things you have to take care of • Testing external services? CDN? Tracking? Why??? • What does HTTP OPTIONS do? • Bandwidth: Shiny top of the fold header image and no CDN? :P
27. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 HTTP/1.1 vs. HTTP/2 • Some things change. • Some don’t. • What to focus on first? HTTP/2 Performance Anti-patterns by Ilya Grigorik
 https://docs.google.com/presentation/d/1_SMrVmiMxW2X1QZ1EcCnLKSosiD0PppP70Q3bw-l5Lg/present
28. Recap: Website
29. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Recap: Website 1. #WebPerf FTW 2. Seriously watch your bandwidth 3. Plenty other things to take care of: Test Data, Double Opt-in, CSRF, Auth-Tokens 4. Iterate slowly and communicate with other people 5. Never test external partners $$$
 (tracking, advertisers, etc.)
30. Performance Testing
 (RESTful) APIs
31. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 API characteristics • Usually XML or JSON responses • “Sequential” API flow, RESTful API flow, HATEOS / Hypermedia • Authenticated (authentication required or available auth tokens via test data) • No (browser) client, no client behavior – *sigh* :D • Requests from different clients (browser, mobile app, IoT, etc.) should be well-formed and ask for something really really specific • And... there is client logic (filters, paging, hypermedia)
32. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Things you have to take care of • Authentication (Basic Auth, SingleSignOn, oAuth, etc.) • HTTP headers (caching, ETags, gzip) • Header and content extractions ('id' to follow, 'auth-token' to use) • Test data in general • Correct HTTP (error) response codes • HTTP (error) codes in response body (response is HTTP200, content says 401 or “authentication required”) • RESTful (HTTP PUT vs. HTTP POST) • Auto follow of hypermedia links means you need a correct and full API specification :) • Rate limiting: Watch HTTP429 / too many requests, watch the fallback • One request per item vs. a request for a filterable batched result • Polling? Every minute? Caching fun. Don't send requests on a specific and fixed time... cron-style • No client handling / bad handling for HTTP5xx errors
33. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Advanced API performance testing
 – SOA & µService • Your environment is really distributed
 – monitoring and request tracing are must haves • Try to isolate each service and test it
 – firewall? proxy to get traffic into the data center... • Try to combine test scenarios of services for a particular user journey and test their contracts
34. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Advanced API performance testing
 – Serverless (here: AWS Lambda) • AWS Lambda limits*
 – e.g. 100 concurrent invocations • AWS API Gateway limits*
 – e.g. request throttling 1000 requests per second (rps) • What are your Service Contracts / SLAs? What happens in cascades? * ask AWS, they have knobs ;-) http://docs.aws.amazon.com/lambda/latest/dg/limits.html http://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html https://stormforger.com/presentations/serverles-microservices-vs-monolitic-beanstalk-app-loadtest/ https://stormforger.com/blog/2017/03/22/advices-on-performance-from-aws-well-architected-framework/
35. Recap: APIs
36. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Recap: APIs 1. Authentication 2. API flow and user journey 3. There actually is “client logic“ 4. Iterate slowly and communicate with other people
37. Wrap-up
38. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Wrap-up – How to do it? 1. Set a goal, set non-functional-requirements 2.Discuss user journey 3. Create minimal viable test case version
 (use HAR to scaffold for websites) 4. Test slowly 5. Iterate 6. Heads up with your team – communicate! 7. “Stale” your scenario 8. Run with more traffic for specific test type 9.Identify Problems, Bottlenecks 10.Fix stuff 11.Repeat
39. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Wrap-up Performance testing website and performance testing APIs Both is complex and there are plenty of things to take care of.
40. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Wrap-up Performance testing website and performance testing APIs Needs to involve people from all needed departments/professions in your organization.
41. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Wrap-up Performance testing website and performance testing APIs Should be started early.
42. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Wrap-up Performance testing website and performance testing APIs Should be started early – really!
43. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Wrap-up Performance testing website and performance testing APIs Is nothing to get stuck in the
 “Not Invented Here Syndrome”! – There are a lot of tools out there! https://en.wikipedia.org/wiki/Not_invented_here
44. Do you have questions? support@stormforger.com · https://stormforger.com · +49 221 64 30 51 28 Wrap-up Performance testing website and performance testing APIs #PerfTest early.
 #PerfTest often.
 #PerfTest continuously.
45. https://stormforger.com Thank you! :) Questions? Lars Wolff – @larsvegas – support@stormforger.com