How Fast Is My App? Performance Testing 101

Product owners and stakeholders love to ask about performance. “Is the app fast? How fast?” What do those questions even mean? Are they interested in the number of transactions per second? The time for the page to load? How many users it can support at once? Performance testing is a much broader topic than just being able to say an application is “fast” or “slow.” There is load testing, stress testing, and soak testing. You can measure transaction times, concurrency, and capacity. You can address some performance issues with more systems, others with more memory or faster processors or a better network. It all depends on what you mean by "How fast?" Join Gene Gotimer as he showcases open source tools like JMeter and Gatling to measure different types of performance testing. He'll also discuss how Firefox and Chrome can show what the user experience is like in terms of performance. You will leave with a better understanding of what performance testing is, as well as some tools to get started implementing it in your delivery pipeline.

Transcript

1. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 1 @CoverosGene @dcast_io Agility. Security. Delivered. How Fast is My App? Performance Testing 101 Gene Gotimer gene.gotimer@coveros.com @CoverosGene @CoverosGene @dcast_io © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED.
2. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 2 @CoverosGene @dcast_io About Coveros • Coveros helps companies accelerate the delivery of secure, reliable software using agile methods • Agile & DevOps Services • DevOps Implementation • DevSecOps Integration • Agile Transformations & Coaching • Agile Software Development • Agile Testing & Automation • Agile, DevOps, Testing, Security Training • Open Source Products • SecureCI – Secure DevOps toolchain • Selenified – Agile test framework Development Platforms
3. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 3 @CoverosGene @dcast_io Selected Commercial Clients
4. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 4 @CoverosGene @dcast_io Selected Federal Clients
5. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 5 @CoverosGene @dcast_io What is Performance Testing? I Love Lucy “Job Switching” © 1952 CBS
6. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 6 @CoverosGene @dcast_io Principles of Performance Testing
7. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 7 @CoverosGene @dcast_io Page Loading Performance • Users are selfish when it comes to page loading times. They don't care what is slowing it down or why, they just want the page to load quickly. • If they think it is too slow for them, it is too slow for them. • Browsers often have performance measurement and analysis capabilities in the web development tools.
8. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 8 @CoverosGene @dcast_io Chrome Developer Tools
9. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 9 @CoverosGene @dcast_io Firefox Web Developer Tools
10. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 10 @CoverosGene @dcast_io Google PageSpeed Insights https://developers.google.com/speed/pagespeed/insights/
11. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 11 @CoverosGene @dcast_io Key Takeaway • If the user says the page is loading too slowly for them, it is loading too slowly for them. No matter how fast it loads. • Use browser and web site tools to measure page load speeds.
12. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 12 @CoverosGene @dcast_io Principles of Performance Testing What are we measuring? • How long does it take? • How big a system do we need? • How much can we handle at once? Time behavior Resource utilization Capacity
13. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 13 @CoverosGene @dcast_io Types of Performance Testing • Load testing • Stress testing • Scalability testing • Spike testing • Endurance testing • Concurrency testing • Capacity testing
14. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 14 @CoverosGene @dcast_io Types of Performance Testing • Load testing • Stress testing • Scalability testing • Spike testing • Endurance testing • Concurrency testing • Capacity testing
15. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 15 @CoverosGene @dcast_io Types of Performance Testing • Load testing • Stress testing • Scalability testing • Spike testing • Endurance testing • Concurrency testing • Capacity testing
16. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 16 @CoverosGene @dcast_io Types of Performance Testing • Load testing • Stress testing • Scalability testing • Spike testing • Endurance testing • Concurrency testing • Capacity testing
17. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 17 @CoverosGene @dcast_io Types of Performance Testing • Load testing • Stress testing • Scalability testing • Spike testing • Endurance testing • Concurrency testing • Capacity testing
18. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 18 @CoverosGene @dcast_io Types of Performance Testing • Load testing • Stress testing • Scalability testing • Spike testing • Endurance testing • Concurrency testing • Capacity testing
19. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 19 @CoverosGene @dcast_io Types of Performance Testing • Load testing • Stress testing • Scalability testing • Spike testing • Endurance testing • Concurrency testing • Capacity testing
20. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 20 @CoverosGene @dcast_io Key Takeaway • Performance testing is a field, not a particular kind of test. Understand what you are trying to test for.
21. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 21 @CoverosGene @dcast_io Sorting App • Spring Boot Java app • Static web page, links to REST calls • Sorting algorithms to sort 20,000 shuffled words • Different speeds to showcase performance results • Gatling and JMeter tests https://github.com/Coveros/sorting
22. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 22 @CoverosGene @dcast_io Gatling • Load test tool • Open source version • Simulation scripts are written in Scala • but you don't need to know much • Includes a web recorder https://gatling.io
23. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 23 @CoverosGene @dcast_io Gatling
24. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 24 @CoverosGene @dcast_io Apache JMeter • Load test tool • Free, open source • Very popular • Written in Java but tests are configured in XML or via GUI • Tests HTTP, HTTPS, SOAP, REST, JDBC, and more • Includes a web recorder https://jmeter.apache.org
25. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 25 @CoverosGene @dcast_io Apache JMeter
26. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 26 @CoverosGene @dcast_io Types of Performance Testing • Load testing • Stress testing • Scalability testing • Spike testing • Endurance testing • Concurrency testing • Capacity testing
27. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 27 @CoverosGene @dcast_io Testing Pyramid Unit Testing Functional Testing End- to-end Testing Unit test times, isolated load tests Full-blown transaction numbers via the UI, endurance, capacity, and scalability Baseline and trends of a selection of functional tests Load testing via the API User performance testing, concurrency, stress, and spike testing Load testing via the UI
28. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 28 @CoverosGene @dcast_io Unit Test Performance • Performance of unit tests on critical logic can catch changes in performance early. • Repeat each test to create a baseline and assume some variation in runtimes. • For example, if it usually takes 4.5s, you might fail at 5s or 6s
29. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 29 @CoverosGene @dcast_io Key Takeaway • Remember the testing pyramid. Use a mix of isolated and end-to-end and in between.
30. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 30 @CoverosGene @dcast_io Trends- not just absolute numbers • Short JMeter or Gatling test • On development system, no isolation • 10 concurrent users for 10 minutes • Track the trend • Are we getting slower or faster? • Full load and performance test • Dedicated environment, no other traffic • Production-sized servers • 1,000 concurrent users for 1 hour • What is the sustained capacity and throughput?
31. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 31 @CoverosGene @dcast_io JMeter Database Testing • Used pgBadger to analyze PostgreSQL query logs • 4 queries represented 85% of the traffic • Wrote 4 JDBC Sampler tests in JMeter to match those queries • Looping in roughly the same proportion • Created baseline, then watched trends • Database changes were tested for performance impacts http://pgbadger.darold.net
32. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 32 @CoverosGene @dcast_io Key Takeaway • Shift performance testing left. Get a baseline as early as possible. • Trends may be easier to test for and just as actionable as absolute performance numbers.
33. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 33 @CoverosGene @dcast_io Testing is not enough • Don't let performance be optional. • Pick a performance metric, make it a quality gate, fail builds that do not pass. • If you don't enforce it, it will be ignored.
34. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 34 @CoverosGene @dcast_io Key Takeaway • Break the build if you know something is too slow.
35. © COPYRIGHT 2020 COVEROS, INC. ALL RIGHTS RESERVED. 35 @CoverosGene @dcast_io #Coveros5 • If the user says the page is loading too slowly for them, it is loading too slowly for them. No matter how fast it is. • Performance testing is a field, not a particular kind of test. Understand what you are trying to test for. • Remember the testing pyramid. Use a mix of isolated and end-to-end and in between. • Trends may be easier to test for and just as actionable as absolute performance numbers. • Break the build if you know it is too slow.