Performance Testing for DevOps in the Cloud

Performance tests are not only an important instrument for understanding a system and its runtime environment. It’s also essential in order to check stability and scalability – non-functional requirements that might be decisive for success, for example in e-commerce business.

Stability and fast response times are finally one of the success reasons of every cloud based set up. But won't AWS scale for me as long as I can afford it? Yes, but… AWS only operates and scales resources. They won't automatically make your system fast, stable and – more importantly – scalable.

Performance Testing is crucial to understand your system, architecture design, and your cloud hosting environment as recommended in the Performance & Efficiency Pillar of the AWS well-architected framework.

This talk shows how such and comparable questions can be clarified with performance tests and how DevOps teams benefit from regular test practise.

It will be discussed which obstacles have to be overcome to do continuous testing and what proved as best practise.

Transcript

1. Performance Testing for DevOps in the Cloud Lars Wolff – @larsvegas
 AWS Pop-up Loft Berlin 2018
2. Lars Wolff · @larsvegas EHLO AWS Pop-up Loft 2018 • Lars Wolff
 @larsvegas • Background:
 software engineering, consulting and agile coaching • Focus on agility, performance and ship it! • AWS-UserGroup-Cologne • Co-curating AWS Community Days • More Community Events: WebPerfMeetup CGN, DevHouseFriday • Co-founder & CEO https://stormforger.com
3. Performance Testing SaaS for DevOps Empower organizations to embrace
 continuous performance testing
4. Performance
5. Lars Wolff · @larsvegas Performance • Frontend Performance • Backend Performance
6. Lars Wolff · @larsvegas Performance The ability of a system to execute one task
 within a defined dimension = efficiency of a system
 ↪ 1 server manages 250 rps with p99 500ms.
7. Scalability
8. ≠ Performance Scalability
9. Lars Wolff · @larsvegas Scalability capacity 0 1000 2000 3000 4000 resources 5 10 15 20 25 30 Effectivity by which the capacity can be increased through adding resources
10. What makes me recognize that I have a performance problem? http://www.slideshare.net/jboner/scalability-availability-stability-patterns/15-How_do_I_know_if
11. Lars Wolff · @larsvegas If your system is slow for one single user.
12. What makes me recognize that I have a scaling problem? http://www.slideshare.net/jboner/scalability-availability-stability-patterns/15-How_do_I_know_if
13. If your system is fast for single users but slow under load. https://www.flickr.com/photos/jamespaullong/with/714577655/
14. Performance ⚡ Scalability – Robert Johnson https://www.facebook.com/note.php?note_id=409881258919 “…Efficiency projects rarely give you enough improvement to have a big effect on scaling.” “Scaling usually hurts efficiency…”
15. Back to Topic:
 The Cloud™
16. ☁The Cloud☁ • IaaS, PaaS, … • APIs and automation • on-demand • cost-effective • scalable
17. Performance testing in the cloud …finally!
18. Yes, and why?
19. The Cloud™ already scales for us, doesn’t it?
20. scaling resources
 ≠ scaling applications
21. Scaling resources vs. scaling applications Elastic Load
 Balancing EC2 Auto Scaling Group MySQL DB Instance
22. And managed services? • AWS takes care of everything! • AWS Lambda • AWS DynamoDB • AWS Kinesis/SQS
23. Complex systems
25. – Physics “Complex systems are systems with high interdependency.”
26. Complexity • Complexity has not disappeared • {S,P,I}aaS & Co. hide them (partly) • Effect on performance and scalability is not trivial
27. Understand the behavior • Individual usage • System & software architecture • Cloud/runtime environment • Used services and their behavior
28. ¯\_(ツ)_/¯
30. Speed!

31. Availability!
32. Performance Testing
33. “In software engineering,
 performance testing is in general,
 a testing practice performed to
 determine how a system performs
 in terms of responsiveness and stability
 under a particular workload.”
 – Wikipedia https://en.wikipedia.org/wiki/Software_performance_testing
34. Lars Wolff · @larsvegas Performance Testing • Stimulating the system with a defined load • Observing behavior • Verifying characteristics • Understanding behavior
35. Lars Wolff · @larsvegas (
36. Performance Tests • Collection of
 non-functional test methods • Not to be defined with
 absolute selectivity • Different aims and perspectives Load Testing Stress Testing Spike Testing Soak Testing Endurance Testing Resilience Testing Configuration Testing Scalability Testing
37. Lars Wolff · @larsvegas Performance Tests? • Not only for the big ones™! • Not only for testing releases • Evaluation of… • technology, • proof of concepts, • troubleshooting and debugging
38. less traffic
 ≠
 no performance problems
39. How do you do it?
40. Lars Wolff · @larsvegas Performance Testing Maturity • non at all tests • tests if problems during production • (short) tests before large-scale (sales & marketing) campaigns / • regularly tests / / • tests as a regular, integrated DevOpsQA instrument / / / / /
41. https://en.wikipedia.org/wiki/DevOps DevOps
42. – Allison McKnight, Etsy (@aemcknig) “Without a performance culture, performance is not sustainable” https://speakerdeck.com/aemcknig/building-performance-for-the-long-term#4
43. Lars Wolff · @larsvegas Challenges • (usually) no tooling problem • organizational obstacles, e.g. buy-in
 (management & team) • visibility in your own systems • start!
44. Starting …but how?
45. Lars Wolff · @larsvegas 5 steps to Perf Tests™ • bring together the stakeholders • define goals & scope • raise and determine requirements • assign the Who? and What? • execute tests together 1 development operations QA product management marketing
46. Lars Wolff · @larsvegas Bringing together the stakeholders • Development, operations, QA, product development, management and marketing • Why? Performance (testing) is a highly orthogonal topic! • All parts matter in preparation, planing and execution
47. Lars Wolff · @larsvegas Defining goals • Defining goals beforehand (why are we doing tests?) • Important!
 Always define goals (and requirements) from the specialist side ☝ • A goal can be very simple at first, e.g. “surviving Black Friday well” or “we want to become 20% faster”.
48. Lars Wolff · @larsvegas Where to start with testing? • defining System Under Test (SUT) • finding the right scope • perimeter/end-to-end tests • components/service tests
49. System Under Test • testing “from top” (end-to-end) or “sideward” against single services • for the compartment side, only integrative tests are interesting • testing single services for the purpose of trouble shooting (technical perspective)
50. Lars Wolff · @larsvegas Define requirements • making assumptions for non-functional requirements (NFR) • assistance via “baseline” performance tests and performance budgets • simple requirements might be: “deliver sites under 2sec”, “10k checkouts per hour”, … • also here: start simple, learn, improve, repeat! • do sanity checks!
51. Lars Wolff · @larsvegas 1 development operations QA scenarios • bouncing visitor • returning customer • walk-in customer • … product management marketing workloads • Black Friday peak hour • average peak hour • 3 hour fire sale • … Who? What?
52. Lars Wolff · @larsvegas Testing together • Especially before the first tests: involve all the stakeholders! • GO! • Present setup, assumptions and findings • Check or rework scenario sanity • Repeat!
53. Lars Wolff · @larsvegas How to begin? • Pragmatism over perfection • from … to … • simple to (more) complex, in small steps • selectively over regular to continuous testing • deduce, set and document assumptions • approach goals and problems from the business side of view
54. Performance tests in the cloud!
55. Load Testing • Tests with normal expectable load • Determination of latencies, throughput, error rates, … • Verification of non-functional requirements (SLA)
56. Stress Testing • Tests with increased up to extreme load • Understanding system behavior under (over)load • Identify limits and capacities
57. Scalability Testing • How effective are more resources transformed into capacity? • Foundation for capacity planning and cost estimation
58. Spike Testing • How does the system react during (extreme) peak loads? • Are we reacting fast enough? • Occasions: Marketing , Reddit, Hacker News, …, TV shows
59. Configuration Testing • How does the behavior change if the configuration modifies? • Series of test executions • Investigate the impact of the environment on your own system
60. …in the cloud • Types of instances • Auto scaling settings • Throughput provisioning • Ideal usage of services?
61. …and in general… • Hypervisor • Operating system • Web & application server • Software configuration • Software dependencies
62. Availability & Resilience Testing • (Zero-Downtime) deployments under load • Changes in infrastructure • Error scenarios • Failover testing • => Chaos engineering
63. Comply with the
 AWS Well-Architected Framework
 
 Load Testing is the tool to determine the right {architecture, instance type, services} https://d1.awsstatic.com/whitepapers/architecture/AWS_Well-Architected_Framework.pdf
64. Lars Wolff · @larsvegas Sounds like a long journey…
65. …but, actually, it’s a feedback loop. Aspects System under Test (SuT) Stakeholder Requirements Test Strategy Definition
 (test scenario) Implementation
 (test scenario) Execution
 (test case) Result Interpretation & Reporting Business Needs
66. Warroom
67. Scrum Flow Community of Practise (CoP)
68. Scaling Spotify à la Henrik Kniberg https://blog.crisp.se/wp-content/uploads/2012/11/SpotifyScaling.pdf Guild
69. DevOps Kanban Special Interest Group (SIG)
70. In SAFe® StormForger empowers to: • Raise, define and validate non-function-requirements (NFR) • Collaborate on NFRs, test scenarios and test results in CoP • Create test scenarios in development phase (shift left testing) • Execute test runs in every stage of the continuous delivery pipeline fully automated in the cloud • To deserve a Built-In Quality approach Community of Practice (CoP)
71. Lars Wolff · @larsvegas Conclusion • Scaling resources ≠ scaling applications • Complexity is still present – understanding behavior is enormously important • Performance (testing) is an orthogonal topic • Pragmatism over perfection • Define goals from the business side of view • Start! Iterate!
72. Thanks! Lars Wolff @larsvegas support@stormforger.com https://stormforger.com
73. Performance Testing SaaS for DevOps Empower organizations to embrace
 continuous performance testing