Why Load & Performance Testing the Cloud?

At the AWS PopUp Loft in Berlin I gave a talk on why load and performance testing is still (or especially) relevant in the era of cloud infrastructure. The presentation was given on 23.10.2015.

Abstract:
The Cloud™ is infinite and scalable. Period.
Why is it important to test for performance and scalability characteristics of a cloud-based system? Won't AWS scale for me as long as I can afford it to?
Yes, but… AWS only operates and scales resources. They won't automatically make your system fast, stable and – more importantly – scalable. Performance Testing is crucial to understand your system and architecture design and your cloud hosting environment.

Transcript

1. Why Load & Performance Testing the Cloud? Sebastian Cohnen (@tisba)
2. • Focus on web performance and architectures • 6+ years consulting and development • Performance engineering & regular execution of dynamic performance analysis for customers EHLO! Sebastian Cohnen (@tisba)
 CEO & CTO StormForger.com
3. StormForger.com • SaaS Product & Consulting on Performance and Load Testing • We take care of the heavy lifting and pave the way for regular performance assessments • Free Preflight Tests while in Beta™
4. Load & Performance Testing
5. –Wikipedia “In software engineering, performance testing is in general, a testing practice performed to determine how a system performs in terms of responsiveness and stability under a particular workload.”
6. Performance Testing • A category of non-functional testing methods • Induce a well defined workload • Observe the systems behavior
7. There are many forms… • Load Testing • Stress Testing • Spike Testing • Soak Testing • Configuration Testing
8. Whatever you do…
10. Performance
11. ≠ Scalability Performance
12. Performance vs Scalability Performance
 
 "1 instance can handle 250 API requests per second with p99 at 50ms…" Scalability
 
 "10 instances can handle 2500 API requests per second with p99 at 50ms…"
13. You maybe want both! • Performance is important, because it defines a baseline for quality and capacity. It's also a measurement on efficiency. • Scalability is important, because it will tell you if and how efficiently you can grow capacity.
14. The Cloud • IaaS, PaaS, … • Automatization • On-Demand • Scaling http://geek-and-poke.com/
15. Back to Topic…
16. Why should I care?
17. Why should I care? …about load and performance testing the cloud?
18. "The Cloud is Scalable!"* …as long as I can afford it to be… *famous last words…
19. Scaling Applications Scaling Resources
20. Scaling Applications Scaling Resources ≠
21. Understanding
22. You need to understand… • Your (Distributed, μService, …) Application • Your Software Architecture • Cloud Environment • Used Services and their behavior
23. Complexity has not simply vanished! Complexity has not simply vanished!
24. Performance Testing …and some more reasons for testing!
25. Configuration Testing • Auto Scaling • Throughput Provisioning (EBS, DynamoDB, …) • Instance Types for EC2/RDS/EC/ES/… • ELB -> Web Server -> App Server -> Backend Services -> …
26. Performance Testing • Determine speed, throughput, capacity, … of your system • Figure out the capacity per resource
27. Scalability Analysis • How does my capacity increases with additional resources? • Requirement for Capacity Planning and Cost Estimation
28. Load Testing • What happens when I'm on hacker news? (Spike Testing) • How does your system and environment behave under serious stress? (Stress Testing) • What happens over longer period of times? (Soak Testing)
29. Availability & Resiliency • (Zero-Downtime) Deployment under load • Understanding failure scenarios • Failover mechanisms
30. Proof of Concept • Design an Architecture on AWS • Implement a Prototyp • (Discuss it with AWS Solution Architects) • Validate! Run Tests, observe, learn and improve!
31. Embrace the Cloud! • Multiple, scalable test environments used to be cumbersome and expensive • Use automatization wherever you can: • Services, Infrastructure, Servers, Code, Data…
32. One more thing…
33. Is this different than it used to be with "traditional hosting"?
34. Thanks! Questions? Sebastian Cohnen (@tisba)
 sebastian@stormforger.com