#Perftest Early and Often – Ensure reliable {Applications, APIs} by continuous testing of performance characteristics

Load Testing is not only an important instrument for understanding a system’s behavior and its runtime environment but also great to check characteristics of non-functional requirements like performance, scalability, reliability, resiliency. All of these are essential to continuously deliver high-quality applications in cloud environments.

This talk covers how to make performance testing a daily habit for everyone in DevOps organizations. Learn how to match non-functional requirements, continuously deliver high-quality applications and let a sustainable performance culture evolve.

Lars Wolff is Co-founder and CEO of StormForger – a load & performance testing SaaS for DevOps organizations.
His journey led from failing 😉 to building and running web-based systems in small to large teams for products with no-traffic-at-all to multi region high-traffic production workloads.
Marked by pain, consulting and agile coaching became his main profession and finally boiled down to found StormForger because performance matters! With StormForger he helps teams to deliver fast, scalable, reliable systems in a cloudy digital product world. He’s in love with The Cloud™, DevOps, Scrum, Kanban, Whiteboards and Post-its®.

Transcript

1. Cologne Microservices User Group Perftest Early and Often – Ensure Reliable {Applications, APIs} with Continuous Testing of Performance Characteristics
2. Co-founder & CEO StormForger GmbH @larsvegas support@stormforger.com Software Development Consulting Agile Coaching Performance Testing Ship it! LARS WOLFF
3. COMPLEX SYSTEMS ARE SYSTEMS WITH HIGH INTERDEPENDENCY – PHYSICS
5. AT SPEED OF INNOVATION
6. CHALLENGES TO SUCCESSFULLY RUN WORKLOADS IN THE CLOUD Performance matters -Speed and responsiveness? -Conversion? -Customer Satisfaction? Scalability is key for the cloud -Capacity? -Thresholds for scaling? -Service Utilisation and Impact? Reliability is not negotiable -System Behaviour? -Fault Tolerance? -Fail-over and Self-healing? Cost Efficiency is business crucial -Service Configuration Efficiency? -Service Utilisation? -Efficient Budget Spendings?
7. AUTOMATED TESTING ✅ OF PERFORMANCE CHARACTERISTICS #
8. PERFORMANCE TESTING IS HARD
9. -Organizational obstacles, e.g. buy-in & priority -Testing goal & scope -Visibility in your own systems -The right performance test tool for the job -Start! $ PERFORMANCE TESTING IS HARD %&
10. -A fast pace innovation needs regular/automated testing -Complexity inherits known unkowns -Impact is uncertain -Complexity inherits a lot of unknown unknowns -Impact is unpredictable PERFORMANCE TESTING IS HARD
11. LET’S MAKE PERFORMANCE TESTING EASY
12. APPROACH
13. APPROACH -A Vision & Mission (SLAs, SLOs) -Requirements (SLIs and non-functional-requirements) -A easy load test tool -Iterative approach -Easy test case management -Pragmatism over perfection -Happy-path first, off-happy-path ASAP, fully-fledged maybe (not) -Automation and continuous feedback
14. “PERF TEST EARLY AND OFTEN” – PEOPLE WHO HAVE BEEN IN A LOT OF PAIN IN PRODUCTION ISSUES
16. Test Case -Create a first, simple Test Case -Q: Test Data? -Q: How much traffic? ITERATION 1
17. Test Case -Ask “Analytics” for the hour of the highest number of users in a week. -Analytics: We have 10.000 visits in this hour. The Users visits flower-list, one flower detail. 30% check out and payment. ITERATION 2
18. Test Case -Test Environment & Load Generator Setup -Since Infrastructure is code: Just start your test environment! ITERATION 3
19. WAR ROOM
20. Run Test -20 Minutes -Invite ALL stakeholders! -It’s about collaboration RUN
21. Inspect Report -Response Time? -If errors occurred: Do you see these in your monitoring, too? RESULTS
22. -Meet and communicate to stakeholders: -ONE Statement like We aim for baseline numbers. We tested flower-list and flower-detail. We created 4x traffic of peak hour. We saw 20% of errors. But always responded in under 1000ms (p99). -Ask Are you happy? -Ask Whats’ next? Checkout? RESULTS ' development ( operations ) QA * product + management , marketing
23. Test Case -Add Checkout ITERATION 4
24. Run Test -20 Minutes -Invite ALL stakeholders! -It’s about collaboration RUN
25. ITERATE AND MOVE FORWARD
26. -
27. Non-functional requirements -No errors! -Each request must be faster than 250ms (p99) NFR CHECKS
28. Automate -Run test runs from any pipeline -Run automated checks of non-functional requirements INTEGRATE
29. --
30. IT’S A FEEDBACK LOOP Requirements Definition (test scenario) Execution (test case) Result Interpretation & Reporting Stakeholder & Business Needs
31. START CREATING A COMMUNITY OF PRACTICE SPECIAL INTEREST GROUP SINCE PERFORMANCE TESTING IS AN ORTHOGONAL TOPIC
32. MORE CHALLENGES
33. MORE CHALLENGES -The Business in general and changing or unclear requirements -The runtime environment: The Cloud™ / K8S – These are complex systems, too! -Testing on service/component level vs. Testing E2E – however think in user journeys. -Test Scenarios itself can get tricky (Esoteric protocols, scripting needs, …)
34. ALL TOGEHTER
35. ALL TOGETHER -Start early -Pragmatism over perfection -Create assumptions and validate, challenge assumptions with requirements -Focus on your App/API capabilities, requirements, scenarios, test data, traffic schema -Communicate -Iterate -Test often, automate, integrate -Let a sustainable performance culture evolve
36. Thank you very much! Questions? support@stormforger.com