Presentation WASP Software Technology Cluster 2025

Transcript

1. Philipp Leitner Associate Professor philipp.leitner@chalmers.se http://icet-lab.eu Adventures in Software Performance Engineering
2. Chalmers 2
3. Chalmers 3
4. Chalmers 4 https://www.geeksforgeeks.org/performance-testing-software-testing/ In practice: Performance testing is often the “last step” before release
5. Chalmers 5 Luca Traini. Exploring performance assurance practices and challenges in agile software development: An ethnographic study. Empirical Software Engineering. 2022. “performance tests were executed at most once per release” “35 performance requirements were never tested in the last 6 releases, i.e., since 2 years”
6. Chalmers 6 “Shift Left” https://devopedia.org/shift-left
7. Chalmers 7 But … how?
8. Chalmers 8 Microbenchmarking with JMH
9. Chalmers 9 Some issues Slow Difficult to Write
10. Chalmers 10 Competitor 1 Competitor 2
11. Chalmers 11 Competitor 1 Competitor 2
12. Chalmers 12 Competitor 1 Competitor 2 ????
13. Chalmers 13
14. Chalmers 14 Costa, Bezemer, Leitner, Andrzejak (2021). What's Wrong with My Benchmark Results? Studying Bad Practices in JMH Benchmarks. IEEE Transactions on Software Engineering Bad Practices often massively affect benchmark results
15. Chalmers 15 Some issues and solutions: Benchmarks are difficult to write Benchmark bug finders using static analysis Benchmark generators Costa, Bezemer, Leitner, Andrzejak (2021). What's Wrong with My Benchmark Results? Studying Bad Practices in JMH Benchmarks. IEEE Transactions on Software Engineering Jangali, Tang, Alexandersson, Leitner, Yang, Shang (2022). Automated Generation and Evaluation of JMH Microbenchmark Suites from Unit Tests. IEEE Transactions on Software Engineering Rodriguez-Cancio, Combemale, Baudry (2016). Automatic microbenchmark generation to prevent dead code elimination and constant folding. In ASE '16
16. Chalmers 16 Laaber, Leitner (2018). An Evaluation of Open-source Software Microbenchmark Suites for Continuous Performance Assessment. In Proceedings of the 15th International Conference on Mining Software Repositories (MSR)
17. Chalmers 17 Laaber, Würsten, Gall, Leitner (2020). Dynamically Reconfiguring Software Microbenchmarks: Reducing Execution Time without Sacrificing Result Quality. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pp. 989–1001, New York, NY, USA.
18. Chalmers 18 Laaber, Würsten, Gall, Leitner (2020). Dynamically Reconfiguring Software Microbenchmarks: Reducing Execution Time without Sacrificing Result Quality. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pp. 989–1001, New York, NY, USA.
19. Chalmers 19 Laaber, Würsten, Gall, Leitner (2020). Dynamically Reconfiguring Software Microbenchmarks: Reducing Execution Time without Sacrificing Result Quality. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pp. 989–1001, New York, NY, USA.
20. Chalmers 20 Some issues and solutions: Benchmarks take a long time to execute Smart reconfiguration Benchmark selection Laaber, Würsten, Gall, Leitner (2020) Dynamically reconfiguring software microbenchmarks: reducing execution time without sacrificing result quality. In ESEC/FSE 2020 Traini, Cortellessa, Di Pompeo, Tucci (2023) Towards effective assessment of steady state performance in Java software: are we there yet? In Empirical Software Engineering Laaber, Gall, Leitner (2021) Applying test case prioritization to software microbenchmarks. In Empirical Software Engineering
21. Chalmers 21 Fundamental problems remain … Microbenchmarks cannot provide immediate feedback No “as you type” performance assessment
22. Chalmers 22 Maybe LLMs will generate optimal code for us?
23. Chalmers 23 Maybe LLMs will generate optimal code for us?
24. Chalmers 24 Maybe LLMs will generate optimal code for us? Some early results: Under submission
25. Chalmers 25 Shifting Left Software Performance Engineering Image credits: https://www.flaticon.com/