Slide 1
Slide 1 text
Philipp Leitner
Associate Professor
[email protected]
http://icet-lab.eu
Adventures in Software Performance Engineering
Slide 2
Slide 2 text
Chalmers 2
Slide 3
Slide 3 text
Chalmers 3
Slide 4
Slide 4 text
Chalmers 4
https://www.geeksforgeeks.org/performance-testing-software-testing/
In practice:
Performance testing is
often the “last step”
before release
Slide 5
Slide 5 text
Chalmers 5
Luca Traini. Exploring performance assurance practices and challenges in agile software development: An
ethnographic study. Empirical Software Engineering. 2022.
“performance tests were executed at most
once per release”
“35 performance requirements were never
tested in the last 6 releases, i.e., since 2
years”
Slide 6
Slide 6 text
Chalmers 6
“Shift Left”
https://devopedia.org/shift-left
Slide 7
Slide 7 text
Chalmers 7
But … how?
Slide 8
Slide 8 text
Chalmers 8
Microbenchmarking with JMH
Slide 9
Slide 9 text
Chalmers 9
Some issues
Slow
Difficult to Write
Slide 10
Slide 10 text
Chalmers 10
Competitor 1 Competitor 2
Slide 11
Slide 11 text
Chalmers 11
Competitor 1 Competitor 2
Slide 12
Slide 12 text
Chalmers 12
Competitor 1 Competitor 2
????
Slide 13
Slide 13 text
Chalmers 13
Slide 14
Slide 14 text
Chalmers 14
Costa, Bezemer, Leitner, Andrzejak (2021). What's Wrong with My Benchmark Results?
Studying Bad Practices in JMH Benchmarks. IEEE Transactions on Software Engineering
Bad Practices often
massively affect
benchmark results
Slide 15
Slide 15 text
Chalmers 15
Some issues and solutions:
Benchmarks are difficult to write
Benchmark bug finders using static analysis
Benchmark generators
Costa, Bezemer, Leitner, Andrzejak (2021). What's Wrong with My Benchmark Results?
Studying Bad Practices in JMH Benchmarks. IEEE Transactions on Software Engineering
Jangali, Tang, Alexandersson, Leitner, Yang, Shang (2022). Automated Generation and
Evaluation of JMH Microbenchmark Suites from Unit Tests. IEEE Transactions on
Software Engineering
Rodriguez-Cancio, Combemale, Baudry (2016). Automatic microbenchmark generation
to prevent dead code elimination and constant folding. In ASE '16
Slide 16
Slide 16 text
Chalmers 16
Laaber, Leitner (2018). An Evaluation of Open-source Software Microbenchmark Suites for
Continuous Performance Assessment. In Proceedings of the 15th International Conference
on Mining Software Repositories (MSR)
Slide 17
Slide 17 text
Chalmers 17
Laaber, Würsten, Gall, Leitner (2020). Dynamically Reconfiguring Software Microbenchmarks: Reducing
Execution Time without Sacrificing Result Quality. In Proceedings of the 28th ACM Joint Meeting on
European Software Engineering Conference and Symposium on the Foundations of Software Engineering
(ESEC/FSE), pp. 989–1001, New York, NY, USA.
Slide 18
Slide 18 text
Chalmers 18
Laaber, Würsten, Gall, Leitner (2020). Dynamically Reconfiguring Software Microbenchmarks: Reducing
Execution Time without Sacrificing Result Quality. In Proceedings of the 28th ACM Joint Meeting on
European Software Engineering Conference and Symposium on the Foundations of Software Engineering
(ESEC/FSE), pp. 989–1001, New York, NY, USA.
Slide 19
Slide 19 text
Chalmers 19
Laaber, Würsten, Gall, Leitner (2020). Dynamically Reconfiguring Software Microbenchmarks: Reducing
Execution Time without Sacrificing Result Quality. In Proceedings of the 28th ACM Joint Meeting on
European Software Engineering Conference and Symposium on the Foundations of Software Engineering
(ESEC/FSE), pp. 989–1001, New York, NY, USA.
Slide 20
Slide 20 text
Chalmers 20
Some issues and solutions:
Benchmarks take a long time to execute
Smart reconfiguration
Benchmark selection
Laaber, Würsten, Gall, Leitner (2020) Dynamically reconfiguring software microbenchmarks:
reducing execution time without sacrificing result quality. In ESEC/FSE 2020
Traini, Cortellessa, Di Pompeo, Tucci (2023) Towards effective assessment of steady
state performance in Java software: are we there yet? In Empirical Software Engineering
Laaber, Gall, Leitner (2021) Applying test case prioritization to software microbenchmarks.
In Empirical Software Engineering
Slide 21
Slide 21 text
Chalmers 21
Fundamental problems remain …
Microbenchmarks cannot provide immediate feedback
No “as you type” performance assessment
Slide 22
Slide 22 text
Chalmers 22
Maybe LLMs will generate optimal code for us?
Slide 23
Slide 23 text
Chalmers 23
Maybe LLMs will generate optimal code for us?
Slide 24
Slide 24 text
Chalmers 24
Maybe LLMs will generate optimal code for us?
Some early results:
Under submission
Slide 25
Slide 25 text
Chalmers 25
Shifting Left Software Performance Engineering
Image credits: https://www.flaticon.com/