How We Benchmarked Quarkus: Patterns and anti-patterns

Holly, Eric, and Franz discuss some of the lessons learned in creating the new Quarkus benchmark suite.

Transcript

1. Holly Cummins Eric Deandrea Francesco Nigro How We Benchmarked Quarkus Patterns and anti-patterns Quarkus Insights April 20, 2026
2. @holly_cummins @edeandrea tl;dr
3. Quarkus is fast + efficient.
5. quarkusio/spring-quarkus-perf-comparison
6. @holly_cummins @edeandrea The backstory
7. @holly_cummins @edeandrea The backstory What problem were we solving?
8. before
9. before what Quarkus version is this?
10. before what Quarkus version is this? when was it measured?
11. before what Quarkus version is this? where is the source code? when was it measured?
12. before what Quarkus version is this? where is the source code? when was it measured? how can I reproduce?
13. before what Quarkus version is this? where is the source code? when was it measured? how can I reproduce? what framework is this?
14. before
15. before why is throughput missing?
16. before why is throughput missing? is throughput so bad Quarkus had to hide the numbers?
18. shadow-benchmarking
20. home-rolled benchmark home-rolled benchmark home-rolled benchmark home-rolled benchmark home-r benchm home-rolled benchmark
21. duplicated effort benchmarking anti-patterns
22. We needed a benchmark.
23. We made a benchmark.
24. after
25. open source after
26. traceable open source after
27. @holly_cummins @edeandrea The backstory What could possibly go wrong?
29. Building a benchmark is easy.
30. Building a benchmark is easy. Building a good benchmark is hard.
35. Benchmarks are like puppies
36. Benchmarks are like puppies
41. decisions,
42. decisions, decisions,
43. decisions, decisions, decisions
45. decisions
46. decisions application code
47. decisions application code
48. decisions application code how the app is executed
49. decisions application code how the app is executed execution environment
50. decisions application code how the app is executed execution environment
51. Every decision changes the numbers.
52. Every decision changes the numbers.
53. How should we make decisions?
54. How should we make decisions?
55. Guiding Principles
56. Guiding Principles Parity Like-for-like comparison App code should be equivalent
57. Guiding Principles Parity Like-for-like comparison App code should be equivalent Normalness Representative of a typical app No occult tuning
58. Guiding Principles Parity Like-for-like comparison App code should be equivalent Normalness Representative of a typical app No occult tuning High quality Model best app dev practices Model best performance practices
59. Guiding Principles Parity Like-for-like comparison App code should be equivalent Normalness Representative of a typical app No occult tuning High quality Model best app dev practices Model best performance practices Test framework, not infrastructure Results not dominated by database Aim to be CPU-bound
60. reproducibility - can we get the same answer repeatedly? - is there noise in the results from things outside our control? - are we reporting useful metrics? - does this help us make a decision? - is it answering a question we actually care about? - is this close to real-world? - is this representative of the way applications will be run? realism relevance
61. realism reproducibility relevance - best practices for performance testing - parity (ie fairness) - test the framework, not the infrastructure - normalness - best practices for application code
62. It’s easy to make all three worse realism reproducibility relevance
63. For improvements, choose one :( reproducibility realism relevance
65. echo "1" > /sys/devices/system/cpu/intel_pstate/no_turbo … —cpuset-cpus 1,4,6,8
66. echo "1" > /sys/devices/system/cpu/intel_pstate/no_turbo … —cpuset-cpus 1,4,6,8 But Francesco, no one would run a real application like that?
67. echo "1" > /sys/devices/system/cpu/intel_pstate/no_turbo … —cpuset-cpus 1,4,6,8 But Francesco, no one would run a real application like that? reproducibility realism relevance
68. Benchmarking is a post-truth discipline. realism reproducibility relevance
69. @holly_cummins @edeandrea Some of our learnings
70. @holly_cummins @edeandrea Mistakes we’ve made
71. @holly_cummins @edeandrea Decisions we’ve made
72. reproducibility realism relevance out-of-the-box or tuned?
73. reproducibility realism relevance how long should warmup be?
74. reproducibility realism relevance pinning work to cores
75. transactional reads? reproducibility realism relevance
76. reproducibility realism relevance database container networking
77. reproducibility realism relevance deterministic cpu frequency (spoiler: it isn’t by default)
78. reproducibility realism relevance how much data should be in the database?
79. reproducibility realism relevance stale database images (oops)
80. Decisions are hard (and unfair). Can we just … measure both ways?
82. JVM Native
83. Spring 4 Spring 3
84. Out of the box Tuned
85. With Virtual Threads No Virtual Threads
86. AOT Normal
87. AOT Normal Leyden AOT
88. AOT Normal Leyden AOT Spring AOT
89. Combinatorics.
90. Our current measurement matrix is … a lot. Out-of-the-box Tuned
91. @holly_cummins @edeandrea How universal are the results?
92. “It depends.”
93. The ultimate validation
94. The ultimate validation They could run 3 times denser deployments without sacrificing availability and response times of services”, Thornsten reiterated.
96. I tried your benchmark, and Quarkus is only 1.4x faster
97. decisions application code how the app is executed execution environment
98. decisions application code how the app is executed execution environment
99. decisions application code how the app is executed execution environment
100. decisions application code how the app is executed execution environment
101. hardware schedulers make a big difference
102. @holly_cummins @edeandrea The mistakes we didn’t make
103. @holly_cummins @edeandrea How not to benchmark
104. reproducibility realism relevance Running on a laptop don’t do this
105. reproducibility realism relevance don’t do this Other work running on the same machine (the load driver counts!)
106. reproducibility realism relevance Not having a clear question don’t do this
107. reproducibility realism relevance Not measuring what you think you’re measuring don’t do this
108. reproducibility realism relevance A measurement of the wrong bottleneck is a useless measurement
109. solution: active benchmarking
110. reproducibility realism relevance Coordinated omission don’t do this
111. reproducibility realism relevance Varying multiple things at once don’t do this
112. reproducibility realism relevance Measuring multiple things at once don’t do this
113. @holly_cummins @edeandrea What next?
114. reproducibility realism relevance more varied + complex application
115. reproducibility realism relevance -energy measurements -cost measurements
116. reproducibility realism relevance -energy measurements -cost measurements
117. Questions?