When benchmarks go bad - what I learned from measuring performance wrong

The world of performance analysis is littered with flawed claims, cognitive biases, dangerous intuitions, and beguiling
fallacies. Sadly, Holly has been guilty of all of the above! Repeatedly. But this is a no-judgement zone. Some
measurement anti-patterns are subtle, and some are downright counter-intuitive. In this talk, Holly will explain why
measuring performance is important, and talk through some of the ways it can go wrong. That would be depressing if that
was all there was, so she’ll also introduce a toolbox of questions and principles that you can use to improve the
performance of your own applications.

These include:

- How to set up a test system
- Recommended load generators
- The USE method

Transcript

1. holly cummins Devoxx UK when benchmarks go bad what I learned from measuring performance wrong
4. measure don’t guess
5. measure don’t guess is just the beginning
6. quarkusio/spring-quarkus-perf-comparison
7. was this a good measurement?
8. was this a good measurement?
9. was this a good measurement?
10. what was wrong?
12. building a benchmark is easy.
13. building a benchmark is easy. building a good benchmark is hard.
14. you are not measuring what you think you are measuring
15. yanmwytam you are not measuring what you think you are measuring
16. workload environment what’s run how it’s run
17. anatomy of a benchmark
18. anatomy of a benchmark
19. anatomy of a benchmark
20. anatomy of a benchmark
21. anatomy of a benchmark
22. anatomy of a benchmark
23. anatomy of a benchmark
24. anatomy of a benchmark
25. anatomy of a benchmark
26. reproducibility
27. reproducibility
28. reproducibility
29. you are not measuring what you think you are you are measuring noise
30. variation caused by environment issues ~40%
31. state is noise
32. caches are noise
33. sync && echo 3 > /proc/sys/vm/drop_caches
34. container images get cached
35. reproducibility stale database images (oops)
36. reproducibility stale database images (oops)
37. realism
38. realism
39. realism
40. even on rack-mounted servers cpu frequency is not deterministic
41. -disable turboboost -disable intel speedshift -set scaling_governor to ‘performance’ -set process priority (no more nice guy)
42. reproducibility load generation on the same machine
43. reproducibility load generation on the same machine
44. reproducibility if you can’t do that, set cpu affinity
45. everything is connected to everything problems may have asymmetric effects
47. quarkus can handle 1.8x more load
48. quarkus can handle 1.8x more load we thought we were isolating the app to 4 cores, but the app, load generator, and database were sharing all 16
49. quarkus can handle 1.8x more load we thought we were isolating the app to 4 cores, but the app, load generator, and database were sharing all 16 - Hibernate second-level cache was also enabled between these runs - Note the November 2025 dates on these charts - These runs had some other issues we discovered later; look at the latest data for our most-correct comparison quarkus can handle 3x more load
50. # disable turbo boost echo 1 >/sys/devices/system/cpu/intel_pstate/no_turbo # drop pagecache, dentries and inodes sync sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches’ taskset --cpu-list 0-3 java -Xms512m -Xmx512m -XX: +UseNUMA -XX:+UnlockDiagnosticVMOptions -XX: +DebugNonSafepoints -jar <whatever>
51. # disable turbo boost echo 1 >/sys/devices/system/cpu/intel_pstate/no_turbo # drop pagecache, dentries and inodes sync sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches’ taskset --cpu-list 0-3 java -Xms512m -Xmx512m -XX: +UseNUMA -XX:+UnlockDiagnosticVMOptions -XX: +DebugNonSafepoints -jar <whatever>
52. no one would run a real app like this
53. realism
54. realism be cautious of micro-benchmarks
55. realism warm up before measuring
56. realism have data in the database
58. I tried your benchmark, and Quarkus is only 1.4x faster
59. hardware schedulers make a big difference
60. relevance
61. relevance is the bottleneck what you think?
62. https://quarkus.io/blog/reactive-crud-performance-case-study/
63. https://quarkus.io/blog/reactive-crud-performance-case-study/ spring quarkus initial test 338 req/s 390 req/s with faster disk [not measured] 25,599 req/s
64. https://quarkus.io/blog/reactive-crud-performance-case-study/ spring quarkus initial test 338 req/s 390 req/s with faster disk [not measured] 25,599 req/s these are suspiciously similar
65. https://quarkus.io/blog/reactive-crud-performance-case-study/ spring quarkus initial test 338 req/s 390 req/s with faster disk [not measured] 25,599 req/s
66. https://quarkus.io/blog/reactive-crud-performance-case-study/ spring quarkus initial test 338 req/s 390 req/s with faster disk [not measured] 25,599 req/s
67. https://quarkus.io/blog/reactive-crud-performance-case-study/ spring quarkus initial test 338 req/s 390 req/s with faster disk [not measured] 25,599 req/s
68. ca
69. ca 0.14% faster
70. ca 0.14% faster 0.26% better
71. ca metrics are suspiciously similar 0.14% faster 0.26% better
72. does this tell us quarkus is as fast as rust?
73. does this tell us quarkus is as fast as rust?
74. no. does this tell us quarkus is as fast as rust?
75. it does tell us switching to rust for performance might be a waste of effort
76. it does tell us switching to rust for performance might be a waste of effort
77. relevance load generation on a different machine
78. consider limiting app cores to ensure load driving and database are not the bottleneck
79. relevance measuring multiple things at once
80. relevance varying multiple things at once
81. everything is connected to everything
82. anatomy of a benchmark
83. anatomy of a benchmark
84. coordinated omission
85. coordinated omission
86. if your system under starts slowing down, the injector slows down too, which makes your measures look better than they should
87. Coordinate omission makes what you think you are measuring response time when it’s really measures the service time component of latency. Service time is how long it takes to do the work. Response time is the amount of time spent waiting before the work starts.
88. not ok JMeter wrk ok gatling wrk2 HyperFoil
89. - never use self-reported values - measure time to first request measuring start time
90. ts=$(_date) while ! (curl -sf http://localhost:8080/fruits > /dev/ null) do # Spin here for max precision : done TTFR=$((($(_date) - ts)/1000000))
91. ts=$(_date) while ! (curl -sf http://localhost:8080/fruits > /dev/ null) do # Spin here for max precision : done TTFR=$((($(_date) - ts)/1000000))
92. # Start the client loop before the application so it's already polling ( while true; do # Try to open a TCP connection to the target host and port and use file descriptor 3 for it if exec 3<>/dev/tcp/"$HOST"/"$PORT"; then # Send HTTP GET request to the server if ! echo -e "GET $URL_PATH HTTP/1.0\r\nHost: $HOST\r\nConnection: close\r\n\r\n" >&3; then exec 3>&- continue fi # Read the HTTP response status line and extract the status code if ! read -r _ status_code _ <&3; then exec 3>&- continue fi # Close the file descriptor exec 3>&- # If we got a 200 OK response, exit the loop if [[ "$status_code" == "200" ]]; then break fi fi # Spin here and do nothing rather than waiting some arbitrary unlucky timing done # Record the timestamp when we successfully got a 200 response _date > "$END_TS_FILE" ) 2>/dev/null & CURL_PID=$! # Record start time and launch the application. # Redirect and exec inside the subshell so the application process directly replaces the subshell, making $APP_PID its actual PID. ts=$(_date) ( exec $RUN_CMD &>"$LOG_FILE" ) & APP_PID=$! # Ensure cleanup on exit (e.g. on timeout) trap "kill -15 $APP_PID $CURL_PID 2>/dev/null; wait $APP_PID 2>/dev/null || true; rm -f $END_TS_FILE" EXIT # Wait for the client loop to get a successful response wait $CURL_PID 2>/dev/null || true TTFR=$(($(cat "$END_TS_FILE") - ts))
93. measuring memory footprint
94. do not just measure heap measure “resident set size” (RSS)
96. there is no “true” RSS:
97. there is no “true” RSS: you can trade off RSS against CPU by shrinking max heap
98. ts=$(_date) while ! (curl -sf http://localhost:8080/fruits > /dev/null) do # Spin here for max precision : done TTFR=$((($(_date) - ts)/1000000)) RSS=`ps -o rss= -p $CURRENT_PID | sed 's/^ *//g'`
99. ts=$(_date) while ! (curl -sf http://localhost:8080/fruits > /dev/null) do # Spin here for max precision : done TTFR=$((($(_date) - ts)/1000000)) RSS=`ps -o rss= -p $CURRENT_PID | sed 's/^ *//g'`
100. ts=$(_date) while ! (curl -sf http://localhost:8080/fruits > /dev/null) do # Spin here for max precision : done TTFR=$((($(_date) - ts)/1000000)) RSS=`ps -o rss= -p $CURRENT_PID | sed 's/^ *//g'` it’s ok to measure time-to- fi rst-request and RSS in the same experiment
101. ts=$(_date) while ! (curl -sf http://localhost:8080/fruits > /dev/null) do # Spin here for max precision : done TTFR=$((($(_date) - ts)/1000000)) RSS=`ps -o rss= -p $CURRENT_PID | sed 's/^ *//g'`
102. do not try to measure RSS and max throughput in the same experiment
103. science 101: do not vary more than one thing at once
104. which is faster, a or b?
105. what even is “faster”? which is faster, a or b?
106. what problem are we trying to solve?
107. “performance” could be
108. “performance” could be response times
109. “performance” could be response times throughput
110. “performance” could be response times throughput memory footprint
111. - ops elasticity - hardware requirements - responsiveness leading indicators for
112. - cost - carbon footprint - user satisfaction leading indicators for
113. - cost - carbon footprint - user satisfaction leading indicators for
115. relevance science 101: do not vary more than one thing at once
116. you are not measuring what you think you are especially if you’re measuring more than one thing
117. how do you know what you’re measuring?
118. mcnamara fallacy
119. validate with data about real world consequences feedback is a friend guides decisions validates measurements
121. 3 times denser deployments without sacrificing availability and response times of services
122. how do you know what you’re measuring?
123. active benchmarking how do you know what you’re measuring?
124. brendan gregg’s USE method
125. for every resource, check -utilization -saturation -errors
126. for every resource, check -utilization -saturation -errors
127. for every resource, check -utilization -saturation -errors
129. had to tune keep-alive to avoid spring errors
130. spring quarkus app with code that deadlocked – 1.75 req/s initial test with correct code 338 req/s 390 req/s with faster disk [not measured] 25,599 req/s https://quarkus.io/blog/reactive-crud-performance-case-study/
131. spring quarkus app with code that deadlocked – 1.75 req/s initial test with correct code 338 req/s 390 req/s with faster disk [not measured] 25,599 req/s 2022-06-17 15:20:44,507 ERROR [org.hib.rea.errors] (vert.x-eventloop-thread-45) HR000057: Failed to execute statement [select zipcode0_.zip as zip1_0_0_, zipcode0_.city as city2_0_0_, zipcode0_.county as county3_0_0_, zipcode0_.state as state4_0_0_, https://quarkus.io/blog/reactive-crud-performance-case-study/ 2022-06-17 15:20:44,507 ERROR [org.hib.rea.errors] (vert.x-eventloop-thread-45) HR000057: Failed to execute statement [select zipcode0_.zip as zip1_0_0_, zipcode0_.city as city2_0_0_, zipcode0_.county as county3_0_0_, zipcode0_.state as state4_0_0_, zipcode0_.timezone as timezone5_0_0_, zipcode0_.type as type6_0_0_ from ZipCode zipcode0_ where zipcode0_.zip=?]: could not load an entity: [com.baeldung.quarkus_project.ZipCode#08231]: java.util.concurrent.CompletionException: io.vertx.core.impl.NoStackTraceThrowable Timeout at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) at io.vertx.core.Future.lambda$toCompletionStage$2(Future.java:362) ... at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: io.vertx.core.impl.NoStackTraceThrowable: Timeout 2022-06-17 15:20:44,507 ERROR [org.hib.rea.errors] (vert.x-eventloop-thread-45) HR000057: Failed to execute statement [select zipcode0_.zip as zip1_0_0_, zipcode0_.city as city2_0_0_, zipcode0_.county as county3_0_0_, zipcode0_.state as state4_0_0_, zipcode0_.timezone as timezone5_0_0_, zipcode0_.type as type6_0_0_ from ZipCode zipcode0_ where zipcode0_.zip=?]: could not load an entity: [com.baeldung.quarkus_project.ZipCode#08231]: java.util.concurrent.CompletionException: io.vertx.core.impl.NoStackTraceThrowable Timeout at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) at io.vertx.core.Future.lambda$toCompletionStage$2(Future.java:362) ... at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: io.vertx.core.impl.NoStackTraceThrowable: Timeout
132. spring quarkus app with code that deadlocked – 1.75 req/s initial test with correct code 338 req/s 390 req/s with faster disk [not measured] 25,599 req/s 2022-06-17 15:20:44,507 ERROR [org.hib.rea.errors] (vert.x-eventloop-thread-45) HR000057: Failed to execute statement [select zipcode0_.zip as zip1_0_0_, zipcode0_.city as city2_0_0_, zipcode0_.county as county3_0_0_, zipcode0_.state as state4_0_0_, https://quarkus.io/blog/reactive-crud-performance-case-study/ 2022-06-17 15:20:44,507 ERROR [org.hib.rea.errors] (vert.x-eventloop-thread-45) HR000057: Failed to execute statement [select zipcode0_.zip as zip1_0_0_, zipcode0_.city as city2_0_0_, zipcode0_.county as county3_0_0_, zipcode0_.state as state4_0_0_, zipcode0_.timezone as timezone5_0_0_, zipcode0_.type as type6_0_0_ from ZipCode zipcode0_ where zipcode0_.zip=?]: could not load an entity: [com.baeldung.quarkus_project.ZipCode#08231]: java.util.concurrent.CompletionException: io.vertx.core.impl.NoStackTraceThrowable Timeout at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) at io.vertx.core.Future.lambda$toCompletionStage$2(Future.java:362) ... at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: io.vertx.core.impl.NoStackTraceThrowable: Timeout 2022-06-17 15:20:44,507 ERROR [org.hib.rea.errors] (vert.x-eventloop-thread-45) HR000057: Failed to execute statement [select zipcode0_.zip as zip1_0_0_, zipcode0_.city as city2_0_0_, zipcode0_.county as county3_0_0_, zipcode0_.state as state4_0_0_, zipcode0_.timezone as timezone5_0_0_, zipcode0_.type as type6_0_0_ from ZipCode zipcode0_ where zipcode0_.zip=?]: could not load an entity: [com.baeldung.quarkus_project.ZipCode#08231]: java.util.concurrent.CompletionException: io.vertx.core.impl.NoStackTraceThrowable Timeout at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) at io.vertx.core.Future.lambda$toCompletionStage$2(Future.java:362) ... at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: io.vertx.core.impl.NoStackTraceThrowable: Timeout caused by misuse of @Transactional
133. spring quarkus app with code that deadlocked – 1.75 req/s initial test with correct code 338 req/s 390 req/s with faster disk [not measured] 25,599 req/s 2022-06-17 15:20:44,507 ERROR [org.hib.rea.errors] (vert.x-eventloop-thread-45) HR000057: Failed to execute statement [select zipcode0_.zip as zip1_0_0_, zipcode0_.city as city2_0_0_, zipcode0_.county as county3_0_0_, zipcode0_.state as state4_0_0_, https://quarkus.io/blog/reactive-crud-performance-case-study/ 2022-06-17 15:20:44,507 ERROR [org.hib.rea.errors] (vert.x-eventloop-thread-45) HR000057: Failed to execute statement [select zipcode0_.zip as zip1_0_0_, zipcode0_.city as city2_0_0_, zipcode0_.county as county3_0_0_, zipcode0_.state as state4_0_0_, zipcode0_.timezone as timezone5_0_0_, zipcode0_.type as type6_0_0_ from ZipCode zipcode0_ where zipcode0_.zip=?]: could not load an entity: [com.baeldung.quarkus_project.ZipCode#08231]: java.util.concurrent.CompletionException: io.vertx.core.impl.NoStackTraceThrowable Timeout at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) at io.vertx.core.Future.lambda$toCompletionStage$2(Future.java:362) ... at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: io.vertx.core.impl.NoStackTraceThrowable: Timeout 2022-06-17 15:20:44,507 ERROR [org.hib.rea.errors] (vert.x-eventloop-thread-45) HR000057: Failed to execute statement [select zipcode0_.zip as zip1_0_0_, zipcode0_.city as city2_0_0_, zipcode0_.county as county3_0_0_, zipcode0_.state as state4_0_0_, zipcode0_.timezone as timezone5_0_0_, zipcode0_.type as type6_0_0_ from ZipCode zipcode0_ where zipcode0_.zip=?]: could not load an entity: [com.baeldung.quarkus_project.ZipCode#08231]: java.util.concurrent.CompletionException: io.vertx.core.impl.NoStackTraceThrowable Timeout at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) at io.vertx.core.Future.lambda$toCompletionStage$2(Future.java:362) ... at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: io.vertx.core.impl.NoStackTraceThrowable: Timeout caused by misuse of @Transactional and, yes, it’s a bad error message and we fi xed that :)
134. https://quarkus.io/blog/reactive-crud-performance-case-study/ spring quarkus app with code that deadlocked – 1.75 req/s initial test with correct code 338 req/s 390 req/s with faster disk [not measured] 25,599 req/s
135. https://quarkus.io/blog/reactive-crud-performance-case-study/ spring quarkus app with code that deadlocked – 1.75 req/s initial test with correct code 338 req/s 390 req/s with faster disk [not measured] 25,599 req/s
136. https://quarkus.io/blog/reactive-crud-performance-case-study/ spring quarkus app with code that deadlocked – 1.75 req/s initial test with correct code 338 req/s 390 req/s with faster disk [not measured] 25,599 req/s
137. https://quarkus.io/blog/reactive-crud-performance-case-study/ spring quarkus app with code that deadlocked – 1.75 req/s initial test with correct code 338 req/s 390 req/s with faster disk [not measured] 25,599 req/s
138. https://quarkus.io/blog/reactive-crud-performance-case-study/ spring quarkus app with code that deadlocked – 1.75 req/s initial test with correct code 338 req/s 390 req/s with faster disk [not measured] 25,599 req/s
140. - can we get the same answer repeatedly?
141. - can we get the same answer repeatedly? - are results deterministic?
142. - can we get the same answer repeatedly? - are results deterministic? - is this close to real-world?
143. - can we get the same answer repeatedly? - are results deterministic? - is this close to real-world? - is this representative of the way applications will be run?
144. - can we get the same answer repeatedly? - are results deterministic? - are we reporting useful metrics? - is this close to real-world? - is this representative of the way applications will be run?
145. - can we get the same answer repeatedly? - are results deterministic? - are we reporting useful metrics? - does this help us make a decision? - is this close to real-world? - is this representative of the way applications will be run?
146. - can we get the same answer repeatedly? - are results deterministic? - are we reporting useful metrics? - does this help us make a decision? - is it answering a question we actually care about? - is this close to real-world? - is this representative of the way applications will be run?
147. it’s easy to make all three worse
148. for improvements, choose one :(
150. - evaluating before-and- after of performance tweaks - comparing frameworks against each other
151. - evaluating before-and- after of performance tweaks - comparing frameworks against each other - capacity planning - cost estimation - carbon footprint estimation
152. - evaluating before-and- after of performance tweaks - comparing frameworks against each other - answering questions - making decisions - capacity planning - cost estimation - carbon footprint estimation
153. the distillate
154. - measuring what you want to measure is hard the distillate
155. - measuring what you want to measure is hard - aim for reproducibility the distillate
156. - measuring what you want to measure is hard - aim for reproducibility - isolate application on its own hardware the distillate
157. - measuring what you want to measure is hard - aim for reproducibility - isolate application on its own hardware - disable turbo-boost the distillate
158. - measuring what you want to measure is hard - aim for reproducibility - isolate application on its own hardware - disable turbo-boost - aim for realism the distillate
159. - measuring what you want to measure is hard - aim for reproducibility - isolate application on its own hardware - disable turbo-boost - aim for realism - prod-like hardware, app, and data the distillate
160. - measuring what you want to measure is hard - aim for reproducibility - isolate application on its own hardware - disable turbo-boost - aim for realism - prod-like hardware, app, and data - aim for relevance the distillate
161. - measuring what you want to measure is hard - aim for reproducibility - isolate application on its own hardware - disable turbo-boost - aim for realism - prod-like hardware, app, and data - aim for relevance - similar measurements for different things usually mean you’re just measuring a sneaky bottleneck the distillate
162. - measuring what you want to measure is hard - aim for reproducibility - isolate application on its own hardware - disable turbo-boost - aim for realism - prod-like hardware, app, and data - aim for relevance - similar measurements for different things usually mean you’re just measuring a sneaky bottleneck - vary one thing at a time the distillate
163. - measuring what you want to measure is hard - aim for reproducibility - isolate application on its own hardware - disable turbo-boost - aim for realism - prod-like hardware, app, and data - aim for relevance - similar measurements for different things usually mean you’re just measuring a sneaky bottleneck - vary one thing at a time - validate measurements by active benchmarking the distillate
164. - measuring what you want to measure is hard - aim for reproducibility - isolate application on its own hardware - disable turbo-boost - aim for realism - prod-like hardware, app, and data - aim for relevance - similar measurements for different things usually mean you’re just measuring a sneaky bottleneck - vary one thing at a time - validate measurements by active benchmarking - measure a metric with business relevance the distillate
165. https://hollycummins.com/when-benchmarks-go-bad/