Share
Embed
Slide 1
Slide 1 text
Devoxx
when benchmarks go bad
what we learned by measuring performance wrong
holly cummins
francesco nigro
Slide 2
Slide 2 text
No content
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
measure don’t guess
Slide 5
Slide 5 text
measure don’t guess
is just the beginning
Slide 6
Slide 6 text
quarkusio/spring-quarkus-perf-comparison
Slide 7
Slide 7 text
was this a good
measurement?
Slide 8
Slide 8 text
was this a good
measurement?
Slide 9
Slide 9 text
what was wrong?
Slide 10
Slide 10 text
No content
Slide 11
Slide 11 text
building a benchmark is easy.
Slide 12
Slide 12 text
building a benchmark is easy.
building a good benchmark is hard.
Slide 13
Slide 13 text
you are not measuring what
you think you are measuring
Slide 14
Slide 14 text
you are not measuring what
you think you are measuring
yanmwytam
Slide 15
Slide 15 text
you are not measuring what
you think you are measuring
yanmwytam
Slide 16
Slide 16 text
workload
environment
what’s run
how it’s run
Slide 17
Slide 17 text
anatomy of a benchmark
Slide 18
Slide 18 text
anatomy of a benchmark
Slide 19
Slide 19 text
anatomy of a benchmark
Slide 20
Slide 20 text
anatomy of a benchmark
Slide 21
Slide 21 text
anatomy of a benchmark
Slide 22
Slide 22 text
anatomy of a benchmark
Slide 23
Slide 23 text
anatomy of a benchmark
Slide 24
Slide 24 text
anatomy of a benchmark
Slide 25
Slide 25 text
anatomy of a benchmark
Slide 26
Slide 26 text
reproducibility
Slide 27
Slide 27 text
reproducibility
Slide 28
Slide 28 text
reproducibility
Slide 29
Slide 29 text
reproducibility
Slide 30
Slide 30 text
you are not measuring what you think you are
you are measuring noise
Slide 31
Slide 31 text
~40%
we saw
variation caused by
environment issues
Slide 32
Slide 32 text
reproducibility
load generation on the same machine
Slide 33
Slide 33 text
reproducibility
load generation on the same machine
Slide 34
Slide 34 text
realism
Slide 35
Slide 35 text
realism
Slide 36
Slide 36 text
realism
Slide 37
Slide 37 text
cpu frequency is not deterministic
even on rack-mounted servers
Slide 38
Slide 38 text
reproducibility
-disable turboboost
-disable intel speedshift
-set scaling_governor to ‘performance’
-set process priority (no more nice guy)
Slide 39
Slide 39 text
state is noise
Slide 40
Slide 40 text
caches are noise
Slide 41
Slide 41 text
sync && echo 3 > /proc/sys/vm/drop_caches
Slide 42
Slide 42 text
container images get cached
Slide 43
Slide 43 text
reproducibility
stale database images
(oops)
Slide 44
Slide 44 text
reproducibility
stale database images
(oops)
Slide 45
Slide 45 text
everything is connected to everything
‘noise’ may have asymmetric effects
Slide 46
Slide 46 text
No content
Slide 47
Slide 47 text
quarkus can handle 1.8x more load than spring
Slide 48
Slide 48 text
quarkus can handle 1.8x more load than spring
we thought we were isolating the app to 4 cores, but the
app, load generator, and database were sharing all 16
Slide 49
Slide 49 text
fi
xed: quarkus can handle 3x more load than spring!
quarkus can handle 1.8x more load than spring
we thought we were isolating the app to 4 cores, but the
app, load generator, and database were sharing all 16
- Hibernate second-level cache was also enabled between these runs
- Note the November 2025 dates on these charts
- These runs had some other issues we discovered later; look at the latest data for our most-correct
comparison
Slide 50
Slide 50 text
# disable turbo boost
echo 1 >/sys/devices/system/cpu/intel_pstate/no_turbo
# drop pagecache, dentries and inodes
sync
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches’
taskset --cpu-list 0-3 java -Xms512m -Xmx512m -XX:
+UseNUMA -XX:+UnlockDiagnosticVMOptions -XX:
+DebugNonSafepoints -jar
Slide 51
Slide 51 text
# disable turbo boost
echo 1 >/sys/devices/system/cpu/intel_pstate/no_turbo
# drop pagecache, dentries and inodes
sync
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches’
taskset --cpu-list 0-3 java -Xms512m -Xmx512m -XX:
+UseNUMA -XX:+UnlockDiagnosticVMOptions -XX:
+DebugNonSafepoints -jar
Slide 52
Slide 52 text
no one would run a real app like this
Slide 53
Slide 53 text
realism
Slide 54
Slide 54 text
realism
be cautious of micro-benchmarks
Slide 55
Slide 55 text
realism
warm up before measuring
Slide 56
Slide 56 text
realism
have data in the database
Slide 57
Slide 57 text
No content
Slide 58
Slide 58 text
No content
Slide 59
Slide 59 text
I tried your
benchmark, and Quarkus
is only 1.4x faster
Slide 60
Slide 60 text
realism
operating system
schedulers make
a big difference
Slide 61
Slide 61 text
relevance
Slide 62
Slide 62 text
are we measuring the right thing?
Slide 63
Slide 63 text
which is faster, a or b?
Slide 64
Slide 64 text
which is faster, a or b?
what even is “faster”?
Slide 65
Slide 65 text
No content
Slide 66
Slide 66 text
No content
Slide 67
Slide 67 text
what problem are you
trying to solve?
Slide 68
Slide 68 text
“performance”
could be
Slide 69
Slide 69 text
“performance”
could be
throughput
Slide 70
Slide 70 text
“performance”
could be
response times
throughput
Slide 71
Slide 71 text
“performance”
could be
response times
throughput
memory
footprint
Slide 72
Slide 72 text
“performance”
could be
throughput
start time
response times
memory
footprint
Slide 73
Slide 73 text
throughput
start time
response times
memory
footprint
Slide 74
Slide 74 text
operational elasticity
throughput
start time
response times
memory
footprint
Slide 75
Slide 75 text
start time
response times
hardware requirements
operational elasticity
throughput
memory
footprint
Slide 76
Slide 76 text
start time
user satisfaction
response times
hardware requirements
operational elasticity
throughput
memory
footprint
Slide 77
Slide 77 text
start time
user satisfaction
response times
hardware requirements
operational elasticity
throughput
memory
footprint
Slide 78
Slide 78 text
start time
user satisfaction
response times
hardware requirements
operational elasticity
throughput
memory
footprint
Slide 79
Slide 79 text
start time
user satisfaction
response times
hardware requirements
operational elasticity
throughput
memory
footprint
Slide 80
Slide 80 text
start time
user satisfaction
response times
hardware requirements
operational elasticity
throughput
memory
footprint
Slide 81
Slide 81 text
start time
user satisfaction
response times
hardware requirements
operational elasticity
throughput
memory
footprint
Slide 82
Slide 82 text
leading indicators
lagging indicators
Slide 83
Slide 83 text
you are not measuring what
you think you are measuring
yanmwytam
Slide 84
Slide 84 text
measuring response
times the wrong way
Slide 85
Slide 85 text
measuring response
times the wrong way
Slide 86
Slide 86 text
No content
Slide 87
Slide 87 text
No content
Slide 88
Slide 88 text
coordinated omission
Slide 89
Slide 89 text
coordinated omission
2.5s 2.1s
Slide 90
Slide 90 text
coordinated omission
2.5s 2.1s
Slide 91
Slide 91 text
coordinated omission
2.5s 2.1s 1.9s 2.3s
Slide 92
Slide 92 text
coordinated omission
2.5s 2.1s 1.9s 2.3s
Slide 93
Slide 93 text
coordinated omission
2.5s 2.1s 1.9s 2.3s 20.9s
21.4s
Slide 94
Slide 94 text
coordinated omission
2.5s 2.1s 1.9s 2.3s 20.9s
21.4s
Slide 95
Slide 95 text
coordinated omission
2.5s 2.1s 1.9s 2.3s 20.9s
21.4s
1.8s 2.2s
Slide 96
Slide 96 text
coordinated omission
what?! probably
should be >20s
2.5s 2.1s 1.9s 2.3s 20.9s
21.4s
1.8s 2.2s
Slide 97
Slide 97 text
coordinated omission
- if your system under starts slowing down, the injector
slows down too
- makes your measures look better than they should
Slide 98
Slide 98 text
No content
Slide 99
Slide 99 text
not ok
Slide 100
Slide 100 text
not ok
JMeter
Slide 101
Slide 101 text
not ok
JMeter
wrk
Slide 102
Slide 102 text
not ok
JMeter
wrk
ok
Slide 103
Slide 103 text
not ok
JMeter
wrk
ok
Gatling
Slide 104
Slide 104 text
not ok
JMeter
wrk
ok
Gatling
wrk2
Slide 105
Slide 105 text
not ok
JMeter
wrk
ok
Gatling
wrk2
HyperFoil
Slide 106
Slide 106 text
measuring start time the wrong way
- never use self-reported values
- measure time to first request
Slide 107
Slide 107 text
ts=$(_date)
while ! (curl -sf http://localhost:8080/fruits > /dev/
null)
do
# Spin here for max precision
:
done
TTFR=$((($(_date) - ts)/1000000))
Slide 108
Slide 108 text
ts=$(_date)
while ! (curl -sf http://localhost:8080/fruits > /dev/
null)
do
# Spin here for max precision
:
done
TTFR=$((($(_date) - ts)/1000000))
Slide 109
Slide 109 text
# Start the client loop before the application so it's already polling
(
while true; do
# Try to open a TCP connection to the target host and port and use file descriptor 3 for it
if exec 3<>/dev/tcp/"$HOST"/"$PORT"; then
# Send HTTP GET request to the server
if ! echo -e "GET $URL_PATH HTTP/1.0\r\nHost: $HOST\r\nConnection: close\r\n\r\n" >&3; then
exec 3>&continue
fi
# Read the HTTP response status line and extract the status code
if ! read -r _ status_code _ <&3; then
exec 3>&continue
fi
# Close the file descriptor
exec 3>&# If we got a 200 OK response, exit the loop
if [[ "$status_code" == "200" ]]; then
break
fi
fi
# Spin here and do nothing rather than waiting some arbitrary unlucky timing
done
# Record the timestamp when we successfully got a 200 response
_date > "$END_TS_FILE"
) 2>/dev/null &
CURL_PID=$!
# Record start time and launch the application.
# Redirect and exec inside the subshell so the application process directly replaces the subshell, making $APP_PID its actual PID.
ts=$(_date)
( exec $RUN_CMD &>"$LOG_FILE" ) &
APP_PID=$!
# Ensure cleanup on exit (e.g. on timeout)
trap "kill -15 $APP_PID $CURL_PID 2>/dev/null; wait $APP_PID 2>/dev/null || true; rm -f $END_TS_FILE" EXIT
# Wait for the client loop to get a successful response
wait $CURL_PID 2>/dev/null || true
TTFR=$(($(cat "$END_TS_FILE") - ts))
Slide 110
Slide 110 text
measuring
memory footprint
Slide 111
Slide 111 text
do not just measure heap
measure “resident set size” (RSS)
Slide 112
Slide 112 text
heap
ff
o -heap memory
Slide 113
Slide 113 text
heap
o -heap memory
ff
usually smaller than the heap
Slide 114
Slide 114 text
ts=$(_date)
while ! (curl -sf http://localhost:8080/fruits > /dev/null)
do
# Spin here for max precision
:
done
TTFR=$((($(_date) - ts)/1000000))
RSS=`ps -o rss= -p $CURRENT_PID | sed 's/^ *//g'`
Slide 115
Slide 115 text
ts=$(_date)
it’s ok to measure time-to- rst-request
while ! (curl -sf http://localhost:8080/fruits
> /dev/null)
do
and RSS in the same experiment
# Spin here for max precision
:
done
TTFR=$((($(_date) - ts)/1000000))
fi
RSS=`ps -o rss= -p $CURRENT_PID | sed 's/^ *//g'`
Slide 116
Slide 116 text
ts=$(_date)
while ! (curl -sf http://localhost:8080/fruits > /dev/null)
do
# Spin here for max precision
:
done
TTFR=$((($(_date) - ts)/1000000))
RSS=`ps -o rss= -p $CURRENT_PID | sed 's/^ *//g'`
Slide 117
Slide 117 text
all these metrics affect each other
Slide 118
Slide 118 text
do not try to measure RSS and max
throughput in the same experiment
Slide 119
Slide 119 text
memory
Slide 120
Slide 120 text
what jvms do not
do with memory
no jvm would ever do this!
Slide 121
Slide 121 text
if work is happening, the jvm
will use up the capacity to
delay the next gc
Slide 122
Slide 122 text
No content
Slide 123
Slide 123 text
there is no “true” RSS:
Slide 124
Slide 124 text
there is no “true” RSS:
you can trade off RSS against CPU by
shrinking max heap
Slide 125
Slide 125 text
science 101:
do not vary more
than one thing at once
Slide 126
Slide 126 text
you are not measuring what you think you are
especially if you’re measuring more than one thing
Slide 127
Slide 127 text
are we measuring what
we think we are?
Slide 128
Slide 128 text
https://quarkus.io/blog/reactive-crud-performance-case-study/
Slide 129
Slide 129 text
https://quarkus.io/blog/reactive-crud-performance-case-study/
spring
quarkus
initial test
338 req/s
390 req/s
with faster disk
[not measured]
25,599 req/s
Slide 130
Slide 130 text
https://quarkus.io/blog/reactive-crud-performance-case-study/
initial test
spring
quarkus
338 req/s
390 req/s
these
are
suspiciously
similar
with faster disk
[not measured]
25,599 req/s
Slide 131
Slide 131 text
https://quarkus.io/blog/reactive-crud-performance-case-study/
spring
quarkus
initial test
338 req/s
390 req/s
with faster disk
[not measured]
25,599 req/s
Slide 132
Slide 132 text
https://quarkus.io/blog/reactive-crud-performance-case-study/
spring
quarkus
initial test
338 req/s
390 req/s
with faster disk
[not measured]
25,599 req/s
Slide 133
Slide 133 text
https://quarkus.io/blog/reactive-crud-performance-case-study/
spring
quarkus
initial test
338 req/s
390 req/s
with faster disk
[not measured]
25,599 req/s
Slide 134
Slide 134 text
relevance
is the bottleneck
what you think?
Slide 135
Slide 135 text
ca
Slide 136
Slide 136 text
ca
0.14% faster
Slide 137
Slide 137 text
ca
0.14% faster
0.26% better
Slide 138
Slide 138 text
ca
0.14% faster
0.26% better
metrics are suspiciously similar
Slide 139
Slide 139 text
does this tell us quarkus
is as fast as rust?
Slide 140
Slide 140 text
does this tell us quarkus
is as fast as rust?
Slide 141
Slide 141 text
does this tell us quarkus
is as fast as rust?
no.
Slide 142
Slide 142 text
it does tell us switching to
rust for performance
might be a waste of effort
Slide 143
Slide 143 text
it does tell us switching to
rust for performance
might be a waste of effort
Slide 144
Slide 144 text
No content
Slide 145
Slide 145 text
load generation on a different machine can be a fail
Slide 146
Slide 146 text
network can become
the bottleneck
Slide 147
Slide 147 text
all-in-one-topology
-pin load driver, database, and app to named cores
-use taskset, not --cpus
-use numa for memory affinity
Slide 148
Slide 148 text
how do you know what
you’re measuring?
Slide 149
Slide 149 text
how do you know what
you’re measuring?
active benchmarking
Slide 150
Slide 150 text
brendan gregg’s
USE method
Slide 151
Slide 151 text
for every resource, check
-utilization
-saturation
-errors
Slide 152
Slide 152 text
for every resource, check
-utilization
-saturation
-errors
Slide 153
Slide 153 text
for every resource, check
-utilization
-saturation
-errors
Slide 154
Slide 154 text
Running 30s test @ http://localhost:8080/fruits
2 threads and 100 connections
Thread Stats
Avg
Stdev
Max
+/- Stdev
Latency
14.88ms
14.36ms 343.93ms
90.45%
Req/Sec
6564.03
1260.71 7405.00
96.77
203485 requests in 30.01s, 162.60MB read
Requests/sec: 6780.57
Transfer/sec:
5.42MB
Socket errors: connectionErrors 2007, requestTimeouts 0
Slide 155
Slide 155 text
No content
Slide 156
Slide 156 text
we had to tune keep-alive
to avoid spring errors
Slide 157
Slide 157 text
https://quarkus.io/blog/reactive-crud-performance-case-study/
spring
quarkus
app with code
that deadlocked
–
1.75 req/s
initial test
with correct code
338 req/s
390 req/s
with faster disk
[not measured]
25,599 req/s
Slide 158
Slide 158 text
2022-06-17 15:20:44,507 ERROR [org.hib.rea.errors] (vert.x-eventloop-thread-45)
HR000057: Failed to execute statement [select
https://quarkus.io/blog/reactive-crud-performance-case-study/
zipcode0_.zip as zip1_0_0_, zipcode0_.city as city2_0_0_, zipcode0_.county as county3_0_0_, zipcode0_.state as state4_0_0_,
zipcode0_.timezone as timezone5_0_0_, zipcode0_.type as type6_0_0_ from ZipCode zipcode0_ where zipcode0_.zip=?]: could not load an
entity: [com.baeldung.quarkus_project.ZipCode#08231]: java.util.concurrent.CompletionException: io.vertx.core.impl.NoStackTraceThrowable
Timeout
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at io.vertx.core.Future.lambda$toCompletionStage$2(Future.java:362)
...
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
spring
quarkus
Caused by: io.vertx.core.impl.NoStackTraceThrowable: Timeout
2022-06-17 15:20:44,507 ERROR [org.hib.rea.errors] (vert.x-eventloop-thread-45) HR000057: Failed to execute statement [select
app with code
zipcode0_.zip as zip1_0_0_, zipcode0_.city as city2_0_0_, zipcode0_.county
as county3_0_0_, zipcode0_.state as state4_0_0_,
–
1.75 req/s
zipcode0_.timezone as timezone5_0_0_, zipcode0_.type as type6_0_0_ from ZipCode zipcode0_ where
zipcode0_.zip=?]:
could not load an
that deadlocked
entity: [com.baeldung.quarkus_project.ZipCode#08231]: java.util.concurrent.CompletionException:
io.vertx.core.impl.NoStackTraceThrowable
Timeout
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
initial test
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
338 req/s
390 req/s
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
with correct code
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at io.vertx.core.Future.lambda$toCompletionStage$2(Future.java:362)
with faster disk
[not measured]
25,599 req/s
...
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.vertx.core.impl.NoStackTraceThrowable: Timeout
2022-06-17 15:20:44,507 ERROR [org.hib.rea.errors] (vert.x-eventloop-thread-45) HR000057: Failed to execute statement [select
zipcode0_.zip as zip1_0_0_, zipcode0_.city as city2_0_0_, zipcode0_.county as county3_0_0_, zipcode0_.state as state4_0_0_,
Slide 159
Slide 159 text
2022-06-17 15:20:44,507 ERROR [org.hib.rea.errors] (vert.x-eventloop-thread-45)
HR000057: Failed to execute statement [select
https://quarkus.io/blog/reactive-crud-performance-case-study/
zipcode0_.zip as zip1_0_0_, zipcode0_.city as city2_0_0_, zipcode0_.county as county3_0_0_, zipcode0_.state as state4_0_0_,
zipcode0_.timezone as timezone5_0_0_, zipcode0_.type as type6_0_0_ from ZipCode zipcode0_ where zipcode0_.zip=?]: could not load an
entity: [com.baeldung.quarkus_project.ZipCode#08231]: java.util.concurrent.CompletionException: io.vertx.core.impl.NoStackTraceThrowable
Timeout
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at io.vertx.core.Future.lambda$toCompletionStage$2(Future.java:362)
...
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
spring
quarkus
Caused by: io.vertx.core.impl.NoStackTraceThrowable: Timeout
2022-06-17 15:20:44,507 ERROR [org.hib.rea.errors] (vert.x-eventloop-thread-45) HR000057: Failed to execute statement [select
app with code
zipcode0_.zip as zip1_0_0_, zipcode0_.city as city2_0_0_, zipcode0_.county
as county3_0_0_, zipcode0_.state as state4_0_0_,
–
1.75 req/s
zipcode0_.timezone as timezone5_0_0_, zipcode0_.type as type6_0_0_ from ZipCode zipcode0_ where
zipcode0_.zip=?]:
could not load an
that deadlocked
entity: [com.baeldung.quarkus_project.ZipCode#08231]: java.util.concurrent.CompletionException:
io.vertx.core.impl.NoStackTraceThrowable
Timeout
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
initial test
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
338 req/s
390 req/s
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
with correct code
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at io.vertx.core.Future.lambda$toCompletionStage$2(Future.java:362)
with faster disk
[not measured]
25,599 req/s
...
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.vertx.core.impl.NoStackTraceThrowable: Timeout
caused by misuse of @Transactional
2022-06-17 15:20:44,507 ERROR [org.hib.rea.errors] (vert.x-eventloop-thread-45) HR000057: Failed to execute statement [select
zipcode0_.zip as zip1_0_0_, zipcode0_.city as city2_0_0_, zipcode0_.county as county3_0_0_, zipcode0_.state as state4_0_0_,
Slide 160
Slide 160 text
2022-06-17 15:20:44,507 ERROR [org.hib.rea.errors] (vert.x-eventloop-thread-45)
HR000057: Failed to execute statement [select
https://quarkus.io/blog/reactive-crud-performance-case-study/
zipcode0_.zip as zip1_0_0_, zipcode0_.city as city2_0_0_, zipcode0_.county as county3_0_0_, zipcode0_.state as state4_0_0_,
zipcode0_.timezone as timezone5_0_0_, zipcode0_.type as type6_0_0_ from ZipCode zipcode0_ where zipcode0_.zip=?]: could not load an
entity: [com.baeldung.quarkus_project.ZipCode#08231]: java.util.concurrent.CompletionException: io.vertx.core.impl.NoStackTraceThrowable
Timeout
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at io.vertx.core.Future.lambda$toCompletionStage$2(Future.java:362)
...
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
spring
quarkus
Caused by: io.vertx.core.impl.NoStackTraceThrowable: Timeout
2022-06-17 15:20:44,507 ERROR [org.hib.rea.errors] (vert.x-eventloop-thread-45) HR000057: Failed to execute statement [select
app with code
zipcode0_.zip as zip1_0_0_, zipcode0_.city as city2_0_0_, zipcode0_.county
as county3_0_0_, zipcode0_.state as state4_0_0_,
–
1.75 req/s
zipcode0_.timezone as timezone5_0_0_, zipcode0_.type as type6_0_0_ from ZipCode zipcode0_ where
zipcode0_.zip=?]:
could not load an
that deadlocked
entity: [com.baeldung.quarkus_project.ZipCode#08231]: java.util.concurrent.CompletionException:
io.vertx.core.impl.NoStackTraceThrowable
Timeout
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
initial test
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
338 req/s
390 req/s
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
with correct code
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at io.vertx.core.Future.lambda$toCompletionStage$2(Future.java:362)
with faster disk
[not measured]
25,599 req/s
...
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
and, yes, it’s a bad error message and we xed that :)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.vertx.core.impl.NoStackTraceThrowable: Timeout
caused by misuse of @Transactional
fi
2022-06-17 15:20:44,507 ERROR [org.hib.rea.errors] (vert.x-eventloop-thread-45) HR000057: Failed to execute statement [select
zipcode0_.zip as zip1_0_0_, zipcode0_.city as city2_0_0_, zipcode0_.county as county3_0_0_, zipcode0_.state as state4_0_0_,
Slide 161
Slide 161 text
https://quarkus.io/blog/reactive-crud-performance-case-study/
spring
quarkus
app with code
that deadlocked
–
1.75 req/s
initial test
with correct code
338 req/s
390 req/s
with faster disk
[not measured]
25,599 req/s
Slide 162
Slide 162 text
https://quarkus.io/blog/reactive-crud-performance-case-study/
spring
quarkus
app with code
that deadlocked
–
1.75 req/s
initial test
with correct code
338 req/s
390 req/s
with faster disk
[not measured]
25,599 req/s
Slide 163
Slide 163 text
https://quarkus.io/blog/reactive-crud-performance-case-study/
spring
quarkus
app with code
that deadlocked
–
1.75 req/s
initial test
with correct code
338 req/s
390 req/s
with faster disk
[not measured]
25,599 req/s
Slide 164
Slide 164 text
https://quarkus.io/blog/reactive-crud-performance-case-study/
spring
quarkus
app with code
that deadlocked
–
1.75 req/s
initial test
with correct code
338 req/s
390 req/s
with faster disk
[not measured]
25,599 req/s
Slide 165
Slide 165 text
https://quarkus.io/blog/reactive-crud-performance-case-study/
spring
quarkus
app with code
that deadlocked
–
1.75 req/s
initial test
with correct code
338 req/s
390 req/s
with faster disk
[not measured]
25,599 req/s
Slide 166
Slide 166 text
beware the mcnamara fallacy
Slide 167
Slide 167 text
validates measurements
check in with
the real world
guides decisions
Slide 168
Slide 168 text
No content
Slide 169
Slide 169 text
3 times denser deployments
without sacrificing availability and
response times of services
Slide 170
Slide 170 text
3 times denser deployments
without sacrificing availability and
response times of services
Slide 171
Slide 171 text
No content
Slide 172
Slide 172 text
- can we get the same answer
repeatedly?
Slide 173
Slide 173 text
- can we get the same answer
repeatedly?
- are results deterministic?
Slide 174
Slide 174 text
- is this close to real-world?
- can we get the same answer
repeatedly?
- are results deterministic?
Slide 175
Slide 175 text
- can we get the same answer
repeatedly?
- are results deterministic?
- is this close to real-world?
- is this representative of the
way applications will be run?
Slide 176
Slide 176 text
- can we get the same answer
repeatedly?
- are results deterministic?
- are we reporting useful metrics?
- is this close to real-world?
- is this representative of the
way applications will be run?
Slide 177
Slide 177 text
- can we get the same answer
repeatedly?
- are results deterministic?
- are we reporting useful metrics?
- does this help us make a
decision?
- is this close to real-world?
- is this representative of the
way applications will be run?
Slide 178
Slide 178 text
- can we get the same answer
repeatedly?
- are results deterministic?
- are we reporting useful metrics?
- does this help us make a
decision?
- is it answering a question we
actually care about?
- is this close to real-world?
- is this representative of the
way applications will be run?
Slide 179
Slide 179 text
it’s easy to make all three worse
Slide 180
Slide 180 text
for improvements,
choose one :(
Slide 181
Slide 181 text
No content
Slide 182
Slide 182 text
- evaluating before-andafter of performance
tweaks
- comparing frameworks
against each other
Slide 183
Slide 183 text
- evaluating before-andafter of performance
tweaks
- comparing frameworks
against each other
- capacity planning
- cost estimation
- carbon footprint estimation
Slide 184
Slide 184 text
- evaluating before-andafter of performance
tweaks
- comparing frameworks
against each other
- answering questions
- making decisions
- capacity planning
- cost estimation
- carbon footprint estimation
Slide 185
Slide 185 text
the distillate
Slide 186
Slide 186 text
- measuring what you want to measure is hard
the distillate
Slide 187
Slide 187 text
- measuring what you want to measure is hard
- aim for reproducibility
the distillate
Slide 188
Slide 188 text
- measuring what you want to measure is hard
- aim for reproducibility
- isolate application on its own hardware
the distillate
Slide 189
Slide 189 text
- measuring what you want to measure is hard
- aim for reproducibility
- isolate application on its own hardware
- disable turbo-boost
the distillate
Slide 190
Slide 190 text
- measuring what you want to measure is hard
- aim for reproducibility
- isolate application on its own hardware
- disable turbo-boost
- aim for realism
the distillate
Slide 191
Slide 191 text
- measuring what you want to measure is hard
- aim for reproducibility
- isolate application on its own hardware
- disable turbo-boost
- aim for realism
- prod-like hardware, app, and data
the distillate
Slide 192
Slide 192 text
- measuring what you want to measure is hard
- aim for reproducibility
- isolate application on its own hardware
- disable turbo-boost
- aim for realism
- prod-like hardware, app, and data
- aim for relevance
the distillate
Slide 193
Slide 193 text
- measuring what you want to measure is hard
- aim for reproducibility
- isolate application on its own hardware
- disable turbo-boost
- aim for realism
- prod-like hardware, app, and data
- aim for relevance
- you (often) are not measuring what you think you’re
measuring (yanmwytam)
the distillate
Slide 194
Slide 194 text
- measuring what you want to measure is hard
- aim for reproducibility
- isolate application on its own hardware
- disable turbo-boost
- aim for realism
- prod-like hardware, app, and data
- aim for relevance
- you (often) are not measuring what you think you’re
measuring (yanmwytam)
- similar answers for different things usually mean you only
measured a sneaky bottleneck
the distillate
Slide 195
Slide 195 text
- measuring what you want to measure is hard
- aim for reproducibility
- isolate application on its own hardware
- disable turbo-boost
- aim for realism
- prod-like hardware, app, and data
- aim for relevance
- you (often) are not measuring what you think you’re
the distillate
measuring (yanmwytam)
- similar answers for different things usually mean you only
measured a sneaky bottleneck
- vary one thing at a time
Slide 196
Slide 196 text
- measuring what you want to measure is hard
- aim for reproducibility
- isolate application on its own hardware
- disable turbo-boost
- aim for realism
- prod-like hardware, app, and data
- aim for relevance
- you (often) are not measuring what you think you’re
the distillate
measuring (yanmwytam)
- similar answers for different things usually mean you only
measured a sneaky bottleneck
- vary one thing at a time
- validate measurements by active benchmarking
Slide 197
Slide 197 text
- measuring what you want to measure is hard
- aim for reproducibility
- isolate application on its own hardware
- disable turbo-boost
- aim for realism
- prod-like hardware, app, and data
- aim for relevance
- you (often) are not measuring what you think you’re
the distillate
measuring (yanmwytam)
- similar answers for different things usually mean you only
measured a sneaky bottleneck
- vary one thing at a time
- validate measurements by active benchmarking
- measure a metric your boss’s boss cares about
Slide 198
Slide 198 text
https://hollycummins.com/when-benchmarks-go-bad-devoxx-be/
(and yes, Holly drew the slides: https://hollycummins.com/how-to-draw-part-i/)
Slide 199
Slide 199 text
https://hollycummins.com/when-benchmarks-go-bad-devoxx-be/
(and yes, Holly drew the slides: https://hollycummins.com/how-to-draw-part-i/)