Quarkus_OpenTelemetry_Performance_-_OTel_Java_-_Jan_2026_-_small.pdf

Transcript

1. The Performance Impact of Telemetry on Java Frameworks Bruno Baptista IBM OpenTelemetry Java Meeting January 2026
2. Thanks to Luis Barreiro, IBM Benjamin Evans, IBM Francesco Nigro, IBM
3. Summary 1. Purpose 2. A Result 3. The Context of that result 4. The runs 5. Conclusions
4. Purpose To capture the impact of observability in the user experience when using a Java Framework, like Quarkus app under high load.
5. Quarkus is a cloud-native Java framework designed to create fast startup, low memory usage application while spreading developer joy. https://quarkus.io
6. A Result… What so you expect?
8. One observation Request Count Mean (ms) 50% 90% 99% 99.90% 99.99% Micrometer (MM) ext. 1799523 0.92 0.57 0.96 11.27 34.17 59.42 MM+ OpenTelemetry (OTEL) ext. 1800825 2.30 0.68 1.44 36.61 198.09 243.97 Observability OFF 1796804 1.02 0.56 0.92 11.99 76.02 119.27 OpenTelemetry Java AGENT 1800703 2.98 0.73 1.64 92.71 171.88 195.56 • Average of 3 runs • All values are rounded and durations presented in milliseconds (ms)
9. One observation Request Count Mean (ms) 50% 90% 99% 99.90% 99.99% Micrometer (MM) ext. 1799523 0.92 0.57 0.96 11.27 34.17 59.42 MM+ OpenTelemetry (OTEL) ext. 1800825 2.30 0.68 1.44 36.61 198.09 243.97 Observability OFF 1796804 1.02 0.56 0.92 11.99 76.02 119.27 OpenTelemetry Java AGENT 1800703 2.98 0.73 1.64 92.71 171.88 195.56 • 50% percentile (median) more meaningful than the mean. Not a normal distribution! • 99.9% and 99.99% will be noisy. Higher variance between runs.
10. One observation • 1500 Req/s on POST • 6000 Req/s on GET • Quarkus version 3.24.0.CR1 • 10% OTel spans sampling rate • Stock G1 GC • No JVM arguments set.
11. The Context of that result
12. What are we doing here? • We wanted a "Real world app" • Effects of: ◦ quarkus-micrometer-opentelemetry extension ▪ Quarkus-micrometer ▪ Quarkus-opentelemetry ▪ Micrometer to OpenTelemetry bridge ◦ OpenTelemetry Java agent v2.11.0 ▪ Bytecode instrumentation ▪ Marked Stable but instrumentations are not • Comparable results
13. The App The CRUD app is a server made with: • Quarkus! • REST (Jackson) • OpenAPI • Hibernate • Panache • Hibernate validator • PostgreSQL • Entity - DTO mapping • Only imperative business logic (no reactive) No logs during execution…
14. The initial baseline constraints • We use custom hardware • 2 CPUs • 1 GB heap memory • OpenTelemetry export is performed with ◦ Default span batch size of 512 ◦ 10% sampled spans ◦ JVM defaults… Meaning G1 GC • Running as a Java process (java -jar …) • No quarkus native compilation (for now) But we did many other combinations…
15. One observation • 1500 Req/s on POST • 6000 Req/s on GET • Quarkus version 3.24.0.CR1 • 10% OTel spans sampling rate • Stock G1 GC • No JVM arguments set.
16. The runs
17. So many runs… Throughpu t (PUT - GET) req/s Quarkus version Sample ratio (1%, 10%) Export Batch size Heap memory size G1 GC G1 GC with -Xms set Parallel GC Parallel GC with -Xms 500-1500 X X X 1000-3000 X X X X 1000-4000 X X X X X X X 1500-6000 X X X X X X X 2000-8000 X X X X X X X • Batch sizes (Nr. of spans): 256, 512, 1024 • Heap (MB): 256, 364, 512, 1024, 2048, 4096 • Later runs executed for: Baseline, MM, MM+OTel and OTel Java agent. • 2 and 4 CPU… But that is story for another day.
18. Optimized app (Not in the provided order) • We removed 1 service • Removed JMS, after fixing a bug showing up under load: https://github.com/messaginghub/pooled-jms/pull/30 • We fixed 4 bugs in Quarkus • Optimized hibernate: ◦ merge->persist ◦ No Session on read ◦ Use projection to directly to get a DTO. • Streamline Connection pools quarkus.datasource.jdbc.max-size=50 # This number is a bit arbitrary and could be tuned quarkus.datasource.jdbc.detect-statement-leaks=false # small memory optimization allowed because we use Hibernate quarkus.thread-pool.max-threads=50 quarkus.thread-pool.core-threads=50 quarkus.thread-pool.keep-alive-time=1d # make sure we reuse, not recreate quarkus.vertx.event-loops-pool-size=1 # Because we use only 2 CPU
19. We went from 1000/3000 req/s with a frequent noise to 1000/4000 with consistent results
20. But then… We used CPUs from the same Non-uniform memory access (NUMA) core and went From 1000/4000 to 1500/6000!
21. But then… We started experimenting with garbage collection: • G1 • Parallel • The effects of setting the minimal heap at startup • Pre-touch … And was possible to get 2000/8000.
22. Streamlining the app allowed to double the throughput
23. Percentiles according to load • Quarkus version 3.24.0.CR1 • 10% OTel spans sampling rate • Parallel GC • Min. Heap memory set to 2GB • Base case with no Observability
24. Percentiles according to load - Baseline vs MM
25. Percentiles according to load - Baseline vs MM+OTel
26. Percentiles according to load - Baseline vs OTel Agent
27. What about the garbage collector?
28. The Garbage collector effect - Baseline • 1500 Req/s on POST • 6000 Req/s on GET • Quarkus version 3.24.0.CR1 • 10% OTel spans sampling rate • 2GB max heap • Base case with no Observability
29. The Garbage collector effect - Baseline vs MM
30. The Garbage collector effect - Baseline vs MM+OTel
31. The Garbage collector effect - Baseline vs OTel Java Agent
32. Why so much difference because of the GC? • G1 GC is optimized for predictable pause times: ◦ Memory divided in regions (young and old generation per region) ◦ Incremental and concurrent compaction on the regions • Parallel GC is optimized for throughput. ◦ Stop the world ◦ Scans all heap • However: ◦ For small heaps, < ~8GB! The scanning speed is big, outperforming G1. ◦ Probably not valid for all types of load. We plan to research that.
33. Startup Time Startup Time (ms) Time to 1st Request(ms) Total time (ms) Increased time % OFF 2256 324 2580 MM 2499 517 3017 16.92% MM+OTEL 2598 289 2887 11.90% AGENT 4329 891 5220 102.32% • 2000 Req/s on POST • 8000 Req/s on GET • Quarkus version 3.24.0.CR1 • 10% OTel spans sampling rate • 1GB max heap • JVM OPTS: -XX:+UseParallelGC -Xms1024m
34. Conclusions
35. Conclusions • Tuning the app's performance is important to notice the observability impact • The Execution context matters • Micrometer metrics + bridge performance penalty is minimal • OpenTelemetry tracing introduces a performance penalty under heavy load • To mitigate the penalty: ◦ Reduce sampling. Below 10% is recommended. ◦ For fairly constant load the Parallel GC works best ◦ Locking the minimum heap memory seems to penalize the G1 GC. • The extension overperforms the agent in most of the tested contexts… • The extension's performance is more predictable (less variance between runs) • The Java agent, on small heaps, works best with the Parallel GC on.
36. Bye! The App: https://github.com/brunobat/quarkus-ob servability-perf/tree/quarkus-3.24-otel Ask me stuff: Bruno Baptista https://www.linkedin.com/in/brunobat/