1 Felix Barnsteiner Software Engineer @Elastic Twitter: @felix_b Elastic APM Java Agent 

2 Agenda Elastic APM 1 How does it work? 3 Overhead/Tuning 4 Future 5 Instrumenting a Spring Boot application 2 

3 Full-stack monitoring with Elastic APM adds end-user experience and application-level monitoring to the stack 

APM Architecture APM data is just another Index 

5 Announcement 

6 Java agent is now in beta! 

7 Supported agents - Python (https://github.com/elastic/apm-agent-python) - Node.js (https://github.com/elastic/apm-agent-nodejs) - Ruby (https://github.com/elastic/apm-agent-ruby) - JS Beta (https://github.com/elastic/apm-agent-js-base) - Go Beta (https://github.com/elastic/apm-agent-go) - Java Beta (https://github.com/elastic/apm-agent-java) All agents are Open Source and available on GitHub 

8 Agenda Elastic APM 1 How does it work? 3 Overhead/Tuning 4 Future 5 Instrumenting a Spring Boot application 2 

9 Download the agent • Download jar from Maven Central • Don't declare a dependency 

10 Start application with Java agent enabled java -javaagent:/path/to/elastic-apm-agent-.jar \ -Delastic.apm.service_name=my-application \ -Delastic.apm.server_url=http://localhost:8200 \ -jar my-spring-boot-application.jar 

11 Demo! 

12 Supported Technologies • Java 7+ • Frameworks/APIs/Libraries ‒ Servlet API ‒ Spring Web MVC/Spring Boot ‒ JDBC • Application Servers/Servlet Containers ‒ Tomcat ‒ WildFly ‒ Jetty ‒ WebSphere ‒ Spring Boot with embedded Tomcat, Jetty and Undertow • Participate in the survey 

13 13 

14 14 

15 15 

16 Agenda Elastic APM 1 How does it work? 3 Overhead/Tuning 4 Future 5 Instrumenting a Spring Boot application 2 

17 Instrumenting Servlets Request Response Agent Agent Servlet. service • Java agents can modify classes when they are loaded • When Java load a Servlet class ‒ Agents adds code before and after Servlet#service ‒ Record Transactions (URL, parameters, response code) ‒ Send to APM Server 

18 Automatic + Manual Instrumentation • Automatically records requests ‒ HTTP traffic for Servlet API-based applications ‒ JDBC queries • Offers API for customization ‒ Application specific metadata ‒ Custom transactions ‒ OpenTracing integration The best of both worlds 

19 Agenda Warning: it gets a bit technical Elastic APM 1 How does it work? 3 Overhead/Tuning 4 Future 5 Instrumenting a Spring Boot application 2 

20 Overhead • 1 operation = recording one HTTP request and one JDBC query • Response Time Overhead ‒ > 1 µs on average (0.000001s) ‒ ~ 3.5 µs for the 99.9th percentile • Allocation Overhead ‒ ~ 75 bytes per operation ‒ It takes ~14 000 operations to allocate 1 MB • Benchmarks are Open Source, feel free to challenge me on them 

21 Benchmark Setup • JMH benchmarks executed on every commit • Results are indexed in Elasticsearch • Kibana dashboard to analyze results • Benchmarks are executed on a dedicated machine ‒ No noisy neighbours • Intel Core i7-7700 CPU @ 3.60GHz 

22 Techniques we use to minimize the overhead • Asynchronous reporting • Avoid GC pauses ‒ GC causes latency spikes ‒ Slower response times for higher percentiles ‒ Predictable overhead • Don't allocate Objects ‒ new operator is the enemy 

23 Profiling with JMC and JFR • Flight Recorder is built into Hotspot VM • Free in test environments -XX:+UnlockCommercialFeatures • Very low overhead, even in microbenchmarks • Track down allocations ‒ Grouped by class ‒ Stack trace for source of allocation • Eliminate allocations ‒ Recycle/reuse objects ‒ Understand the life cycle of your objects JMH tells you that you allocate a gazillion of Objects, now what? 

24 Record request Transaction Object Lifecycle Reporter Thread Application Threads 3 1 4 5 6 2 Empty Recorded Active Object pool Reporter Queue 

25 Transaction Object Lifecycle • Start transaction recording ‒ Take a pre-allocated object out of the object pool ‒ If there are no objects left, we have to resort to allocations • Agent records data from request and response objects ‒ Avoid allocations for example use StringBuilder instead of String concatenation ‒ Currently only Map entry set iterators are allocated (to capture headers and request parameters) • Put recorded transaction into queue ‒ This is a low latency queue, optimized for ITC (LMAX-Exchange/disruptor) ‒ If full, discard transaction • Reporter thread takes transaction out of the queue and batches the data 3 1 4 2 

26 Transaction Object Lifecycle • When batch size is reached, or after timeout, serialize transactions* ‒ Serialization with DSL JSON (https://github.com/ngs-doo/dsl-json) ‒ Zero garbage, no intermediate string representation ‒ Serializes directly into OutputStream of the HTTP request body ‒ Custom garbage free serializer for ISO timestamps ‒ JSON is GZiped • Reset the Transaction object and put it in back to object pool ‒ Transaction is fully mutable ‒ Resets all referenced objects ‒ Immutable Objects can't be reset and create garbage ‒ Favor long timestamp over immutable Instant ‒ Avoid java.util.UUID ‒ Favor primitive types over boxed (int vs Integer) 5 6 *subject to change, we are working on a more efficient streaming-like protocol 

27 Agenda Elastic APM 1 How does it work? 3 Overhead/Tuning 4 Future 5 Instrumenting a Spring Boot application 2 

28 Future • Distributed Tracing ‒ Troubleshoot latency issues and errors in (micro-) service architectures ‒ Correlates the interaction of the services by passing correlation IDs • Better integration of APM/logging/metrics • Support for more frameworks and application servers ‒ Spring Web Reactive ‒ Tell us what you want support for: participate in the survey 

Thank you! ● Documentation : https://www.elastic.co/guide/en/apm/agent/java ● GitHub : https://github.com/elastic/apm-agent-java ● Forums : https://discuss.elastic.co/apm ● Survey : https://discuss.elastic.co/t/apm-agent-surveys/132808 ● Twitter : @felix_b