Elastic APM Java Agent

Transcript

1. 1 Felix Barnsteiner Software Engineer @Elastic Twitter: @felix_b Elastic APM Java Agent
2. 2 Agenda Elastic APM 1 How does it work? 3 Overhead/Tuning 4 Future 5 Instrumenting a Spring Boot application 2
3. 3 Full-stack monitoring with Elastic APM adds end-user experience and application-level monitoring to the stack
4. APM Architecture APM data is just another Index
5. 5 Announcement
6. 6 Java agent is now in beta!
7. 7 Supported agents - Python (https://github.com/elastic/apm-agent-python) - Node.js (https://github.com/elastic/apm-agent-nodejs) - Ruby (https://github.com/elastic/apm-agent-ruby) - JS Beta (https://github.com/elastic/apm-agent-js-base) - Go Beta (https://github.com/elastic/apm-agent-go) - Java Beta (https://github.com/elastic/apm-agent-java) All agents are Open Source and available on GitHub
8. 8 Agenda Elastic APM 1 How does it work? 3 Overhead/Tuning 4 Future 5 Instrumenting a Spring Boot application 2
9. 9 Download the agent • Download jar from Maven Central • Don't declare a dependency
10. 10 Start application with Java agent enabled java -javaagent:/path/to/elastic-apm-agent-<version>.jar \ -Delastic.apm.service_name=my-application \ -Delastic.apm.server_url=http://localhost:8200 \ -jar my-spring-boot-application.jar
11. 11 Demo!
12. 12 Supported Technologies • Java 7+ • Frameworks/APIs/Libraries ‒ Servlet API ‒ Spring Web MVC/Spring Boot ‒ JDBC • Application Servers/Servlet Containers ‒ Tomcat ‒ WildFly ‒ Jetty ‒ WebSphere ‒ Spring Boot with embedded Tomcat, Jetty and Undertow • Participate in the survey
13. 13 13
14. 14 14
15. 15 15
16. 16 Agenda Elastic APM 1 How does it work? 3 Overhead/Tuning 4 Future 5 Instrumenting a Spring Boot application 2
17. 17 Instrumenting Servlets Request Response Agent Agent Servlet. service • Java agents can modify classes when they are loaded • When Java load a Servlet class ‒ Agents adds code before and after Servlet#service ‒ Record Transactions (URL, parameters, response code) ‒ Send to APM Server
18. 18 Automatic + Manual Instrumentation • Automatically records requests ‒ HTTP traffic for Servlet API-based applications ‒ JDBC queries • Offers API for customization ‒ Application specific metadata ‒ Custom transactions ‒ OpenTracing integration The best of both worlds
19. 19 Agenda Warning: it gets a bit technical Elastic APM 1 How does it work? 3 Overhead/Tuning 4 Future 5 Instrumenting a Spring Boot application 2
20. 20 Overhead • 1 operation = recording one HTTP request and one JDBC query • Response Time Overhead ‒ > 1 µs on average (0.000001s) ‒ ~ 3.5 µs for the 99.9th percentile • Allocation Overhead ‒ ~ 75 bytes per operation ‒ It takes ~14 000 operations to allocate 1 MB • Benchmarks are Open Source, feel free to challenge me on them
21. 21 Benchmark Setup • JMH benchmarks executed on every commit • Results are indexed in Elasticsearch • Kibana dashboard to analyze results • Benchmarks are executed on a dedicated machine ‒ No noisy neighbours • Intel Core i7-7700 CPU @ 3.60GHz
22. 22 Techniques we use to minimize the overhead • Asynchronous reporting • Avoid GC pauses ‒ GC causes latency spikes ‒ Slower response times for higher percentiles ‒ Predictable overhead • Don't allocate Objects ‒ new operator is the enemy
23. 23 Profiling with JMC and JFR • Flight Recorder is built into Hotspot VM • Free in test environments -XX:+UnlockCommercialFeatures • Very low overhead, even in microbenchmarks • Track down allocations ‒ Grouped by class ‒ Stack trace for source of allocation • Eliminate allocations ‒ Recycle/reuse objects ‒ Understand the life cycle of your objects JMH tells you that you allocate a gazillion of Objects, now what?
24. 24 Record request Transaction Object Lifecycle Reporter Thread Application Threads 3 1 4 5 6 2 Empty Recorded Active Object pool Reporter Queue
25. 25 Transaction Object Lifecycle • Start transaction recording ‒ Take a pre-allocated object out of the object pool ‒ If there are no objects left, we have to resort to allocations • Agent records data from request and response objects ‒ Avoid allocations for example use StringBuilder instead of String concatenation ‒ Currently only Map entry set iterators are allocated (to capture headers and request parameters) • Put recorded transaction into queue ‒ This is a low latency queue, optimized for ITC (LMAX-Exchange/disruptor) ‒ If full, discard transaction • Reporter thread takes transaction out of the queue and batches the data 3 1 4 2
26. 26 Transaction Object Lifecycle • When batch size is reached, or after timeout, serialize transactions* ‒ Serialization with DSL JSON (https://github.com/ngs-doo/dsl-json) ‒ Zero garbage, no intermediate string representation ‒ Serializes directly into OutputStream of the HTTP request body ‒ Custom garbage free serializer for ISO timestamps ‒ JSON is GZiped • Reset the Transaction object and put it in back to object pool ‒ Transaction is fully mutable ‒ Resets all referenced objects ‒ Immutable Objects can't be reset and create garbage ‒ Favor long timestamp over immutable Instant ‒ Avoid java.util.UUID ‒ Favor primitive types over boxed (int vs Integer) 5 6 *subject to change, we are working on a more efficient streaming-like protocol
27. 27 Agenda Elastic APM 1 How does it work? 3 Overhead/Tuning 4 Future 5 Instrumenting a Spring Boot application 2
28. 28 Future • Distributed Tracing ‒ Troubleshoot latency issues and errors in (micro-) service architectures ‒ Correlates the interaction of the services by passing correlation IDs • Better integration of APM/logging/metrics • Support for more frameworks and application servers ‒ Spring Web Reactive ‒ Tell us what you want support for: participate in the survey
29. Thank you! • Documentation : https://www.elastic.co/guide/en/apm/agent/java • GitHub : https://github.com/elastic/apm-agent-java • Forums : https://discuss.elastic.co/apm • Survey : https://discuss.elastic.co/t/apm-agent-surveys/132808 • Twitter : @felix_b