2024-10-02 dev2next - Application Observability like you've never heard before

Transcript

1. Jonatan Ivanov 2024-10-02 Application Observability like you've never heard before
2. About Me - Spring Team - Micrometer - Spring Cloud, Spring Boot - Spring Observability Team - Seattle Java User Group - develotters.com - @jonatan_ivanov
3. Gauge the audience • Observability in production? • Spring Boot 3? • Micrometer? • Prometheus? • OpenTelemetry? How many people are using:
4. What is Observability?
5. What is Observability? How well we can understand the internals of a system based on its outputs (Providing meaningful information about what happens inside) (Data about your app)
6. Why do we need Observability?
7. Why do we need Observability? Today's systems are increasingly complex (cloud) (Death Star Architecture, Big Ball of Mud)
8. Environments can be chaotic You turn a knob here a little and apps are going down there We need to deal with unknown unknowns We can’t know everything Things can be perceived differently by observers Everything is broken for the users but seems ok to you Why do we need Observability?
9. Why do we need Observability? (business perspective) Reduce lost revenue from production incidents Lower mean time to recovery (MTTR) Require less specialized knowledge Shared method of investigating across system Quantify user experience Don't guess, measure!
10. Logging Metrics Distributed Tracing
11. Logging What happened (why)? Emitting events Metrics What is the context? Aggregating data Distributed Tracing Why happened? Recording causal ordering of events Logging - Metrics - Distributed Tracing
12. Examples Latency Logging (What?) Processing took 140ms Metrics (Context?) P99.999: 140ms Max: 150ms Distributed Tracing (Why?) DB was slow (lot of data was requested) Error Logging (What?) Processing failed (stacktrace?) Metrics (Context?) The error rate is 0.001/sec 2 errors in the last 30 minutes Distributed Tracing (Why?) DB call failed (invalid input)
13. DEMO 🍵 github.com/jonatan-ivanov/teahouse
14. Tea Service 💻 Tealeaf Service Water Service Architecture Tealeaf DB Water DB
15. spring-boot-starter-web spring-boot-starter-data-jpa spring-cloud-starter-openfeign spring-boot-starter-actuator (micrometer-observation) micrometer-registry-prometheus micrometer-tracing-bridge-brave + zipkin-reporter-brave net.ttddyy.observation:datasource-micrometer-spring-boot
16. Let’s make some tea! 🍵
17. through traces TraceID ❮ Exemplars Tags ❯ metrics logs traces
18. Logging With JVM/Spring
19. SLF4J with Logback comes pre-configured SLF4J (Simple Logging Façade for Java) Simple API for logging libraries Logback Natively implements the SLF4J API If you want Log4j2 instead of Logback: - spring-boot-starter-logging + spring-boot-starter-log4j2 Logging with JVM/Spring: SLF4J + Logback
20. Metrics With JVM/Spring
21. Metrics with JVM/Spring: Micrometer Dimensional Metrics library on the JVM Like SLF4J, but for metrics API is independent of the configured metrics backend Supports many backends Comes with spring-boot-actuator Spring projects are instrumented using Micrometer Many third-party libraries use Micrometer
22. Supported metrics backends/formats/protocols Ganglia Graphite Humio InfluxDB JMX KairosDB New Relic (/actuator/metrics) OpenTSDB OTLP Prometheus SignalFx Stackdriver (GCP) StatsD Wavefront (VMware) AppOptics Atlas Azure Monitor CloudWatch (AWS) Datadog Dynatrace Elastic
23. Tracing With JVM/Spring
24. Distributed Tracing with JVM/Spring Boot 2.x: Spring Cloud Sleuth Boot 3.x: Micrometer Tracing (Sleuth w/o Spring dependencies) Provide an abstraction layer on top of tracing libraries - Brave (OpenZipkin), “default” - OpenTelemetry (CNCF), “experimental” Instrumentation for Spring Projects, 3rd party libraries, your app Support for various backends
25. Observation API
26. • Add Logs (application logs) • Add Metrics • Add Distributed Tracing You want to instrument your application…
27. Introducing Observation API • “New” module in Micrometer 1.10 (micrometer-observation) micrometer-core has a dependency on it • Higher level abstraction than metrics, tracing, etc. • Instrument once, configure handlers for multiple purposes (metrics, tracing, logging, etc.) • Already used for most instrumentation shown
28. Observation API basic usage example Observation observation = Observation.start("talk",registry); try { // TODO: scope doSomething(); // ← This is what we’re observing } catch (Exception exception) { observation.error(exception); throw exception; } finally { // TODO: attach tags (key-value) observation.stop(); }
29. Configuring an ObservationHandler (without Boot) ObservationRegistry registry = ObservationRegistry.create(); registry.observationConfig() .observationHandler(new MetricsHandler(...)) .observationHandler(new TracingHandler(...)) .observationHandler(new LoggingHandler(...)) .observationHandler(new AuditEventHandler(...));
30. ObservationHandler with Spring Boot Spring Boot auto-configures handlers for meters and tracing. Boot will also register ObservationHandler beans to the ObservationRegistry that it auto-configures. @Bean ObservationHandler<MyContext> myHandler() { return new MyObservationHandler(); }
31. Observation API shortcuts Observation.createNotStarted("talk",registry) .lowCardinalityKeyValue("conf", "dev2next") .highCardinalityKeyValue("uid", userId) .observe(this::talk); @Observed
32. Let’s code! 󰳕🔭
33. Observation.Context • Holds the state/data of an Observation ◦ e.g.: request/response object • ObservationHandler will receive it • Mutable, you can add data to it ◦ Instrumentation time ◦ Pass data between handler methods
34. Observation Predicate and Filter ObservationPredicate • BiPredicate: (name, context) → Boolean • Whether an Observation is ignored (noop) ObservationFilter • Modify the Observation.Context • Right before ObservationHandler#onStop • e.g. common tags (KeyValues)
35. Let’s look at some code
36. Conventions for instrumentation • Instrumentation by default provides a convention ◦ Naming, tags (KeyValues) • You may want to customize the convention for an instrumentation without rewriting the instrumentation • Control the timing of changing conventions ◦ Convention changes are breaking changes
37. Introducing ObservationConvention • Instrumentation can use a default ObservationConvention while allowing users to provide a custom implementation • Extend a default implementation or implement your own • See, for example, Spring Framework’s docs
38. Let’s look at some code (again)
39. Micrometer Docs Generator
40. Documenting instrumentation • Keeping documentation in sync with the implementation is difficult and error-prone. • Introducing Micrometer Docs Generator • Define an ObservationDocumentation enum for your Observation-based instrumentation and generate documentation based on it as part of the build • Integrate it with ObservationConvention
41. Let’s look at some code (again) (again)
42. Observation API real-world usage examples ServerHttpObservationFilter (Spring MVC) DefaultServerRequestObservationConvention ServerHttpObservationDocumentation RestTemplate
43. What’s ~new? • Improved Exemplars support • MeterProvider • Prometheus Java Client 1.x • New Docs site (micrometer.io) • Observability improvements in the Spring portfolio ◦ Context Propagation + Log Correlation ◦ Auto-Instrumentations, Performance • SBOM Actuator Endpoint
44. What’s ~next? • Exponential Histograms (OTLP) • TestObservationRegistry validation • ProcessInfoContributor • SslInfoContributor • SslHealthIndicator • Spring AI instrumentation
45. DEMO👂🎶 github.com/jonatan-ivanov/teahouse
46. Thank you! @jonatan_ivanov develotters.com github.com/jonatan-ivanov/teahouse (branch: 2024-dev2next) slack.micrometer.io