Tweet
Tweet

Slide 1

Slide 1 text

Micrometer Mastery: Unleash Advanced Observability in your JVM Apps Spring I/O 2024-05-31 Jonatan Ivanov & Tommy Ludwig

Slide 2

Slide 2 text

Spring Team/Micrometer Jonatan Ivanov Tommy Ludwig

Slide 3

Slide 3 text

Agenda ● Brief recap of observability ● Micrometer Observation API ● Micrometer Docs Generator ● Demos

Slide 4

Slide 4 text

Observability basics recap

Slide 5

Slide 5 text

Recap of observability basics ● Purpose ● Metrics ● Tracing ● Out-of-the-box instrumentation for common components ● Spring Boot auto-configuration

Slide 6

Slide 6 text

What is Observability? How well we can understand the internals of a system based on its outputs (Providing meaningful information about what happens inside) (Data about your app)

Slide 7

Slide 7 text

Why do we need Observability? Today's systems are increasingly complex (cloud) (Death Star Architecture, Big Ball of Mud)

Slide 8

Slide 8 text

Environments can be chaotic You turn a knob here a little and apps are going down there We need to deal with unknown unknowns We can’t know everything Things can be perceived differently by observers Everything is broken for the users but seems ok to you Why do we need Observability?

Slide 9

Slide 9 text

Why do we need Observability? (business perspective) Reduce lost revenue from production incidents Lower mean time to recovery (MTTR) Require less specialized knowledge Shared method of investigating across system Quantify user experience Don't guess, measure!

Slide 10

Slide 10 text

Logging Metrics Distributed Tracing

Slide 11

Slide 11 text

Examples Latency Logging (What?) Processing took 140ms Metrics (Context?) P99.999: 140ms Max: 150ms Distributed Tracing (Why?) DB was slow (lot of data was requested) Error Logging (What?) Processing failed (stacktrace?) Metrics (Context?) The error rate is 0.001/sec 2 errors in the last 30 minutes Distributed Tracing (Why?) DB call failed (invalid input)

Slide 12

Slide 12 text

SLF4J with Logback comes pre-configured SLF4J (Simple Logging Façade for Java) Simple API for logging libraries Logback Natively implements the SLF4J API If you want Log4j2 instead of Logback: - spring-boot-starter-logging + spring-boot-starter-log4j2 Logging with JVM/Spring: SLF4J + Logback

Slide 13

Slide 13 text

Metrics with JVM/Spring: Micrometer Dimensional Metrics library on the JVM Like SLF4J, but for metrics API is independent of the configured metrics backend Supports many backends Comes with spring-boot-actuator Spring projects are instrumented using Micrometer Many third-party libraries use Micrometer

Slide 14

Slide 14 text

Supported metrics backends/formats/protocols Ganglia Graphite Humio InfluxDB JMX KairosDB New Relic (/actuator/metrics) OpenTSDB OTLP Prometheus SignalFx Stackdriver (GCP) StatsD Wavefront (VMware) AppOptics Atlas Azure Monitor CloudWatch (AWS) Datadog Dynatrace Elastic

Slide 15

Slide 15 text

Distributed Tracing with JVM/Spring Boot 2.x: Spring Cloud Sleuth Boot 3.x: Micrometer Tracing (Sleuth w/o Spring dependencies) Provide an abstraction layer on top of tracing libraries - Brave (OpenZipkin), “default” - OpenTelemetry (CNCF), “experimental” Instrumentation for Spring Projects, 3rd party libraries, your app Support for various backends

Slide 16

Slide 16 text

DEMO 🍵 github.com/jonatan-ivanov/teahouse

Slide 17

Slide 17 text

Tea Service 💻 Tealeaf Service Water Service Architecture Tealeaf DB Water DB

Slide 18

Slide 18 text

through traces TraceID ❮ Exemplars Tags ❯ metrics logs traces

Slide 19

Slide 19 text

Cover w/ Image So we’re done…? ● Out-of-the-box instrumentation is useful! ● But what about: ○ Custom components ○ Customization of existing instrumentation ○ Understanding how instrumentation works

Slide 20

Slide 20 text

Observation API

Slide 21

Slide 21 text

● Add logs (application logs) ● Add metrics ● Add Distributed Tracing You want to instrument your application…

Slide 22

Slide 22 text

Introducing Observation API ● New module in Micrometer 1.10 (micrometer-observation) - micrometer-core has a dependency on it ● Higher level abstraction than metrics or tracing for timing an operation ● Instrument once, configure handlers for multiple purposes (e.g.: metrics, tracing, logging) ● Already used for most instrumentation shown

Slide 23

Slide 23 text

Observation API basic usage example Observation observation = Observation.start("talk",registry); try { // TODO: scope doSomething(); // ← This is what we’re observing } catch (Exception exception) { observation.error(exception); throw exception; } finally { // TODO: attach tags (key-value) observation.stop(); }

Slide 24

Slide 24 text

ObservationHandler with Spring Boot Spring Boot auto-configures handlers for meters and tracing. Boot will also register ObservationHandler beans to the ObservationRegistry that it auto-configures. @Bean ObservationHandler myHandler() { return new MyObservationHandler(); }

Slide 25

Slide 25 text

Configuring an ObservationHandler (without Boot) ObservationRegistry registry = ObservationRegistry.create(); registry.observationConfig() .observationHandler(new MetricsHandler(...)) .observationHandler(new TracingHandler(...)) .observationHandler(new LoggingHandler(...)) .observationHandler(new AuditEventHandler(...));

Slide 26

Slide 26 text

Let’s look at some code

Slide 27

Slide 27 text

Observation Predicate and Filter ObservationPredicate ● BiPredicate: (name, context) → Boolean ● Whether an Observation is ignored (noop) ObservationFilter ● Modify the Observation.Context ● Right before ObservationHandler#onStop ● e.g. common tags (KeyValues)

Slide 28

Slide 28 text

Conventions for instrumentation ● Instrumentation by default provides a convention ○ Naming, tags (KeyValues) ● You may want to customize the convention for an instrumentation without rewriting the instrumentation ● Control the timing of changing conventions ○ Convention changes are breaking changes

Slide 29

Slide 29 text

Introducing ObservationConvention ● Instrumentation can use a default ObservationConvention while allowing users to provide a custom implementation ● Extend a default implementation or implement your own ● See, for example, Spring Framework’s docs

Slide 30

Slide 30 text

Observation.Context ● Holds the state/data of an Observation ○ e.g. request/response object ● ObservationHandler / ObservationConvention will receive it ● Mutable, you can add data to it ○ Instrumentation time ○ Pass data between handler methods

Slide 31

Slide 31 text

Let’s look at some code (again)

Slide 32

Slide 32 text

Micrometer Docs Generator

Slide 33

Slide 33 text

Documenting instrumentation ● Keeping documentation in sync with the implementation is difficult and error-prone. ● Introducing Micrometer Docs Generator ● Define an ObservationDocumentation enum for your Observation-based instrumentation and generate documentation based on it as part of the build ● Integrate it with ObservationConvention

Slide 34

Slide 34 text

Let’s look at some code (again) (again)

Slide 35

Slide 35 text

Observation API real-world usage examples ServerHttpObservationFilter (Spring MVC) DefaultServerRequestObservationConvention ServerHttpObservationDocumentation RestTemplate

Slide 36

Slide 36 text

DEMO👂🎶 github.com/jonatan-ivanov/teahouse

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

Thank you! github.com/jonatan-ivanov/teahouse (branch: 2024-springio) slack.micrometer.io