Slide 1

Slide 1 text

Jonatan Ivanov 2024-02-23 Observability for Modern JVM Applications

Slide 2

Slide 2 text

About Me - Spring Team - Micrometer - Spring Cloud, Spring Boot - Spring Observability Team - Seattle Java User Group - develotters.com - @jonatan_ivanov

Slide 3

Slide 3 text

Gauge the audience ● Observability in production? ● Spring Boot 3? ● Micrometer? ● Prometheus ● OpenTelemetry? How many people are using:

Slide 4

Slide 4 text

What is Observability?

Slide 5

Slide 5 text

What is Observability? How well we can understand the internals of a system based on its outputs (Providing meaningful information about what happens inside) (Data about your app)

Slide 6

Slide 6 text

Why do we need Observability?

Slide 7

Slide 7 text

Why do we need Observability? Today's systems are increasingly complex (cloud) (Death Star Architecture, Big Ball of Mud)

Slide 8

Slide 8 text

Environments can be chaotic You turn a knob here a little and apps are going down there We need to deal with unknown unknowns We can’t know everything Things can be perceived differently by observers Everything is broken for the users but seems ok to you Why do we need Observability?

Slide 9

Slide 9 text

Why do we need Observability? (business perspective) Reduce lost revenue from production incidents Lower mean time to recovery (MTTR) Require less specialized knowledge Shared method of investigating across system Quantify user experience Don't guess, measure!

Slide 10

Slide 10 text

Logging Metrics Distributed Tracing

Slide 11

Slide 11 text

Logging What happened (why)? Emitting events Metrics What is the context? Aggregating data Distributed Tracing Why happened? Recording causal ordering of events Logging - Metrics - Distributed Tracing

Slide 12

Slide 12 text

Examples Latency Logging Processing took 140ms Metrics P99.999: 140ms Max: 150 ms Distributed Tracing DB was slow (lot of data was requested) Error Logging Processing failed (stacktrace?) Metrics The error rate is 0.001/sec 2 errors in the last 30 minutes Distributed Tracing DB call failed (invalid input)

Slide 13

Slide 13 text

DEMO github.com/jonatan-ivanov/teahouse

Slide 14

Slide 14 text

Tea Service 💻 Tealeaf Service Water Service Architecture Tealeaf DB Water DB

Slide 15

Slide 15 text

spring-boot-starter-web spring-boot-starter-data-jpa spring-cloud-starter-openfeign spring-boot-starter-actuator (micrometer-observation) micrometer-registry-prometheus micrometer-tracing-bridge-brave + zipkin-reporter-brave net.ttddyy.observation:datasource-micrometer-spring-boot

Slide 16

Slide 16 text

Let’s make some tea! 🍵

Slide 17

Slide 17 text

through traces TraceID ❮ Exemplars Tags ❯ metrics logs traces

Slide 18

Slide 18 text

Logging With JVM/Spring

Slide 19

Slide 19 text

SLF4J with Logback comes pre-configured SLF4J (Simple Logging Façade for Java) Simple API for logging libraries Logback Natively implements the SLF4J API If you want Log4j2 instead of Logback: - spring-boot-starter-logging + spring-boot-starter-log4j2 Logging with JVM/Spring: SLF4J + Logback

Slide 20

Slide 20 text

Metrics With JVM/Spring

Slide 21

Slide 21 text

Metrics with JVM/Spring: Micrometer Dimensional Metrics library on the JVM Like SLF4J, but for metrics API is independent of the configured metrics backend Supports many backends Comes with spring-boot-actuator Spring projects are instrumented using Micrometer Many third-party libraries use Micrometer

Slide 22

Slide 22 text

Supported metrics backends/formats/protocols Ganglia Graphite Humio InfluxDB JMX KairosDB New Relic (/actuator/metrics) OpenTSDB OTLP Prometheus SignalFx Stackdriver (GCP) StatsD Wavefront (VMware) AppOptics Atlas Azure Monitor CloudWatch (AWS) Datadog Dynatrace Elastic

Slide 23

Slide 23 text

Tracing With JVM/Spring

Slide 24

Slide 24 text

Distributed Tracing with JVM/Spring Boot 2.x: Spring Cloud Sleuth Boot 3.x: Micrometer Tracing (Sleuth w/o Spring dependencies) Provide an abstraction layer on top of tracing libraries - Brave (OpenZipkin), default - OpenTelemetry (CNCF), experimental Instrumentation for Spring Projects, 3rd party libraries, your app Support for various backends

Slide 25

Slide 25 text

Observation API

Slide 26

Slide 26 text

● Add logs (application logs) ● Add metrics ○ Increment Counters ○ Start/Stop Timers ● Add Distributed Tracing ○ Start/Stop Spans ○ Log Correlation ○ Context Propagation You want to instrument your application…

Slide 27

Slide 27 text

Observation API (since Micrometer 1.10) Observation observation = Observation.start("talk",registry); try { // TODO: scope Thread.sleep(1000); } catch (Exception exception) { observation.error(exception); throw exception; } finally { // TODO: attach tags (key-value) observation.stop(); }

Slide 28

Slide 28 text

Observation API (since Micrometer 1.10) ObservationRegistry registry = ObservationRegistry.create(); registry.observationConfig() .observationHandler(new MeterHandler(...)) .observationHandler(new TracingHandler(...)) .observationHandler(new LoggingHandler(...)) .observationHandler(new AuditEventHandler(...));

Slide 29

Slide 29 text

Observation API (since Micrometer 1.10) Observation.createNotStarted("talk",registry) .lowCardinalityKeyValue("event", "ConFoo") .highCardinalityKeyValue("uid", userId) .observe(this::talk); @Observed

Slide 30

Slide 30 text

Recent changes, What’s next? Improved Exemplars support Observability improvements in the Spring portfolio Better Context Propagation Automatic Log Correlation Additional instrumentations (@Scheduled, R2DBC, JMS, etc.) New Docs site Investigating Virtual Treads / Project Loom (?) Investigating Coordinated Restore at Checkpoint (?)

Slide 31

Slide 31 text

Thank you! @jonatan_ivanov develotters.com github.com/jonatan-ivanov/teahouse slack.micrometer.io