2024-10-01 dev2next - Observability for Modern JVM Applications

Transcript

1. Jonatan Ivanov 2024-10-01 Observability for Modern JVM Applications
2. About Me - Spring Team - Micrometer - Spring Cloud, Spring Boot - Spring Observability Team - Seattle Java User Group - develotters.com - @jonatan_ivanov
3. Gauge the audience • Observability in production? • Spring Boot 3? • Micrometer? • Prometheus? • OpenTelemetry? How many people are using:
4. What is Observability?
5. Various Opinions 3 pillars: Logging, Metrics, Distributed Tracing 4 pillars: + Events/Lineage(?)/Context/Metadata 6 pillars: + Profiles + Exceptions Arbitrary Wide Events, Signals But what about: /health, /info, etc. Service Registry/Discoverability, API Discoverability
6. What is Observability? How well we can understand the internals of a system based on its outputs (Providing meaningful information about what happens inside) (Data about your app)
7. Why do we need Observability?
8. Why do we need Observability? Today's systems are increasingly complex (cloud) (Death Star Architecture, Big Ball of Mud)
9. Environments can be chaotic You turn a knob here a little and apps are going down there We need to deal with unknown unknowns We can’t know everything Things can be perceived differently by observers Everything is broken for the users but seems ok to you Why do we need Observability?
10. Why do we need Observability? (business perspective) Reduce lost revenue from production incidents Lower mean time to recovery (MTTR) Require less specialized knowledge Shared method of investigating across system Quantify user experience Don't guess, measure!
11. Want to improve something? • Measure it first! • Resource utilization (number of instances, cpu, ram, io, etc.)? • Throughput/latency (max.) patterns? • Deployment frequency? • Time to go live? • Time to troubleshoot/recover? • How often are you paged? Why do we need Observability? (Continuous Improvement)
12. • Chaos Engineering • Anomaly Detection • Feature flags • A/B Testing • Auto-tuning • Adaptive Apps Why do we need Observability? (Advanced Capabilities)
13. Logging Metrics Distributed Tracing
14. Logging What happened (why)? Emitting events Metrics What is the context? Aggregating data Distributed Tracing Why happened? Recording causal ordering of events Logging - Metrics - Distributed Tracing
15. Examples Latency Logging (What?) Processing took 140ms Metrics (Context?) P99.999: 140ms Max: 150ms Distributed Tracing (Why?) DB was slow (lot of data was requested) Error Logging (What?) Processing failed (stacktrace?) Metrics (Context?) The error rate is 0.001/sec 2 errors in the last 30 minutes Distributed Tracing (Why?) DB call failed (invalid input)
16. DEMO 🍵 github.com/jonatan-ivanov/teahouse
17. Tea Service 💻 Tealeaf Service Water Service Architecture Tealeaf DB Water DB
18. spring-boot-starter-web spring-boot-starter-data-jpa spring-cloud-starter-openfeign spring-boot-starter-actuator (micrometer-observation) micrometer-registry-prometheus micrometer-tracing-bridge-brave + zipkin-reporter-brave net.ttddyy.observation:datasource-micrometer-spring-boot
19. Let’s make some tea! 🍵
20. by Kenneth Kousen
23. through traces TraceID ❮ Exemplars Tags ❯ metrics logs traces
24. Logging With JVM/Spring
25. SLF4J with Logback comes pre-configured SLF4J (Simple Logging Façade for Java) Simple API for logging libraries Logback Natively implements the SLF4J API If you want Log4j2 instead of Logback: - spring-boot-starter-logging + spring-boot-starter-log4j2 Logging with JVM/Spring: SLF4J + Logback
26. Payload, Access, GC logs Payload logs: Logbook + logbook-spring-boot-starter (auto-configured) Access logs: server.tomcat.accesslog.enabled=true server.jetty.accesslog.enabled=true server.undertow.accesslog.enabled=true GC logs: JVM args
27. Metrics With JVM/Spring
28. Metrics with JVM/Spring: Micrometer Dimensional Metrics library on the JVM Like SLF4J, but for metrics API is independent of the configured metrics backend Supports many backends Comes with spring-boot-actuator Spring projects are instrumented using Micrometer Many third-party libraries use Micrometer
29. Supported metrics backends/formats/protocols Ganglia Graphite Humio InfluxDB JMX KairosDB New Relic (/actuator/metrics) OpenTSDB OTLP Prometheus SignalFx Stackdriver (GCP) StatsD Wavefront (VMware) AppOptics Atlas Azure Monitor CloudWatch (AWS) Datadog Dynatrace Elastic
30. Tracing With JVM/Spring
31. Distributed Tracing with JVM/Spring Boot 2.x: Spring Cloud Sleuth Boot 3.x: Micrometer Tracing (Sleuth w/o Spring dependencies) Provide an abstraction layer on top of tracing libraries - Brave (OpenZipkin), “default” - OpenTelemetry (CNCF), “experimental” Instrumentation for Spring Projects, 3rd party libraries, your app Support for various backends
32. Observation API
33. • Add Logs (application logs) • Add Metrics • Add Distributed Tracing You want to instrument your application…
34. Observation API basic usage example Observation observation = Observation.start("talk",registry); try { // TODO: scope doSomething(); // ← This is what we’re observing } catch (Exception exception) { observation.error(exception); throw exception; } finally { // TODO: attach tags (key-value) observation.stop(); }
35. Configuring an ObservationHandler (without Boot) ObservationRegistry registry = ObservationRegistry.create(); registry.observationConfig() .observationHandler(new MetricsHandler(...)) .observationHandler(new TracingHandler(...)) .observationHandler(new LoggingHandler(...)) .observationHandler(new AuditEventHandler(...));
36. Observation API shortcuts Observation.createNotStarted("talk",registry) .lowCardinalityKeyValue("conf", "dev2next") .highCardinalityKeyValue("uid", userId) .observe(this::talk); @Observed
37. Health Endpoint Is my app healthy (k8s probes)? Dependencies? Info Endpoint Build Info (name, version, git commit, build time): Boot 2.x Java Info (JRE/JVM name, version, vendor): Boot 2.6 OS Info (name, arch, version): Boot 2.7 Process Info (pid, owner, cpus, memory) Boot 3.3, 3.4 Dependencies (SBOM) Boot 3.3 TLS Info (subject, issuer, validity) Boot 3.4 Cloud Info (instanceId, region, account) GC Info, Timezone, Current Time, Language, Start Time, Uptime Spring Boot Actuator
38. Service Discoverability, API Discoverability How many service instances do we have? Where? (host/ip, port, instanceId, region, account) What versions are deployed? (by environment) Eureka, Spring Boot Admin How to call/use them? Spring REST Docs Spring Cloud Contract + Pact Broker Swagger / OpenAPI + ReDoc Spring HATEOAS + HAL Explorer
39. Thank you! @jonatan_ivanov develotters.com github.com/jonatan-ivanov/teahouse (branch: 2024-dev2next) slack.micrometer.io