Future of BEAM observability

Transcript

1. @hauleth
2. Future of BEAM observability
3. • Kobil Systems GmbH • EEF Observability WG • OpenCenus/OpenTelemetry
4. Three pillars of observability • Logs • Metrics • Traces
5. Logs • What have happened? • Sent on each action (ex. request) • Detailed look into service • Always stored in full form • Structured data
6. Metrics • Aggregated numeric values • Cheap on the wire • „Overall preview” of the platform performance • Sometimes sampled
7. Traces • Cross-service data • Very heavy on the wire • Multiple sources • Often sampled
8. BEAM
9. Lager • For a long time THE Erlang logging solution • Heavily rely on Erlang’s parse transformators • Supports all 7 syslog levels • Very configureable • Built in support for log rotation
10. Elixir’s Logger • Built in into Elixir • gen_event based • Textual logs only (with addition of metadata) • Metadata stored as keyword list • Heavily rely on macros • OOtB supports only logging to stdout
11. logger • Built in into OTP’s kernel application • Tries to do as much work as possible within logging process (instead of handler) • Replaces old error_logger module • Supports textual and structured logging (+ metadata)
12. Metrics • Telemetry • Various aggregators: • OpenCensus • Exometer/Elixometer • Folsom • Prometheus • Various implementations of statsd/Influx/graphite/etc. • Propertiary implementations (ex. NewRelic)
13. Unified API • Telemetry (migrated to Erlang) • Telemetry.Metrics (to be migrated) • Telemetry.Pooler
14. Tracing • Black box tracing and debugging in BEAM is top of the class • White box tracing is still lacking
15. „Standards” • OpenZipkin (Zipkin) • OpenTracing (Jaeger) • Numerous propertiary APM protocols (DataDog, NewRelic, etc.)
16. BEAM projects • OpenCensus (to be moved to OpenTelemetry) • Otter (no updates in 2 years) • Spandex (team joined OpenCensus team) • Propertiary libraries: NewRelic and AppSignal
17. Erlang Ecosystem Foundation Observability Working Group
18. Members • Tristan Sloughter (OpenCensus) • Vince Foley (New Relic) • Łukasz Niemier (OpenCensus) • Ilya Khaprov (prometheus.erl, prometheus.ex, OpenCensus) • Bryan Naegele • Greg Mefford (Spandex) • Arkadiusz Gil (Telemetry) • Zach Daniel (Spandex) • Mark Allen (Lager) • Andrew Thompson (Lager) • Jose Valim
19. Goals • Improve white box monitoring (when you can see the code) • Improve black box monitoring (when you do not have access to code) • Provide common interfaces for publishing metrics and traces from OTP and libraries without vendor lock-in • Cooperation with OAM providers (NewRelic, DataDog, etc.) to provide better integrations with BEAM
20. Current projects • Migrate OpenCensus to OpenTelemetry • Merge Spandex with OpenTelemetry • Release stable version of Telemetry and Telemetry.Metrics • Integrate Elixir’s Logger with Erlang’s logger