Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Future of BEAM observability

Future of BEAM observability

Łukasz Niemier

June 18, 2019

More Decks by Łukasz Niemier

Other Decks in Programming


  1. Future of BEAM

    View full-size slide

  2. • Kobil Systems GmbH

    • EEF Observability WG

    • OpenCenus/OpenTelemetry

    View full-size slide

  3. Three pillars of observability
    • Logs

    • Metrics

    • Traces

    View full-size slide

  4. Logs
    • What have happened?

    • Sent on each action (ex. request)

    • Detailed look into service

    • Always stored in full form

    • Structured data

    View full-size slide

  5. Metrics
    • Aggregated numeric values

    • Cheap on the wire

    • „Overall preview” of the platform performance

    • Sometimes sampled

    View full-size slide

  6. Traces
    • Cross-service data

    • Very heavy on the wire

    • Multiple sources

    • Often sampled

    View full-size slide

  7. Lager
    • For a long time THE Erlang logging solution

    • Heavily rely on Erlang’s parse transformators

    • Supports all 7 syslog levels

    • Very configureable

    • Built in support for log rotation

    View full-size slide

  8. Elixir’s Logger
    • Built in into Elixir

    • gen_event based

    • Textual logs only (with addition of metadata)

    • Metadata stored as keyword list

    • Heavily rely on macros

    • OOtB supports only logging to stdout

    View full-size slide

  9. logger
    • Built in into OTP’s kernel application

    • Tries to do as much work as possible within logging
    process (instead of handler)

    • Replaces old error_logger module

    • Supports textual and structured logging (+ metadata)

    View full-size slide

  10. Metrics
    • Telemetry
    • Various aggregators:

    • OpenCensus

    • Exometer/Elixometer

    • Folsom

    • Prometheus

    • Various implementations of statsd/Influx/graphite/etc.

    • Propertiary implementations (ex. NewRelic)

    View full-size slide

  11. Unified API
    • Telemetry (migrated to Erlang)

    • Telemetry.Metrics (to be migrated)
    • Telemetry.Pooler

    View full-size slide

  12. Tracing
    • Black box tracing and debugging in BEAM is top of the

    • White box tracing is still lacking

    View full-size slide

  13. „Standards”
    • OpenZipkin (Zipkin)

    • OpenTracing (Jaeger)

    • Numerous propertiary APM protocols (DataDog,
    NewRelic, etc.)

    View full-size slide

  14. BEAM projects
    • OpenCensus (to be moved to OpenTelemetry)

    • Otter (no updates in 2 years)

    • Spandex (team joined OpenCensus team)

    • Propertiary libraries: NewRelic and AppSignal

    View full-size slide

  15. Erlang Ecosystem
    Foundation Observability
    Working Group

    View full-size slide

  16. Members
    • Tristan Sloughter (OpenCensus)

    • Vince Foley (New Relic)

    • Łukasz Niemier (OpenCensus)

    • Ilya Khaprov (prometheus.erl, prometheus.ex, OpenCensus)

    • Bryan Naegele

    • Greg Mefford (Spandex)

    • Arkadiusz Gil (Telemetry)

    • Zach Daniel (Spandex)

    • Mark Allen (Lager)

    • Andrew Thompson (Lager)

    • Jose Valim

    View full-size slide

  17. Goals
    • Improve white box monitoring (when you can see the

    • Improve black box monitoring (when you do not have
    access to code)

    • Provide common interfaces for publishing metrics and
    traces from OTP and libraries without vendor lock-in

    • Cooperation with OAM providers (NewRelic, DataDog,
    etc.) to provide better integrations with BEAM

    View full-size slide

  18. Current projects
    • Migrate OpenCensus to OpenTelemetry

    • Merge Spandex with OpenTelemetry

    • Release stable version of Telemetry and Telemetry.Metrics

    • Integrate Elixir’s Logger with Erlang’s logger

    View full-size slide