$30 off During Our Annual Pro Sale. View Details »

eBPF in Microservices Observability

JBD
August 18, 2021

eBPF in Microservices Observability

JBD

August 18, 2021
Tweet

More Decks by JBD

Other Decks in Programming

Transcript

  1. @rakyll
    eBPF in Microservices
    Observability
    Jaana Dogan
    Principal Engineer, AWS
    [email protected]

    View Slide

  2. @rakyll
    About me
    ● Not a Linux developer.
    ● Working on monitoring, observability and
    performance.
    ● Multi-tenancy and microservices focus.

    View Slide

  3. @rakyll

    View Slide

  4. @rakyll
    How does eBPF work?
    process
    JIT compiler
    Verifier Sockets
    TCP/IP
    BPF
    Maps
    code
    (accessible from the
    user space)

    View Slide

  5. @rakyll
    Where can eBPF hook into?
    - Kernel and user functions
    - System calls
    - Network events
    - Kernel tracepoints

    View Slide

  6. @rakyll
    Challenges in
    microservices

    View Slide

  7. @rakyll
    Challenges in microservices
    We don’t just monitor VMs or processes.
    We monitor critical paths.

    View Slide

  8. @rakyll
    What’s next?
    service
    service
    database storage
    service

    View Slide

  9. @rakyll
    What’s next?
    service
    service
    database storage
    service

    View Slide

  10. @rakyll
    Challenges in microservices
    Context matters.
    Downstream stack don’t have context.

    View Slide

  11. @rakyll
    What’s next?
    process
    Linux kernel
    process process
    M:N Problem

    View Slide

  12. @rakyll
    What’s next?
    process
    Linux kernel
    process process
    RPCs
    M:N Problem

    View Slide

  13. @rakyll
    What’s next?
    process
    Linux kernel
    process process
    RPCs
    container container
    M:N Problem

    View Slide

  14. @rakyll
    What’s next?
    process
    Linux kernel
    process process
    RPCs
    container container
    Kubernetes pod,
    ECS task
    M:N Problem

    View Slide

  15. @rakyll
    Challenges in microservices
    We initially debug RPCs.
    We debug functions or syscalls secondarily.

    View Slide

  16. @rakyll
    Challenges in microservices
    Too much data.
    Need runtime controls to modify the collection.

    View Slide

  17. @rakyll
    Challenges in microservices
    Instrumentation is a two-year roadmap.
    Data is not consistent.

    View Slide

  18. @rakyll
    Networking observability is core.
    Out of the box instrumentation is essential.
    Extensibility in runtime is critical.
    Decoration and enrichment is needed.

    View Slide

  19. @rakyll
    How does eBPF help?

    View Slide

  20. @rakyll
    Network Diagnostics
    TCP, UDP, HTTP, gRPC metrics
    Inspect protocols (MySQL, Postgres, ...)

    View Slide

  21. @rakyll
    Service Maps

    View Slide

  22. @rakyll
    Distributed Traces
    Automatically create request span if a trace header is present.
    GET /users HTTP/1.1
    Host: users.service
    Accept-Encoding: gzip, deflate
    Connection: Keep-Alive
    Traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

    View Slide

  23. @rakyll
    Fleet-wide Profiling
    context
    kernel

    View Slide

  24. @rakyll
    Decorating with Context
    eBPF agent process
    JIT compiler
    Verifier Sockets
    TCP/IP
    BPF
    Maps
    API Server

    View Slide

  25. @rakyll
    Runtime Extensibility
    eBPF agent process
    JIT compiler
    Verifier Sockets
    TCP/IP
    BPF
    Maps
    code

    View Slide

  26. @rakyll
    Examples
    - Cillium/Hubble
    - Pixie
    - Flowmill

    View Slide

  27. @rakyll
    What’s next?
    - High level language to write probes.
    - Make eBPF agents widely available.
    - More platforms supporting eBPF.
    - Reusable eBPF event processing.

    View Slide

  28. @rakyll
    Thank you
    Jaana Dogan
    [email protected]

    View Slide