$30 off During Our Annual Pro Sale. View Details »

eBPF in Microservices Observability at eBPF Day

JBD
October 11, 2021

eBPF in Microservices Observability at eBPF Day

JBD

October 11, 2021
Tweet

More Decks by JBD

Other Decks in Programming

Transcript

  1. @rakyll
    eBPF in Microservices
    Observability
    Jaana Dogan
    Principal Engineer, AWS
    [email protected]

    View Slide

  2. @rakyll
    About me
    ● Not a Linux developer.
    ● Working on monitoring, observability and
    performance.
    ● Multi-tenancy and microservices focus.

    View Slide

  3. @rakyll

    View Slide

  4. @rakyll
    How does eBPF work?
    process
    JIT compiler
    Verifier Sockets
    TCP/IP
    BPF
    Maps
    code
    (accessible from the
    user space)

    View Slide

  5. @rakyll
    Where can eBPF hook into?
    - Kernel and user functions
    - System calls
    - Network events
    - Kernel tracepoints

    View Slide

  6. @rakyll
    Challenges in
    microservices

    View Slide

  7. @rakyll
    Challenges in microservices
    We don’t just monitor VMs or processes.
    We monitor critical paths.

    View Slide

  8. @rakyll
    What’s next?
    service
    service
    database storage
    service

    View Slide

  9. @rakyll
    What’s next?
    service
    service
    database storage
    service

    View Slide

  10. @rakyll
    Challenges in microservices
    Context matters.
    Downstream stack don’t have context.

    View Slide

  11. @rakyll
    What’s next?
    process
    Linux kernel
    process process
    M:N Problem

    View Slide

  12. @rakyll
    What’s next?
    process
    Linux kernel
    process process
    RPCs
    M:N Problem

    View Slide

  13. @rakyll
    What’s next?
    process
    Linux kernel
    process process
    RPCs
    container container
    M:N Problem

    View Slide

  14. @rakyll
    What’s next?
    process
    Linux kernel
    process process
    RPCs
    container container
    Kubernetes pod,
    ECS task
    M:N Problem

    View Slide

  15. @rakyll
    Challenges in microservices
    First, we debug the path of the request.
    We debug functions or syscalls secondarily.

    View Slide

  16. @rakyll
    Challenges in microservices
    Too much data.
    Need runtime controls to modify the collection.

    View Slide

  17. @rakyll
    Challenges in microservices
    Instrumentation is a two-year roadmap.
    Data is not consistent.

    View Slide

  18. @rakyll
    Recap
    Out of the box instrumentation is critical.
    Networking observability is essential.
    Extensibility in runtime is needed.
    Decoration and enrichment is needed.

    View Slide

  19. @rakyll
    How does eBPF help?

    View Slide

  20. @rakyll
    Network Diagnostics
    TCP, UDP, HTTP, gRPC metrics
    Inspect protocols (MySQL, Postgres, ...)

    View Slide

  21. @rakyll
    Network Diagnostics
    TCP, UDP, HTTP, gRPC metrics
    Inspect protocols (MySQL, Postgres, ...)

    View Slide

  22. @rakyll
    Service Maps

    View Slide

  23. @rakyll
    Distributed Traces
    Automatically create trace span if a trace header is present.
    Your job is to generate and propagate the header.
    GET /users HTTP/1.1
    Host: users.service
    Accept-Encoding: gzip, deflate
    Connection: Keep-Alive
    Traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

    View Slide

  24. @rakyll
    Continuous Profiling

    View Slide

  25. @rakyll
    Extensibility

    View Slide

  26. @rakyll
    Decorating with Context
    eBPF agent process
    JIT compiler
    Verifier Sockets
    TCP/IP
    BPF
    Maps
    API Server

    View Slide

  27. @rakyll
    Decorating with Context

    View Slide

  28. @rakyll
    Several projects...
    - Cillium/Hubble
    - Pixie
    - Flowmill
    - Prodfiler
    - Parca

    View Slide

  29. @rakyll
    What’s next?
    - High level language?
    - More platforms supporting eBPF?
    - Reusable eBPF event processors?
    - Signed programs?

    View Slide

  30. @rakyll
    Thank you
    Jaana Dogan
    [email protected]

    View Slide

  31. @rakyll
    After party?
    Pixie KubeCon Happy Hour hosted by New Relic
    Tomorrow, 8:30 PM PDT
    RSVP: pixiehh.eventbrite.com

    View Slide

  32. @rakyll
    Runtime Extensibility
    eBPF agent process
    JIT compiler
    Verifier Sockets
    TCP/IP
    BPF
    Maps
    code

    View Slide