eBPF in Microservices Observability at eBPF Day

Transcript

1. @rakyll
   eBPF in Microservices
   Observability
   Jaana Dogan
   Principal Engineer, AWS
   jbd@amazon.com
   

   View Slide

2. @rakyll
   About me
   ● Not a Linux developer.
   ● Working on monitoring, observability and
   performance.
   ● Multi-tenancy and microservices focus.
   

   View Slide

3. @rakyll
   

   View Slide

4. @rakyll
   How does eBPF work?
   process
   JIT compiler
   Verifier Sockets
   TCP/IP
   BPF
   Maps
   code
   (accessible from the
   user space)
   

   View Slide

5. @rakyll
   Where can eBPF hook into?
   - Kernel and user functions
   - System calls
   - Network events
   - Kernel tracepoints
   

   View Slide

6. @rakyll
   Challenges in
   microservices
   

   View Slide

7. @rakyll
   Challenges in microservices
   We don’t just monitor VMs or processes.
   We monitor critical paths.
   

   View Slide

8. @rakyll
   What’s next?
   service
   service
   database storage
   service
   

   View Slide

9. @rakyll
   What’s next?
   service
   service
   database storage
   service
   

   View Slide

10. @rakyll
    Challenges in microservices
    Context matters.
    Downstream stack don’t have context.
    

    View Slide

11. @rakyll
    What’s next?
    process
    Linux kernel
    process process
    M:N Problem
    

    View Slide

12. @rakyll
    What’s next?
    process
    Linux kernel
    process process
    RPCs
    M:N Problem
    

    View Slide

13. @rakyll
    What’s next?
    process
    Linux kernel
    process process
    RPCs
    container container
    M:N Problem
    

    View Slide

14. @rakyll
    What’s next?
    process
    Linux kernel
    process process
    RPCs
    container container
    Kubernetes pod,
    ECS task
    M:N Problem
    

    View Slide

15. @rakyll
    Challenges in microservices
    First, we debug the path of the request.
    We debug functions or syscalls secondarily.
    

    View Slide

16. @rakyll
    Challenges in microservices
    Too much data.
    Need runtime controls to modify the collection.
    

    View Slide

17. @rakyll
    Challenges in microservices
    Instrumentation is a two-year roadmap.
    Data is not consistent.
    

    View Slide

18. @rakyll
    Recap
    Out of the box instrumentation is critical.
    Networking observability is essential.
    Extensibility in runtime is needed.
    Decoration and enrichment is needed.
    

    View Slide

19. @rakyll
    How does eBPF help?
    

    View Slide

20. @rakyll
    Network Diagnostics
    TCP, UDP, HTTP, gRPC metrics
    Inspect protocols (MySQL, Postgres, ...)
    

    View Slide

21. @rakyll
    Network Diagnostics
    TCP, UDP, HTTP, gRPC metrics
    Inspect protocols (MySQL, Postgres, ...)
    

    View Slide

22. @rakyll
    Service Maps
    

    View Slide

23. @rakyll
    Distributed Traces
    Automatically create trace span if a trace header is present.
    Your job is to generate and propagate the header.
    GET /users HTTP/1.1
    Host: users.service
    Accept-Encoding: gzip, deflate
    Connection: Keep-Alive
    Traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
    

    View Slide

24. @rakyll
    Continuous Profiling
    

    View Slide

25. @rakyll
    Extensibility
    

    View Slide

26. @rakyll
    Decorating with Context
    eBPF agent process
    JIT compiler
    Verifier Sockets
    TCP/IP
    BPF
    Maps
    API Server
    

    View Slide

27. @rakyll
    Decorating with Context
    

    View Slide

28. @rakyll
    Several projects...
    - Cillium/Hubble
    - Pixie
    - Flowmill
    - Prodfiler
    - Parca
    

    View Slide

29. @rakyll
    What’s next?
    - High level language?
    - More platforms supporting eBPF?
    - Reusable eBPF event processors?
    - Signed programs?
    

    View Slide

30. @rakyll
    Thank you
    Jaana Dogan
    jbd@amazon.com
    

    View Slide

31. @rakyll
    After party?
    Pixie KubeCon Happy Hour hosted by New Relic
    Tomorrow, 8:30 PM PDT
    RSVP: pixiehh.eventbrite.com
    

    View Slide

32. @rakyll
    Runtime Extensibility
    eBPF agent process
    JIT compiler
    Verifier Sockets
    TCP/IP
    BPF
    Maps
    code
    

    View Slide