Upgrade to Pro — share decks privately, control downloads, hide ads and more …

eBPF in Microservices Observability at eBPF Day

E7526ec3e801f8ba99f6746498a154a6?s=47 JBD
October 11, 2021

eBPF in Microservices Observability at eBPF Day

E7526ec3e801f8ba99f6746498a154a6?s=128

JBD

October 11, 2021
Tweet

Transcript

  1. @rakyll eBPF in Microservices Observability Jaana Dogan Principal Engineer, AWS

    jbd@amazon.com
  2. @rakyll About me • Not a Linux developer. • Working

    on monitoring, observability and performance. • Multi-tenancy and microservices focus.
  3. @rakyll

  4. @rakyll How does eBPF work? process JIT compiler Verifier Sockets

    TCP/IP BPF Maps code (accessible from the user space)
  5. @rakyll Where can eBPF hook into? - Kernel and user

    functions - System calls - Network events - Kernel tracepoints
  6. @rakyll Challenges in microservices

  7. @rakyll Challenges in microservices We don’t just monitor VMs or

    processes. We monitor critical paths.
  8. @rakyll What’s next? service service database storage service

  9. @rakyll What’s next? service service database storage service

  10. @rakyll Challenges in microservices Context matters. Downstream stack don’t have

    context.
  11. @rakyll What’s next? process Linux kernel process process M:N Problem

  12. @rakyll What’s next? process Linux kernel process process RPCs M:N

    Problem
  13. @rakyll What’s next? process Linux kernel process process RPCs container

    container M:N Problem
  14. @rakyll What’s next? process Linux kernel process process RPCs container

    container Kubernetes pod, ECS task M:N Problem
  15. @rakyll Challenges in microservices First, we debug the path of

    the request. We debug functions or syscalls secondarily.
  16. @rakyll Challenges in microservices Too much data. Need runtime controls

    to modify the collection.
  17. @rakyll Challenges in microservices Instrumentation is a two-year roadmap. Data

    is not consistent.
  18. @rakyll Recap Out of the box instrumentation is critical. Networking

    observability is essential. Extensibility in runtime is needed. Decoration and enrichment is needed.
  19. @rakyll How does eBPF help?

  20. @rakyll Network Diagnostics TCP, UDP, HTTP, gRPC metrics Inspect protocols

    (MySQL, Postgres, ...)
  21. @rakyll Network Diagnostics TCP, UDP, HTTP, gRPC metrics Inspect protocols

    (MySQL, Postgres, ...)
  22. @rakyll Service Maps

  23. @rakyll Distributed Traces Automatically create trace span if a trace

    header is present. Your job is to generate and propagate the header. GET /users HTTP/1.1 Host: users.service Accept-Encoding: gzip, deflate Connection: Keep-Alive Traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
  24. @rakyll Continuous Profiling

  25. @rakyll Extensibility

  26. @rakyll Decorating with Context eBPF agent process JIT compiler Verifier

    Sockets TCP/IP BPF Maps API Server
  27. @rakyll Decorating with Context

  28. @rakyll Several projects... - Cillium/Hubble - Pixie - Flowmill -

    Prodfiler - Parca
  29. @rakyll What’s next? - High level language? - More platforms

    supporting eBPF? - Reusable eBPF event processors? - Signed programs?
  30. @rakyll Thank you Jaana Dogan jbd@amazon.com

  31. @rakyll After party? Pixie KubeCon Happy Hour hosted by New

    Relic Tomorrow, 8:30 PM PDT RSVP: pixiehh.eventbrite.com
  32. @rakyll Runtime Extensibility eBPF agent process JIT compiler Verifier Sockets

    TCP/IP BPF Maps code