eBPF in Microservices Observability

Transcript

1. @rakyll eBPF in Microservices Observability Jaana Dogan Principal Engineer, AWS jbd@amazon.com
2. @rakyll About me • Not a Linux developer. • Working on monitoring, observability and performance. • Multi-tenancy and microservices focus.
3. @rakyll
4. @rakyll How does eBPF work? process JIT compiler Verifier Sockets TCP/IP BPF Maps code (accessible from the user space)
5. @rakyll Where can eBPF hook into? - Kernel and user functions - System calls - Network events - Kernel tracepoints
6. @rakyll Challenges in microservices
7. @rakyll Challenges in microservices We don’t just monitor VMs or processes. We monitor critical paths.
8. @rakyll What’s next? service service database storage service
9. @rakyll What’s next? service service database storage service
10. @rakyll Challenges in microservices Context matters. Downstream stack don’t have context.
11. @rakyll What’s next? process Linux kernel process process M:N Problem
12. @rakyll What’s next? process Linux kernel process process RPCs M:N Problem
13. @rakyll What’s next? process Linux kernel process process RPCs container container M:N Problem
14. @rakyll What’s next? process Linux kernel process process RPCs container container Kubernetes pod, ECS task M:N Problem
15. @rakyll Challenges in microservices We initially debug RPCs. We debug functions or syscalls secondarily.
16. @rakyll Challenges in microservices Too much data. Need runtime controls to modify the collection.
17. @rakyll Challenges in microservices Instrumentation is a two-year roadmap. Data is not consistent.
18. @rakyll Networking observability is core. Out of the box instrumentation is essential. Extensibility in runtime is critical. Decoration and enrichment is needed.
19. @rakyll How does eBPF help?
20. @rakyll Network Diagnostics TCP, UDP, HTTP, gRPC metrics Inspect protocols (MySQL, Postgres, ...)
21. @rakyll Service Maps
22. @rakyll Distributed Traces Automatically create request span if a trace header is present. GET /users HTTP/1.1 Host: users.service Accept-Encoding: gzip, deflate Connection: Keep-Alive Traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
23. @rakyll Fleet-wide Profiling context kernel
24. @rakyll Decorating with Context eBPF agent process JIT compiler Verifier Sockets TCP/IP BPF Maps API Server
25. @rakyll Runtime Extensibility eBPF agent process JIT compiler Verifier Sockets TCP/IP BPF Maps code
26. @rakyll Examples - Cillium/Hubble - Pixie - Flowmill
27. @rakyll What’s next? - High level language to write probes. - Make eBPF agents widely available. - More platforms supporting eBPF. - Reusable eBPF event processing.
28. @rakyll Thank you Jaana Dogan jbd@amazon.com