eBPF in Microservices Observability

@rakyll eBPF in Microservices Observability Jaana Dogan Principal Engineer, AWS
jbd@amazon.com

@rakyll About me • Not a Linux developer. • Working
on monitoring, observability and performance. • Multi-tenancy and microservices focus.

@rakyll

@rakyll How does eBPF work? process JIT compiler Verifier Sockets
TCP/IP BPF Maps code (accessible from the user space)

@rakyll Where can eBPF hook into? - Kernel and user
functions - System calls - Network events - Kernel tracepoints

@rakyll Challenges in microservices

@rakyll Challenges in microservices We don’t just monitor VMs or
processes. We monitor critical paths.

@rakyll What’s next? service service database storage service

@rakyll Challenges in microservices Context matters. Downstream stack don’t have
context.

@rakyll What’s next? process Linux kernel process process M:N Problem

@rakyll What’s next? process Linux kernel process process RPCs M:N
Problem

@rakyll What’s next? process Linux kernel process process RPCs container
container M:N Problem

@rakyll What’s next? process Linux kernel process process RPCs container
container Kubernetes pod, ECS task M:N Problem

@rakyll Challenges in microservices We initially debug RPCs. We debug
functions or syscalls secondarily.

@rakyll Challenges in microservices Too much data. Need runtime controls
to modify the collection.

@rakyll Challenges in microservices Instrumentation is a two-year roadmap. Data
is not consistent.

@rakyll Networking observability is core. Out of the box instrumentation
is essential. Extensibility in runtime is critical. Decoration and enrichment is needed.

@rakyll How does eBPF help?

@rakyll Network Diagnostics TCP, UDP, HTTP, gRPC metrics Inspect protocols
(MySQL, Postgres, ...)

@rakyll Service Maps

@rakyll Distributed Traces Automatically create request span if a trace
header is present. GET /users HTTP/1.1 Host: users.service Accept-Encoding: gzip, deflate Connection: Keep-Alive Traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

@rakyll Fleet-wide Profiling context kernel

@rakyll Decorating with Context eBPF agent process JIT compiler Verifier
Sockets TCP/IP BPF Maps API Server

@rakyll Runtime Extensibility eBPF agent process JIT compiler Verifier Sockets
TCP/IP BPF Maps code

@rakyll Examples - Cillium/Hubble - Pixie - Flowmill

@rakyll What’s next? - High level language to write probes.
- Make eBPF agents widely available. - More platforms supporting eBPF. - Reusable eBPF event processing.

@rakyll Thank you Jaana Dogan jbd@amazon.com

eBPF in Microservices Observability

eBPF in Microservices Observability

JBD

More Decks by JBD

Other Decks in Programming

Featured

Transcript

@rakyll eBPF in Microservices Observability Jaana Dogan Principal Engineer, AWS

@rakyll About me • Not a Linux developer. • Working

@rakyll

@rakyll How does eBPF work? process JIT compiler Verifier Sockets

@rakyll Where can eBPF hook into? - Kernel and user

@rakyll Challenges in microservices

@rakyll Challenges in microservices We don’t just monitor VMs or

@rakyll What’s next? service service database storage service

@rakyll What’s next? service service database storage service

@rakyll Challenges in microservices Context matters. Downstream stack don’t have

@rakyll What’s next? process Linux kernel process process M:N Problem

@rakyll What’s next? process Linux kernel process process RPCs M:N

@rakyll What’s next? process Linux kernel process process RPCs container

@rakyll What’s next? process Linux kernel process process RPCs container

@rakyll Challenges in microservices We initially debug RPCs. We debug

@rakyll Challenges in microservices Too much data. Need runtime controls

@rakyll Challenges in microservices Instrumentation is a two-year roadmap. Data

@rakyll Networking observability is core. Out of the box instrumentation

@rakyll How does eBPF help?

@rakyll Network Diagnostics TCP, UDP, HTTP, gRPC metrics Inspect protocols

@rakyll Service Maps

@rakyll Distributed Traces Automatically create request span if a trace

@rakyll Fleet-wide Profiling context kernel

@rakyll Decorating with Context eBPF agent process JIT compiler Verifier

@rakyll Runtime Extensibility eBPF agent process JIT compiler Verifier Sockets

@rakyll Examples - Cillium/Hubble - Pixie - Flowmill

@rakyll What’s next? - High level language to write probes.

@rakyll Thank you Jaana Dogan jbd@amazon.com