@rakylleBPF in MicroservicesObservabilityJaana DoganPrincipal Engineer, AWS[email protected]
View Slide
@rakyllAbout me● Not a Linux developer.● Working on monitoring, observability andperformance.● Multi-tenancy and microservices focus.
@rakyll
@rakyllHow does eBPF work?processJIT compilerVerifier SocketsTCP/IPBPFMapscode(accessible from theuser space)
@rakyllWhere can eBPF hook into?- Kernel and user functions- System calls- Network events- Kernel tracepoints
@rakyllChallenges inmicroservices
@rakyllChallenges in microservicesWe don’t just monitor VMs or processes.We monitor critical paths.
@rakyllWhat’s next?serviceservicedatabase storageservice
@rakyllChallenges in microservicesContext matters.Downstream stack don’t have context.
@rakyllWhat’s next?processLinux kernelprocess processM:N Problem
@rakyllWhat’s next?processLinux kernelprocess processRPCsM:N Problem
@rakyllWhat’s next?processLinux kernelprocess processRPCscontainer containerM:N Problem
@rakyllWhat’s next?processLinux kernelprocess processRPCscontainer containerKubernetes pod,ECS taskM:N Problem
@rakyllChallenges in microservicesWe initially debug RPCs.We debug functions or syscalls secondarily.
@rakyllChallenges in microservicesToo much data.Need runtime controls to modify the collection.
@rakyllChallenges in microservicesInstrumentation is a two-year roadmap.Data is not consistent.
@rakyllNetworking observability is core.Out of the box instrumentation is essential.Extensibility in runtime is critical.Decoration and enrichment is needed.
@rakyllHow does eBPF help?
@rakyllNetwork DiagnosticsTCP, UDP, HTTP, gRPC metricsInspect protocols (MySQL, Postgres, ...)
@rakyllService Maps
@rakyllDistributed TracesAutomatically create request span if a trace header is present.GET /users HTTP/1.1Host: users.serviceAccept-Encoding: gzip, deflateConnection: Keep-AliveTraceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
@rakyllFleet-wide Profilingcontextkernel
@rakyllDecorating with ContexteBPF agent processJIT compilerVerifier SocketsTCP/IPBPFMapsAPI Server
@rakyllRuntime ExtensibilityeBPF agent processJIT compilerVerifier SocketsTCP/IPBPFMapscode
@rakyllExamples- Cillium/Hubble- Pixie- Flowmill
@rakyllWhat’s next?- High level language to write probes.- Make eBPF agents widely available.- More platforms supporting eBPF.- Reusable eBPF event processing.
@rakyllThank youJaana Dogan[email protected]
rakyll
twada
takamin55
faithandbrave
niftycorp
okuzawats
cmnakamurashogo
commojun
aridlehoover
techouse
shlominoach
eileencodes
ddemaree
palkan
hursman
bkeepers
philnash
lara
colly
jcasabona
jlugia
jennybc