eBPF in Microservices Observability

Slide 1

Slide 1 text

@rakyll eBPF in Microservices Observability Jaana Dogan Principal Engineer, AWS [email protected]

Slide 2

Slide 2 text

@rakyll About me ● Not a Linux developer. ● Working on monitoring, observability and performance. ● Multi-tenancy and microservices focus.

Slide 3

Slide 3 text

@rakyll

Slide 4

Slide 4 text

@rakyll How does eBPF work? process JIT compiler Verifier Sockets TCP/IP BPF Maps code (accessible from the user space)

Slide 5

Slide 5 text

@rakyll Where can eBPF hook into? - Kernel and user functions - System calls - Network events - Kernel tracepoints

Slide 6

Slide 6 text

@rakyll Challenges in microservices

Slide 7

Slide 7 text

@rakyll Challenges in microservices We don’t just monitor VMs or processes. We monitor critical paths.

Slide 8

Slide 8 text

@rakyll What’s next? service service database storage service

Slide 9

Slide 9 text

@rakyll What’s next? service service database storage service

Slide 10

Slide 10 text

@rakyll Challenges in microservices Context matters. Downstream stack don’t have context.

Slide 11

Slide 11 text

@rakyll What’s next? process Linux kernel process process M:N Problem

Slide 12

Slide 12 text

@rakyll What’s next? process Linux kernel process process RPCs M:N Problem

Slide 13

Slide 13 text

@rakyll What’s next? process Linux kernel process process RPCs container container M:N Problem

Slide 14

Slide 14 text

@rakyll What’s next? process Linux kernel process process RPCs container container Kubernetes pod, ECS task M:N Problem

Slide 15

Slide 15 text

@rakyll Challenges in microservices We initially debug RPCs. We debug functions or syscalls secondarily.

Slide 16

Slide 16 text

@rakyll Challenges in microservices Too much data. Need runtime controls to modify the collection.

Slide 17

Slide 17 text

@rakyll Challenges in microservices Instrumentation is a two-year roadmap. Data is not consistent.

Slide 18

Slide 18 text

@rakyll Networking observability is core. Out of the box instrumentation is essential. Extensibility in runtime is critical. Decoration and enrichment is needed.

Slide 19

Slide 19 text

@rakyll How does eBPF help?

Slide 20

Slide 20 text

@rakyll Network Diagnostics TCP, UDP, HTTP, gRPC metrics Inspect protocols (MySQL, Postgres, ...)

Slide 21

Slide 21 text

@rakyll Service Maps

Slide 22

Slide 22 text

@rakyll Distributed Traces Automatically create request span if a trace header is present. GET /users HTTP/1.1 Host: users.service Accept-Encoding: gzip, deflate Connection: Keep-Alive Traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Slide 23

Slide 23 text

@rakyll Fleet-wide Profiling context kernel

Slide 24

Slide 24 text

@rakyll Decorating with Context eBPF agent process JIT compiler Verifier Sockets TCP/IP BPF Maps API Server

Slide 25

Slide 25 text

@rakyll Runtime Extensibility eBPF agent process JIT compiler Verifier Sockets TCP/IP BPF Maps code

Slide 26

Slide 26 text

@rakyll Examples - Cillium/Hubble - Pixie - Flowmill

Slide 27

Slide 27 text

@rakyll What’s next? - High level language to write probes. - Make eBPF agents widely available. - More platforms supporting eBPF. - Reusable eBPF event processing.

Slide 28

Slide 28 text

@rakyll Thank you Jaana Dogan [email protected]