Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Native eBPF Instrumentation

Cloud Native eBPF Instrumentation

In this talk we are going to see how in Cloud Native environments we have the common issue of having tools to instrument and comprehend the application behaviour at kernel level. To try to solve this problem I'll try to illustrate my opinions on how I used eBPF and eBPF based tools that are both the kernel and Kubernetes aware. In other words, Cloud Native.

Leonardo Di Donato

November 17, 2019

More Decks by Leonardo Di Donato

Other Decks in Programming


  1. @leodido Cloud native eBPF instrumentation ☁ where do we go

    when we all fall in the cloud ☁ Cloud Native Rejekts - San Diego, 2019
  2. @leodido Our kubernetes clusters are made for cloud native software

    development. We need cloud native instrumentation tools @leodido
  3. @leodido Many options strace in code/app as having the custom

    code in the application itself /proc and /sys kernel modules valgrind many non cloud-native options top, htop, iotop, etc. eBPF perf
  4. @leodido hard to write. hard to maintain. crazy stuff. ☠

    Many options strace kernel modules valgrind many non cloud-native options top, htop, iotop, etc. eBPF perf slows down applications. makes them unstable. slows down applications. very limited. invasive. not agnostic. performance impact. in code/app as having the custom code in the application itself /proc and /sys very limited. can see everything. can also use eBPF. limited integration. trace everything. fully programmable. negligible impact. fast. lots of tools!
  5. @leodido Application Kubernetes OS Kernel Hardware Abstraction @leodido You can

    ask everything at this level using an eBPF program!
  6. @leodido eBPF in a Pod Pros: ★ Very customizable ★

    Easy deployment ★ No installations Cons: ★ YAML boilerplate Easy peasy lemon squeezy
  7. @leodido This image uses a compiled version of our BPF

    loader as entrypoint. eBPF in a Pod
  8. @leodido Full example repository on GitHub github.com/bpftools/prometheus-ebpf-example Other Go +

    eBPF examples on GitHub github.com/leodido/go-ebpf-examples eBPF in a Pod
  9. @leodido eBPF using a CRD Pros: ★ No boilerplate ★

    Easy to use ★ A pod on every node ★ Automagically expose a Prometheus endpoint for every map you create Cons: ★ Deploy the controller ★ Not very extensible I’m that Kind of person.
  10. @leodido eBPF using a CRD clang -O2 -target bpf -c

    pkts.c -o pkts.o kubectl create configmap --from-file pkts.o pkts-config -o yaml --dry-run >> “pkts.yaml”
  11. @leodido eBPF with the kubectl Pros: ★ Uses bpftrace (DSL)

    ★ Very powerful ★ Unix philosophy Cons: ★ Does what bpftrace can do ★ No custom logic Like DTrace but for Kubernetes
  12. @leodido eBPF in the kubectl Every time the open syscall

    is executed print the opened file name
  13. @leodido Instrumentation is hard On kubernetes it’s even harder eBPF

    is here to help Cloud native tools are already available
  14. @leodido Kubernetes + eBPF links for y’all • https://github.com/bpftools/kube-bpf •

    https://github.com/iovisor/kubectl-trace • https://github.com/falcosecurity/falco • https://github.com/draios/sysdig • https://github.com/bpftools/linux-observability-with-bpf
  15. @leodido Linux Observability with BPF • By my friend Lorenzo

    Fontana • There’s serious stuff there inside • Complimentary to this talk • go get your copy setns.run/bpf-book Free copy courtesy of Sysdig