Cloud Native eBPF Instrumentation

In this talk we are going to see how in Cloud Native environments we have the common issue of having tools to instrument and comprehend the application behaviour at kernel level. To try to solve this problem I'll try to illustrate my opinions on how I used eBPF and eBPF based tools that are both the kernel and Kubernetes aware. In other words, Cloud Native.

Transcript

1. @leodido Cloud native eBPF instrumentation ☁ where do we go when we all fall in the cloud ☁ Cloud Native Rejekts - San Diego, 2019
2. @leodido Leonardo Di Donato Open Source Software Engineer Tweets at @leodido Friendly person
3. @leodido Kubernetes is hard @leodido
4. @leodido Kernel is hard @leodido
5. @leodido Accidentally… Their names both start with a K Coincidence? I don’t think so
6. @leodido Why kernel instrumentation is harder on Kubernetes? @leodido
7. @leodido Kubernetes is an abstraction layer @leodido
8. @leodido Kubernetes complexity reflects on your ability to observe what’s going on under the abstraction @leodido
9. @leodido Instrumentation for Kubernetes makes me cry @leodido
10. @leodido Instrumentation tools are not cloud-native @leodido
11. @leodido Our kubernetes clusters are made for cloud native software development. We need cloud native instrumentation tools @leodido
12. @leodido What are my options then?
13. @leodido Many options strace in code/app as having the custom code in the application itself /proc and /sys kernel modules valgrind many non cloud-native options top, htop, iotop, etc. eBPF perf
14. @leodido hard to write. hard to maintain. crazy stuff. ☠ Many options strace kernel modules valgrind many non cloud-native options top, htop, iotop, etc. eBPF perf slows down applications. makes them unstable. slows down applications. very limited. invasive. not agnostic. performance impact. in code/app as having the custom code in the application itself /proc and /sys very limited. can see everything. can also use eBPF. limited integration. trace everything. fully programmable. negligible impact. fast. lots of tools!
15. @leodido Ok, but… @leodido
16. @leodido Kubernetes is distributed @leodido
17. @leodido Tooling exists but is not aware of the abstraction @leodido
18. @leodido Tooling exists but it was made for people to use over SSH @leodido
19. @leodido Kubernetes SSH is the kubectl @leodido
20. @leodido Kubernetes SSH is the kubectl kube-cattle @leodido
21. @leodido Application Kubernetes OS Kernel Hardware Abstraction @leodido
22. @leodido Application Kubernetes OS Kernel Hardware Abstraction @leodido The interesting stuff is here.
23. @leodido Application Kubernetes OS Kernel Hardware Abstraction @leodido Knows about the whole thing...
24. @leodido Application Kubernetes OS Kernel Hardware Abstraction @leodido You can ask everything at this level using an eBPF program!
25. @leodido How to integrate eBPF in cloud native infrastructures? They want to be together, we need to help them.
26. @leodido eBPF in a Pod
27. @leodido eBPF using a CRD
28. @leodido eBPF with the kubectl
29. @leodido eBPF in a Pod Pros: ★ Very customizable ★ Easy deployment ★ No installations Cons: ★ YAML boilerplate Easy peasy lemon squeezy
30. @leodido eBPF in a Pod
31. @leodido A Go string constant containing C code! eBPF in a Pod
32. @leodido eBPF in a Pod
33. @leodido The C code eBPF in a Pod
34. @leodido eBPF in a Pod
35. @leodido This image uses a compiled version of our BPF loader as entrypoint. eBPF in a Pod
36. @leodido eBPF in a Pod kubectl apply -f https://raw.githubusercontent.com/bpftools/prometheus-ebpf-example/master/daemonset.yaml kubectl get pods -n bpf-stuff kubectl port-forward daemonset/bpf-program -n bpf-stuff 8080:8080
37. @leodido eBPF in a Pod curl http://127.0.0.1:8080/metrics
38. @leodido Full example repository on GitHub github.com/bpftools/prometheus-ebpf-example Other Go + eBPF examples on GitHub github.com/leodido/go-ebpf-examples eBPF in a Pod
39. @leodido eBPF using a CRD Pros: ★ No boilerplate ★ Easy to use ★ A pod on every node ★ Automagically expose a Prometheus endpoint for every map you create Cons: ★ Deploy the controller ★ Not very extensible I’m that Kind of person.
40. @leodido eBPF using a CRD
41. @leodido eBPF using a CRD clang -O2 -target bpf -c pkts.c -o pkts.o kubectl create configmap --from-file pkts.o pkts-config -o yaml --dry-run >> “pkts.yaml”
42. @leodido eBPF using a CRD
43. @leodido eBPF using a CRD base64 ELF ‍♂
44. @leodido eBPF using a CRD
45. @leodido eBPF using a CRD Comes from github.com/bpftools/kube-bpf
46. @leodido eBPF using a CRD Gets the ELF from the ConfigMap
47. @leodido eBPF using a CRD
48. @leodido eBPF using a CRD github.com/bpftools/kube-bpf
49. @leodido eBPF with the kubectl Pros: ★ Uses bpftrace (DSL) ★ Very powerful ★ Unix philosophy Cons: ★ Does what bpftrace can do ★ No custom logic Like DTrace but for Kubernetes
50. @leodido eBPF in the kubectl Every time the open syscall is executed print the opened file name
51. @leodido eBPF in the kubectl Only on this specific node
52. @leodido eBPF in the kubectl
53. @leodido eBPF in the kubectl
54. @leodido Every time the function is executed print the return value eBPF in the kubectl
55. @leodido eBPF in the kubectl Only on this specific pod
56. @leodido eBPF in the kubectl github.com/iovisor/kubectl-trace
57. @leodido Instrumentation is hard On kubernetes it’s even harder eBPF is here to help Cloud native tools are already available
58. @leodido Kubernetes + eBPF links for y’all • https://github.com/bpftools/kube-bpf • https://github.com/iovisor/kubectl-trace • https://github.com/falcosecurity/falco • https://github.com/draios/sysdig • https://github.com/bpftools/linux-observability-with-bpf
59. @leodido Linux Observability with BPF • By my friend Lorenzo Fontana • There’s serious stuff there inside • Complimentary to this talk • go get your copy setns.run/bpf-book Free copy courtesy of Sysdig
60. @leodido Leonardo Di Donato Open Source Software Engineer Tweets at @leodido Friendly person