Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Native eBPF Instrumentation

Cloud Native eBPF Instrumentation

In this talk we are going to see how in Cloud Native environments we have the common issue of having tools to instrument and comprehend the application behaviour at kernel level. To try to solve this problem I'll try to illustrate my opinions on how I used eBPF and eBPF based tools that are both the kernel and Kubernetes aware. In other words, Cloud Native.

Leonardo Di Donato

November 17, 2019
Tweet

More Decks by Leonardo Di Donato

Other Decks in Programming

Transcript

  1. @leodido Cloud native eBPF instrumentation ☁ where do we go

    when we all fall in the cloud ☁ Cloud Native Rejekts - San Diego, 2019
  2. @leodido Leonardo Di Donato Open Source Software Engineer Tweets at

    @leodido Friendly person
  3. @leodido Kubernetes is hard @leodido

  4. @leodido Kernel is hard @leodido

  5. @leodido Accidentally… Their names both start with a K Coincidence?

    I don’t think so
  6. @leodido Why kernel instrumentation is harder on Kubernetes? @leodido

  7. @leodido Kubernetes is an abstraction layer @leodido

  8. @leodido Kubernetes complexity reflects on your ability to observe what’s

    going on under the abstraction @leodido
  9. @leodido Instrumentation for Kubernetes makes me cry @leodido

  10. @leodido Instrumentation tools are not cloud-native @leodido

  11. @leodido Our kubernetes clusters are made for cloud native software

    development. We need cloud native instrumentation tools @leodido
  12. @leodido What are my options then?

  13. @leodido Many options strace in code/app as having the custom

    code in the application itself /proc and /sys kernel modules valgrind many non cloud-native options top, htop, iotop, etc. eBPF perf
  14. @leodido hard to write. hard to maintain. crazy stuff. ☠

    Many options strace kernel modules valgrind many non cloud-native options top, htop, iotop, etc. eBPF perf slows down applications. makes them unstable. slows down applications. very limited. invasive. not agnostic. performance impact. in code/app as having the custom code in the application itself /proc and /sys very limited. can see everything. can also use eBPF. limited integration. trace everything. fully programmable. negligible impact. fast. lots of tools!
  15. @leodido Ok, but… @leodido

  16. @leodido Kubernetes is distributed @leodido

  17. @leodido Tooling exists but is not aware of the abstraction

    @leodido
  18. @leodido Tooling exists but it was made for people to

    use over SSH @leodido
  19. @leodido Kubernetes SSH is the kubectl @leodido

  20. @leodido Kubernetes SSH is the kubectl kube-cattle @leodido

  21. @leodido Application Kubernetes OS Kernel Hardware Abstraction @leodido

  22. @leodido Application Kubernetes OS Kernel Hardware Abstraction @leodido The interesting

    stuff is here.
  23. @leodido Application Kubernetes OS Kernel Hardware Abstraction @leodido Knows about

    the whole thing...
  24. @leodido Application Kubernetes OS Kernel Hardware Abstraction @leodido You can

    ask everything at this level using an eBPF program!
  25. @leodido How to integrate eBPF in cloud native infrastructures? They

    want to be together, we need to help them.
  26. @leodido eBPF in a Pod

  27. @leodido eBPF using a CRD

  28. @leodido eBPF with the kubectl

  29. @leodido eBPF in a Pod Pros: ★ Very customizable ★

    Easy deployment ★ No installations Cons: ★ YAML boilerplate Easy peasy lemon squeezy
  30. @leodido eBPF in a Pod

  31. @leodido A Go string constant containing C code! eBPF in

    a Pod
  32. @leodido eBPF in a Pod

  33. @leodido The C code eBPF in a Pod

  34. @leodido eBPF in a Pod

  35. @leodido This image uses a compiled version of our BPF

    loader as entrypoint. eBPF in a Pod
  36. @leodido eBPF in a Pod kubectl apply -f https://raw.githubusercontent.com/bpftools/prometheus-ebpf-example/master/daemonset.yaml kubectl

    get pods -n bpf-stuff kubectl port-forward daemonset/bpf-program -n bpf-stuff 8080:8080
  37. @leodido eBPF in a Pod curl http://127.0.0.1:8080/metrics

  38. @leodido Full example repository on GitHub github.com/bpftools/prometheus-ebpf-example Other Go +

    eBPF examples on GitHub github.com/leodido/go-ebpf-examples eBPF in a Pod
  39. @leodido eBPF using a CRD Pros: ★ No boilerplate ★

    Easy to use ★ A pod on every node ★ Automagically expose a Prometheus endpoint for every map you create Cons: ★ Deploy the controller ★ Not very extensible I’m that Kind of person.
  40. @leodido eBPF using a CRD

  41. @leodido eBPF using a CRD clang -O2 -target bpf -c

    pkts.c -o pkts.o kubectl create configmap --from-file pkts.o pkts-config -o yaml --dry-run >> “pkts.yaml”
  42. @leodido eBPF using a CRD

  43. @leodido eBPF using a CRD base64 ELF ‍♂

  44. @leodido eBPF using a CRD

  45. @leodido eBPF using a CRD Comes from github.com/bpftools/kube-bpf

  46. @leodido eBPF using a CRD Gets the ELF from the

    ConfigMap
  47. @leodido eBPF using a CRD

  48. @leodido eBPF using a CRD github.com/bpftools/kube-bpf

  49. @leodido eBPF with the kubectl Pros: ★ Uses bpftrace (DSL)

    ★ Very powerful ★ Unix philosophy Cons: ★ Does what bpftrace can do ★ No custom logic Like DTrace but for Kubernetes
  50. @leodido eBPF in the kubectl Every time the open syscall

    is executed print the opened file name
  51. @leodido eBPF in the kubectl Only on this specific node

  52. @leodido eBPF in the kubectl

  53. @leodido eBPF in the kubectl

  54. @leodido Every time the function is executed print the return

    value eBPF in the kubectl
  55. @leodido eBPF in the kubectl Only on this specific pod

  56. @leodido eBPF in the kubectl github.com/iovisor/kubectl-trace

  57. @leodido Instrumentation is hard On kubernetes it’s even harder eBPF

    is here to help Cloud native tools are already available
  58. @leodido Kubernetes + eBPF links for y’all • https://github.com/bpftools/kube-bpf •

    https://github.com/iovisor/kubectl-trace • https://github.com/falcosecurity/falco • https://github.com/draios/sysdig • https://github.com/bpftools/linux-observability-with-bpf
  59. @leodido Linux Observability with BPF • By my friend Lorenzo

    Fontana • There’s serious stuff there inside • Complimentary to this talk • go get your copy setns.run/bpf-book Free copy courtesy of Sysdig
  60. @leodido Leonardo Di Donato Open Source Software Engineer Tweets at

    @leodido Friendly person