Cloud Native eBPF Instrumentation

In this talk we are going to see how in Cloud Native environments we have the common issue of having tools to instrument and comprehend the application behaviour at kernel level. To try to solve this problem I'll try to illustrate my opinions on how I used eBPF and eBPF based tools that are both the kernel and Kubernetes aware. In other words, Cloud Native.

Transcript

1. @leodido
   Cloud native eBPF
   instrumentation
   ☁ where do we go when we all fall in the cloud ☁
   Cloud Native Rejekts - San Diego, 2019
   

   View Slide

2. @leodido
   Leonardo
   Di Donato
   Open Source Software
   Engineer
   Tweets at @leodido
   Friendly person
   

   View Slide

3. @leodido
   Kubernetes is hard
   @leodido
   

   View Slide

4. @leodido
   Kernel is hard
   @leodido
   

   View Slide

5. @leodido
   Accidentally…
   Their names both start with a K
   Coincidence?
   I don’t think so
   

   View Slide

6. @leodido
   Why kernel instrumentation is
   harder on Kubernetes?
   @leodido
   

   View Slide

7. @leodido
   Kubernetes is an
   abstraction layer
   @leodido
   

   View Slide

8. @leodido
   Kubernetes complexity reflects on
   your ability to observe what’s
   going on under the abstraction
   @leodido
   

   View Slide

9. @leodido
   Instrumentation for Kubernetes
   makes me cry
   @leodido
   

   View Slide

10. @leodido
    Instrumentation tools
    are not cloud-native
    @leodido
    

    View Slide

11. @leodido
    Our kubernetes clusters are made
    for cloud native software
    development.
    We need cloud native
    instrumentation tools
    @leodido
    

    View Slide

12. @leodido
    What are my options then?
    

    View Slide

13. @leodido
    Many options
    strace
    in code/app
    as having the custom code in
    the application itself
    /proc and /sys
    kernel modules valgrind
    many non cloud-native options
    top, htop,
    iotop, etc.
    eBPF
    perf
    

    View Slide

14. @leodido
    hard to write.
    hard to maintain.
    crazy stuff.
    ☠
    Many options
    strace kernel modules valgrind
    many non cloud-native options
    top, htop,
    iotop, etc.
    eBPF
    perf
    slows down applications.
    makes them unstable. slows down applications.
    very limited.
    invasive.
    not agnostic.
    performance impact.
    in code/app
    as having the custom code in
    the application itself
    /proc and /sys
    very limited.
    can see everything.
    can also use eBPF.
    limited integration.
    trace everything.
    fully programmable.
    negligible impact.
    fast.
    lots of tools!
    

    View Slide

15. @leodido
    Ok, but…
    @leodido
    

    View Slide

16. @leodido
    Kubernetes is
    distributed
    @leodido
    

    View Slide

17. @leodido
    Tooling exists but is not aware
    of the abstraction
    @leodido
    

    View Slide

18. @leodido
    Tooling exists but it was made
    for people to use over SSH
    @leodido
    

    View Slide

19. @leodido
    Kubernetes SSH is the
    kubectl
    @leodido
    

    View Slide

20. @leodido
    Kubernetes SSH is the
    kubectl
    kube-cattle
    @leodido
    

    View Slide

21. @leodido
    Application
    Kubernetes
    OS
    Kernel
    Hardware
    Abstraction
    @leodido
    

    View Slide

22. @leodido
    Application
    Kubernetes
    OS
    Kernel
    Hardware
    Abstraction
    @leodido
    The interesting stuff is here.
    

    View Slide

23. @leodido
    Application
    Kubernetes
    OS
    Kernel
    Hardware
    Abstraction
    @leodido
    Knows about the whole thing...
    

    View Slide

24. @leodido
    Application
    Kubernetes
    OS
    Kernel
    Hardware
    Abstraction
    @leodido
    You can ask everything at this
    level using an eBPF program!
    

    View Slide

25. @leodido
    How to integrate eBPF in
    cloud native
    infrastructures?
    They want to be together, we need to help them.
    

    View Slide

26. @leodido
    eBPF in a Pod
    

    View Slide

27. @leodido
    eBPF using a CRD
    

    View Slide

28. @leodido
    eBPF with the kubectl
    

    View Slide

29. @leodido
    eBPF in a Pod
    Pros:
    ★ Very customizable
    ★ Easy deployment
    ★ No installations
    Cons:
    ★ YAML boilerplate
    Easy peasy lemon squeezy
    

    View Slide

30. @leodido
    eBPF in a Pod
    

    View Slide

31. @leodido
    A Go string constant
    containing C code!
    eBPF in a Pod
    

    View Slide

32. @leodido
    eBPF in a Pod
    

    View Slide

33. @leodido
    The C code
    eBPF in a Pod
    

    View Slide

34. @leodido
    eBPF in a Pod
    

    View Slide

35. @leodido
    This image uses a compiled
    version of our BPF loader
    as entrypoint.
    eBPF in a Pod
    

    View Slide

36. @leodido
    eBPF in a Pod
    kubectl apply -f https://raw.githubusercontent.com/bpftools/prometheus-ebpf-example/master/daemonset.yaml
    kubectl get pods -n bpf-stuff
    kubectl port-forward daemonset/bpf-program -n bpf-stuff 8080:8080
    

    View Slide

37. @leodido
    eBPF in a Pod
    curl http://127.0.0.1:8080/metrics
    

    View Slide

38. @leodido
    Full example repository on GitHub
    
    github.com/bpftools/prometheus-ebpf-example
    Other Go + eBPF examples on GitHub
    
    github.com/leodido/go-ebpf-examples
    eBPF in a Pod
    

    View Slide

39. @leodido
    eBPF
    using a CRD
    Pros:
    ★ No boilerplate
    ★ Easy to use
    ★ A pod on every node
    ★ Automagically expose a
    Prometheus endpoint for
    every map you create
    Cons:
    ★ Deploy the controller
    ★ Not very extensible
    I’m that Kind of person.
    

    View Slide

40. @leodido
    eBPF
    using a CRD
    

    View Slide

41. @leodido
    eBPF using a CRD
    clang -O2 -target bpf -c pkts.c -o pkts.o
    kubectl create configmap --from-file pkts.o pkts-config -o yaml --dry-run >> “pkts.yaml”
    

    View Slide

42. @leodido
    eBPF using a CRD
    

    View Slide

43. @leodido
    eBPF using a CRD
    base64 ELF
    ‍♂
    
    

    View Slide

44. @leodido
    eBPF using a CRD
    

    View Slide

45. @leodido
    eBPF using a CRD
    Comes from github.com/bpftools/kube-bpf
    

    View Slide

46. @leodido
    eBPF using a CRD
    Gets the ELF from the ConfigMap
    

    View Slide

47. @leodido
    eBPF using a CRD
    

    View Slide

48. @leodido
    eBPF using a CRD
    github.com/bpftools/kube-bpf
    

    View Slide

49. @leodido
    eBPF with the
    kubectl
    Pros:
    ★ Uses bpftrace (DSL)
    ★ Very powerful
    ★ Unix philosophy
    Cons:
    ★ Does what bpftrace can do
    ★ No custom logic
    Like DTrace but for Kubernetes
    

    View Slide

50. @leodido
    eBPF in the kubectl
    Every time the open syscall is executed print the opened file name
    

    View Slide

51. @leodido
    eBPF in the kubectl
    Only on this specific node
    

    View Slide

52. @leodido
    eBPF in the kubectl
    

    View Slide

53. @leodido
    eBPF in the kubectl
    

    View Slide

54. @leodido
    Every time the function is executed
    print the return value
    eBPF in the kubectl
    

    View Slide

55. @leodido
    eBPF in the kubectl
    Only on this specific pod
    

    View Slide

56. @leodido
    eBPF in the kubectl
    github.com/iovisor/kubectl-trace
    

    View Slide

57. @leodido
    Instrumentation is hard
    On kubernetes it’s even harder
    eBPF is here to help
    Cloud native tools are already
    available
    

    View Slide

58. @leodido
    Kubernetes + eBPF links for y’all
    ● https://github.com/bpftools/kube-bpf
    ● https://github.com/iovisor/kubectl-trace
    ● https://github.com/falcosecurity/falco
    ● https://github.com/draios/sysdig
    ● https://github.com/bpftools/linux-observability-with-bpf
    

    View Slide

59. @leodido
    Linux Observability with BPF
    ● By my friend Lorenzo Fontana
    ● There’s serious stuff there inside
    ● Complimentary to this talk
    ● go get your copy
    setns.run/bpf-book
    Free copy courtesy of Sysdig
    

    View Slide

60. @leodido
    Leonardo
    Di Donato
    Open Source Software
    Engineer
    Tweets at @leodido
    Friendly person
    

    View Slide