Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Native eBPF Instrumentation

Cloud Native eBPF Instrumentation

In this talk we are going to see how in Cloud Native environments we have the common issue of having tools to instrument and comprehend the application behaviour at kernel level. To try to solve this problem I'll try to illustrate my opinions on how I used eBPF and eBPF based tools that are both the kernel and Kubernetes aware. In other words, Cloud Native.

Leonardo Di Donato

November 17, 2019
Tweet

More Decks by Leonardo Di Donato

Other Decks in Programming

Transcript

  1. @leodido
    Cloud native eBPF
    instrumentation
    ☁ where do we go when we all fall in the cloud ☁
    Cloud Native Rejekts - San Diego, 2019

    View Slide

  2. @leodido
    Leonardo
    Di Donato
    Open Source Software
    Engineer
    Tweets at @leodido
    Friendly person

    View Slide

  3. @leodido
    Kubernetes is hard
    @leodido

    View Slide

  4. @leodido
    Kernel is hard
    @leodido

    View Slide

  5. @leodido
    Accidentally…
    Their names both start with a K
    Coincidence?
    I don’t think so

    View Slide

  6. @leodido
    Why kernel instrumentation is
    harder on Kubernetes?
    @leodido

    View Slide

  7. @leodido
    Kubernetes is an
    abstraction layer
    @leodido

    View Slide

  8. @leodido
    Kubernetes complexity reflects on
    your ability to observe what’s
    going on under the abstraction
    @leodido

    View Slide

  9. @leodido
    Instrumentation for Kubernetes
    makes me cry
    @leodido

    View Slide

  10. @leodido
    Instrumentation tools
    are not cloud-native
    @leodido

    View Slide

  11. @leodido
    Our kubernetes clusters are made
    for cloud native software
    development.
    We need cloud native
    instrumentation tools
    @leodido

    View Slide

  12. @leodido
    What are my options then?

    View Slide

  13. @leodido
    Many options
    strace
    in code/app
    as having the custom code in
    the application itself
    /proc and /sys
    kernel modules valgrind
    many non cloud-native options
    top, htop,
    iotop, etc.
    eBPF
    perf

    View Slide

  14. @leodido
    hard to write.
    hard to maintain.
    crazy stuff.

    Many options
    strace kernel modules valgrind
    many non cloud-native options
    top, htop,
    iotop, etc.
    eBPF
    perf
    slows down applications.
    makes them unstable. slows down applications.
    very limited.
    invasive.
    not agnostic.
    performance impact.
    in code/app
    as having the custom code in
    the application itself
    /proc and /sys
    very limited.
    can see everything.
    can also use eBPF.
    limited integration.
    trace everything.
    fully programmable.
    negligible impact.
    fast.
    lots of tools!

    View Slide

  15. @leodido
    Ok, but…
    @leodido

    View Slide

  16. @leodido
    Kubernetes is
    distributed
    @leodido

    View Slide

  17. @leodido
    Tooling exists but is not aware
    of the abstraction
    @leodido

    View Slide

  18. @leodido
    Tooling exists but it was made
    for people to use over SSH
    @leodido

    View Slide

  19. @leodido
    Kubernetes SSH is the
    kubectl
    @leodido

    View Slide

  20. @leodido
    Kubernetes SSH is the
    kubectl
    kube-cattle
    @leodido

    View Slide

  21. @leodido
    Application
    Kubernetes
    OS
    Kernel
    Hardware
    Abstraction
    @leodido

    View Slide

  22. @leodido
    Application
    Kubernetes
    OS
    Kernel
    Hardware
    Abstraction
    @leodido
    The interesting stuff is here.

    View Slide

  23. @leodido
    Application
    Kubernetes
    OS
    Kernel
    Hardware
    Abstraction
    @leodido
    Knows about the whole thing...

    View Slide

  24. @leodido
    Application
    Kubernetes
    OS
    Kernel
    Hardware
    Abstraction
    @leodido
    You can ask everything at this
    level using an eBPF program!

    View Slide

  25. @leodido
    How to integrate eBPF in
    cloud native
    infrastructures?
    They want to be together, we need to help them.

    View Slide

  26. @leodido
    eBPF in a Pod

    View Slide

  27. @leodido
    eBPF using a CRD

    View Slide

  28. @leodido
    eBPF with the kubectl

    View Slide

  29. @leodido
    eBPF in a Pod
    Pros:
    ★ Very customizable
    ★ Easy deployment
    ★ No installations
    Cons:
    ★ YAML boilerplate
    Easy peasy lemon squeezy

    View Slide

  30. @leodido
    eBPF in a Pod

    View Slide

  31. @leodido
    A Go string constant
    containing C code!
    eBPF in a Pod

    View Slide

  32. @leodido
    eBPF in a Pod

    View Slide

  33. @leodido
    The C code
    eBPF in a Pod

    View Slide

  34. @leodido
    eBPF in a Pod

    View Slide

  35. @leodido
    This image uses a compiled
    version of our BPF loader
    as entrypoint.
    eBPF in a Pod

    View Slide

  36. @leodido
    eBPF in a Pod
    kubectl apply -f https://raw.githubusercontent.com/bpftools/prometheus-ebpf-example/master/daemonset.yaml
    kubectl get pods -n bpf-stuff
    kubectl port-forward daemonset/bpf-program -n bpf-stuff 8080:8080

    View Slide

  37. @leodido
    eBPF in a Pod
    curl http://127.0.0.1:8080/metrics

    View Slide

  38. @leodido
    Full example repository on GitHub

    github.com/bpftools/prometheus-ebpf-example
    Other Go + eBPF examples on GitHub

    github.com/leodido/go-ebpf-examples
    eBPF in a Pod

    View Slide

  39. @leodido
    eBPF
    using a CRD
    Pros:
    ★ No boilerplate
    ★ Easy to use
    ★ A pod on every node
    ★ Automagically expose a
    Prometheus endpoint for
    every map you create
    Cons:
    ★ Deploy the controller
    ★ Not very extensible
    I’m that Kind of person.

    View Slide

  40. @leodido
    eBPF
    using a CRD

    View Slide

  41. @leodido
    eBPF using a CRD
    clang -O2 -target bpf -c pkts.c -o pkts.o
    kubectl create configmap --from-file pkts.o pkts-config -o yaml --dry-run >> “pkts.yaml”

    View Slide

  42. @leodido
    eBPF using a CRD

    View Slide

  43. @leodido
    eBPF using a CRD
    base64 ELF
    ‍♂

    View Slide

  44. @leodido
    eBPF using a CRD

    View Slide

  45. @leodido
    eBPF using a CRD
    Comes from github.com/bpftools/kube-bpf

    View Slide

  46. @leodido
    eBPF using a CRD
    Gets the ELF from the ConfigMap

    View Slide

  47. @leodido
    eBPF using a CRD

    View Slide

  48. @leodido
    eBPF using a CRD
    github.com/bpftools/kube-bpf

    View Slide

  49. @leodido
    eBPF with the
    kubectl
    Pros:
    ★ Uses bpftrace (DSL)
    ★ Very powerful
    ★ Unix philosophy
    Cons:
    ★ Does what bpftrace can do
    ★ No custom logic
    Like DTrace but for Kubernetes

    View Slide

  50. @leodido
    eBPF in the kubectl
    Every time the open syscall is executed print the opened file name

    View Slide

  51. @leodido
    eBPF in the kubectl
    Only on this specific node

    View Slide

  52. @leodido
    eBPF in the kubectl

    View Slide

  53. @leodido
    eBPF in the kubectl

    View Slide

  54. @leodido
    Every time the function is executed
    print the return value
    eBPF in the kubectl

    View Slide

  55. @leodido
    eBPF in the kubectl
    Only on this specific pod

    View Slide

  56. @leodido
    eBPF in the kubectl
    github.com/iovisor/kubectl-trace

    View Slide

  57. @leodido
    Instrumentation is hard
    On kubernetes it’s even harder
    eBPF is here to help
    Cloud native tools are already
    available

    View Slide

  58. @leodido
    Kubernetes + eBPF links for y’all
    ● https://github.com/bpftools/kube-bpf
    ● https://github.com/iovisor/kubectl-trace
    ● https://github.com/falcosecurity/falco
    ● https://github.com/draios/sysdig
    ● https://github.com/bpftools/linux-observability-with-bpf

    View Slide

  59. @leodido
    Linux Observability with BPF
    ● By my friend Lorenzo Fontana
    ● There’s serious stuff there inside
    ● Complimentary to this talk
    ● go get your copy
    setns.run/bpf-book
    Free copy courtesy of Sysdig

    View Slide

  60. @leodido
    Leonardo
    Di Donato
    Open Source Software
    Engineer
    Tweets at @leodido
    Friendly person

    View Slide