Cloud Native eBPF Instrumentation

Slide 1

Slide 1 text

@leodido Cloud native eBPF instrumentation ☁ where do we go when we all fall in the cloud ☁ Cloud Native Rejekts - San Diego, 2019

Slide 2

Slide 2 text

@leodido Leonardo Di Donato Open Source Software Engineer Tweets at @leodido Friendly person

Slide 3

Slide 3 text

@leodido Kubernetes is hard @leodido

Slide 4

Slide 4 text

@leodido Kernel is hard @leodido

Slide 5

Slide 5 text

@leodido Accidentally… Their names both start with a K Coincidence? I don’t think so

Slide 6

Slide 6 text

@leodido Why kernel instrumentation is harder on Kubernetes? @leodido

Slide 7

Slide 7 text

@leodido Kubernetes is an abstraction layer @leodido

Slide 8

Slide 8 text

@leodido Kubernetes complexity reflects on your ability to observe what’s going on under the abstraction @leodido

Slide 9

Slide 9 text

@leodido Instrumentation for Kubernetes makes me cry @leodido

Slide 10

Slide 10 text

@leodido Instrumentation tools are not cloud-native @leodido

Slide 11

Slide 11 text

@leodido Our kubernetes clusters are made for cloud native software development. We need cloud native instrumentation tools @leodido

Slide 12

Slide 12 text

@leodido What are my options then?

Slide 13

Slide 13 text

@leodido Many options strace in code/app as having the custom code in the application itself /proc and /sys kernel modules valgrind many non cloud-native options top, htop, iotop, etc. eBPF perf

Slide 14

Slide 14 text

@leodido hard to write. hard to maintain. crazy stuff. ☠ Many options strace kernel modules valgrind many non cloud-native options top, htop, iotop, etc. eBPF perf slows down applications. makes them unstable. slows down applications. very limited. invasive. not agnostic. performance impact. in code/app as having the custom code in the application itself /proc and /sys very limited. can see everything. can also use eBPF. limited integration. trace everything. fully programmable. negligible impact. fast. lots of tools!

Slide 15

Slide 15 text

@leodido Ok, but… @leodido

Slide 16

Slide 16 text

@leodido Kubernetes is distributed @leodido

Slide 17

Slide 17 text

@leodido Tooling exists but is not aware of the abstraction @leodido

Slide 18

Slide 18 text

@leodido Tooling exists but it was made for people to use over SSH @leodido

Slide 19

Slide 19 text

@leodido Kubernetes SSH is the kubectl @leodido

Slide 20

Slide 20 text

@leodido Kubernetes SSH is the kubectl kube-cattle @leodido

Slide 21

Slide 21 text

@leodido Application Kubernetes OS Kernel Hardware Abstraction @leodido

Slide 22

Slide 22 text

@leodido Application Kubernetes OS Kernel Hardware Abstraction @leodido The interesting stuff is here.

Slide 23

Slide 23 text

@leodido Application Kubernetes OS Kernel Hardware Abstraction @leodido Knows about the whole thing...

Slide 24

Slide 24 text

@leodido Application Kubernetes OS Kernel Hardware Abstraction @leodido You can ask everything at this level using an eBPF program!

Slide 25

Slide 25 text

@leodido How to integrate eBPF in cloud native infrastructures? They want to be together, we need to help them.

Slide 26

Slide 26 text

@leodido eBPF in a Pod

Slide 27

Slide 27 text

@leodido eBPF using a CRD

Slide 28

Slide 28 text

@leodido eBPF with the kubectl

Slide 29

Slide 29 text

@leodido eBPF in a Pod Pros: ★ Very customizable ★ Easy deployment ★ No installations Cons: ★ YAML boilerplate Easy peasy lemon squeezy

Slide 30

Slide 30 text

@leodido eBPF in a Pod

Slide 31

Slide 31 text

@leodido A Go string constant containing C code! eBPF in a Pod

Slide 32

Slide 32 text

@leodido eBPF in a Pod

Slide 33

Slide 33 text

@leodido The C code eBPF in a Pod

Slide 34

Slide 34 text

@leodido eBPF in a Pod

Slide 35

Slide 35 text

@leodido This image uses a compiled version of our BPF loader as entrypoint. eBPF in a Pod

Slide 36

Slide 36 text

@leodido eBPF in a Pod kubectl apply -f https://raw.githubusercontent.com/bpftools/prometheus-ebpf-example/master/daemonset.yaml kubectl get pods -n bpf-stuff kubectl port-forward daemonset/bpf-program -n bpf-stuff 8080:8080

Slide 37

Slide 37 text

@leodido eBPF in a Pod curl http://127.0.0.1:8080/metrics

Slide 38

Slide 38 text

@leodido Full example repository on GitHub github.com/bpftools/prometheus-ebpf-example Other Go + eBPF examples on GitHub github.com/leodido/go-ebpf-examples eBPF in a Pod

Slide 39

Slide 39 text

@leodido eBPF using a CRD Pros: ★ No boilerplate ★ Easy to use ★ A pod on every node ★ Automagically expose a Prometheus endpoint for every map you create Cons: ★ Deploy the controller ★ Not very extensible I’m that Kind of person.

Slide 40

Slide 40 text

@leodido eBPF using a CRD

Slide 41

Slide 41 text

@leodido eBPF using a CRD clang -O2 -target bpf -c pkts.c -o pkts.o kubectl create configmap --from-file pkts.o pkts-config -o yaml --dry-run >> “pkts.yaml”

Slide 42

Slide 42 text

@leodido eBPF using a CRD

Slide 43

Slide 43 text

@leodido eBPF using a CRD base64 ELF ‍♂

Slide 44

Slide 44 text

@leodido eBPF using a CRD

Slide 45

Slide 45 text

@leodido eBPF using a CRD Comes from github.com/bpftools/kube-bpf

Slide 46

Slide 46 text

@leodido eBPF using a CRD Gets the ELF from the ConfigMap

Slide 47

Slide 47 text

@leodido eBPF using a CRD

Slide 48

Slide 48 text

@leodido eBPF using a CRD github.com/bpftools/kube-bpf

Slide 49

Slide 49 text

@leodido eBPF with the kubectl Pros: ★ Uses bpftrace (DSL) ★ Very powerful ★ Unix philosophy Cons: ★ Does what bpftrace can do ★ No custom logic Like DTrace but for Kubernetes

Slide 50

Slide 50 text

@leodido eBPF in the kubectl Every time the open syscall is executed print the opened file name

Slide 51

Slide 51 text

@leodido eBPF in the kubectl Only on this specific node

Slide 52

Slide 52 text

@leodido eBPF in the kubectl

Slide 53

Slide 53 text

@leodido eBPF in the kubectl

Slide 54

Slide 54 text

@leodido Every time the function is executed print the return value eBPF in the kubectl

Slide 55

Slide 55 text

@leodido eBPF in the kubectl Only on this specific pod

Slide 56

Slide 56 text

@leodido eBPF in the kubectl github.com/iovisor/kubectl-trace

Slide 57

Slide 57 text

@leodido Instrumentation is hard On kubernetes it’s even harder eBPF is here to help Cloud native tools are already available

Slide 58

Slide 58 text

@leodido Kubernetes + eBPF links for y’all ● https://github.com/bpftools/kube-bpf ● https://github.com/iovisor/kubectl-trace ● https://github.com/falcosecurity/falco ● https://github.com/draios/sysdig ● https://github.com/bpftools/linux-observability-with-bpf

Slide 59

Slide 59 text

@leodido Linux Observability with BPF ● By my friend Lorenzo Fontana ● There’s serious stuff there inside ● Complimentary to this talk ● go get your copy setns.run/bpf-book Free copy courtesy of Sysdig

Slide 60

Slide 60 text

@leodido Leonardo Di Donato Open Source Software Engineer Tweets at @leodido Friendly person