eBPF for Security Observability

As seen at DevOpsDays Amsterdam and KCD Berlin

1. eBPF for Security Observability Liz Rice | @lizrice Chief Open Source Officer, Isovalent
2. @lizrice
3. @lizrice What is ? extended Berkeley Packet Filter
4. @lizrice What is ? Makes the kernel programmable
5. @lizrice userspace kernel app event system calls eBPF program Run custom code in the kernel
6. @lizrice SEC("kprobe/sys_execve") int hello(void *ctx) { bpf_trace_printk("Hello World!"); return 0; } $ sudo ./hello bash-20241 [004] d... 84210.752785: 0: Hello World! bash-20242 [004] d... 84216.321993: 0: Hello World! bash-20243 [004] d... 84225.858880: 0: Hello World! Info about process that called execve syscall + userspace code to load eBPF program eBPF Hello World
7. Dynamic changes to kernel behaviour
8. Dynamic tracing tools
9. @lizrice userspace kernel Tracing tool event eBPF program Use eBPF to collect event metrics eBPF Map metrics load Gather & show metrics
10. @lizrice eBPF tracing tools from iovisor/bcc
11. @lizrice eBPF tracing - opensnoop ~/bcc/libbpf-tools$ sudo ./opensnoop PID COMM FD ERR PATH 5040 node 21 0 /proc/5132/cmdline 5040 node 21 0 /proc/6460/cmdline 5040 node 21 0 /proc/6460/cmdline 6461 opensnoop 18 0 /etc/localtime 5040 node 21 0 /proc/5132/cmdline 5040 node 21 0 /proc/6460/cmdline 5060 node 23 0 /home/liz/.vscode-server/data/User/workspaceStorage/48b53 5040 node 21 0 /proc/5132/cmdline 5040 node 21 0 /proc/6460/cmdline 5040 node 21 0 /proc/5132/cmdline 5040 node 21 0 /proc/6460/cmdline …
12. eBPF and Kubernetes
13. @lizrice userspace kernel pod container pod container container One kernel per host
14. @lizrice userspace kernel networking access files create containers One kernel per host pod container pod container container
15. @lizrice userspace kernel app app pods networking access files create containers Kernel aware of everything on the host
16. @lizrice userspace app kernel app pods networking access files create containers eBPF programs can be aware of everything
17. @lizrice $ kubectl gadget trace open NODE NAMESPACE POD CONTAINER PID COMM FD ERR PATH kind-2-control-plane default xwing spaceship 361876 vi 3 0 /etc/passwd eBPF tracing on Kubernetes - Inspektor Gadget Kubernetes info
18. @lizrice eBPF observability tools -
19. @lizrice eBPF observability tools - Cilium Hubble
20. eBPF observability
21. eBPF security observability
22. @lizrice Security observability
23. @lizrice Security observability
24. @lizrice What activity do we care about for security? eBPF programs
25. @lizrice Syscall checks within the kernel
26. @lizrice TOCTTOU vulnerabilities with syscalls For more details • Leo Di Donato & KP Singh at CN eBPF Day 2021 • Rex Guo & Junyuan Zeng at DEFCON 29 on Phantom attacks Attacker changes params after inspection
27. @lizrice Need to make the check at the right place
28. @lizrice Linux Security Modules • Stable interface • Safe places to make checks
29. @lizrice BPF LSM • Stable interface • Safe places to make checks + eBPF benefits • Dynamic • Protect pre-existing processes
30. @lizrice $ sudo ./chmoddemo & [1] 7631 $ sudo cat /sys/kernel/debug/tracing/trace_pipe chmod-7776 [001] d... 38197.342160: bpf_trace_printk: lsm path_chmod liz BPF LSM hook has kernel info populated SEC("lsm/path_chmod") int BPF_PROG(path_chmod, const struct path *path, umode_t mode) { bpf_printk("lsm path_chmod %s\n", path->dentry->d_iname); return 0; } Filename known to kernel
31. @lizrice BPF LSM • Stable interface • Safe places to make checks + eBPF benefits • Dynamic • Protect pre-existing processes But needs kernel 5.7+ & Kubernetes context?
32. How stable is the Linux kernel?
33. @lizrice Cilium Tetragon • Safe places to make checks + eBPF benefits • Dynamic • Protect pre-existing processes Uses kernel knowledge to hook into sufficiently stable functions Adds Kubernetes context
34. @lizrice Photo credit: Bibafu A Tetragonisca angustula bee guarding the nest-entrance
35. @lizrice apiVersion: cilium.io/v1alpha1 kind: TracingPolicy metadata: name: "etc-files" spec: kprobes: - call: "fd_install" … matchArgs: - index: 1 operator: "Prefix" values: - "/etc/" … Cilium Tetragon tracing policy + Policy “follows” file descriptor through read, write & close events
36. @lizrice $ kubectl logs ds/tetragon -c export-stdout -f | tetragon observe 🚀 process default/xwing /usr/bin/vi /etc/passwd 📬 open default/xwing /usr/bin/vi /etc/passwd 📪 close default/xwing /usr/bin/vi 📬 open default/xwing /usr/bin/vi /etc/passwd 📝 write default/xwing /usr/bin/vi /etc/passwd 1275 bytes 📪 close default/xwing /usr/bin/vi 💥 exit default/xwing /usr/bin/vi /etc/passwd 0 Cilium Tetragon observe Policy events Kubernetes info
37. @lizrice Combined network and runtime visibility
38. eBPF preventative runtime security
39. @lizrice Network policy → eBPF programs drop packets
40. @lizrice Preventative actions from user space
41. @lizrice Preventative actions from kernel
42. @lizrice $ kubectl logs ds/tetragon -c export-stdout -f | tetragon observe 🚀 process default/xwing /usr/bin/vi /etc/passwd 📬 open default/xwing /usr/bin/vi /etc/passwd 📪 close default/xwing /usr/bin/vi 📬 open default/xwing /usr/bin/vi /etc/passwd 📝 write default/xwing /usr/bin/vi /etc/passwd 1269 bytes 💥 exit default/xwing /usr/bin/vi /etc/passwd SIGKILL Cilium Tetragon observe Killed before write
43. eBPF security observability • Dynamic instrumentation - zero app modifications • Contextual information, Kubernetes identity-aware • Option for runtime enforcement from the kernel
44. Thank you! cilium/tetragon @ciliumproject cilium.io | ebpf.io @lizrice