Upgrade to Pro — share decks privately, control downloads, hide ads and more …

eBPF for Security Observability

eBPF for Security Observability

As seen at DevOpsDays Amsterdam and KCD Berlin

676c8aec28ade455c442e648abfa1db5?s=128

Liz Rice

June 24, 2022
Tweet

More Decks by Liz Rice

Other Decks in Technology

Transcript

  1. eBPF for Security Observability Liz Rice | @lizrice Chief Open

    Source Officer, Isovalent
  2. @lizrice

  3. @lizrice What is ? extended Berkeley Packet Filter

  4. @lizrice What is ? Makes the kernel programmable

  5. @lizrice userspace kernel app event system calls eBPF program Run

    custom code in the kernel
  6. @lizrice SEC("kprobe/sys_execve") int hello(void *ctx) { bpf_trace_printk("Hello World!"); return 0;

    } $ sudo ./hello bash-20241 [004] d... 84210.752785: 0: Hello World! bash-20242 [004] d... 84216.321993: 0: Hello World! bash-20243 [004] d... 84225.858880: 0: Hello World! Info about process that called execve syscall + userspace code to load eBPF program eBPF Hello World
  7. Dynamic changes to kernel behaviour

  8. Dynamic tracing tools

  9. @lizrice userspace kernel Tracing tool event eBPF program Use eBPF

    to collect event metrics eBPF Map metrics load Gather & show metrics
  10. @lizrice eBPF tracing tools from iovisor/bcc

  11. @lizrice eBPF tracing - opensnoop ~/bcc/libbpf-tools$ sudo ./opensnoop PID COMM

    FD ERR PATH 5040 node 21 0 /proc/5132/cmdline 5040 node 21 0 /proc/6460/cmdline 5040 node 21 0 /proc/6460/cmdline 6461 opensnoop 18 0 /etc/localtime 5040 node 21 0 /proc/5132/cmdline 5040 node 21 0 /proc/6460/cmdline 5060 node 23 0 /home/liz/.vscode-server/data/User/workspaceStorage/48b53 5040 node 21 0 /proc/5132/cmdline 5040 node 21 0 /proc/6460/cmdline 5040 node 21 0 /proc/5132/cmdline 5040 node 21 0 /proc/6460/cmdline …
  12. eBPF and Kubernetes

  13. @lizrice userspace kernel pod container pod container container One kernel

    per host
  14. @lizrice userspace kernel networking access files create containers One kernel

    per host pod container pod container container
  15. @lizrice userspace kernel app app pods networking access files create

    containers Kernel aware of everything on the host
  16. @lizrice userspace app kernel app pods networking access files create

    containers eBPF programs can be aware of everything
  17. @lizrice $ kubectl gadget trace open NODE NAMESPACE POD CONTAINER

    PID COMM FD ERR PATH kind-2-control-plane default xwing spaceship 361876 vi 3 0 /etc/passwd eBPF tracing on Kubernetes - Inspektor Gadget Kubernetes info
  18. @lizrice eBPF observability tools -

  19. @lizrice eBPF observability tools - Cilium Hubble

  20. eBPF observability

  21. eBPF security observability

  22. @lizrice Security observability

  23. @lizrice Security observability

  24. @lizrice What activity do we care about for security? eBPF

    programs
  25. @lizrice Syscall checks within the kernel

  26. @lizrice TOCTTOU vulnerabilities with syscalls For more details • Leo

    Di Donato & KP Singh at CN eBPF Day 2021 • Rex Guo & Junyuan Zeng at DEFCON 29 on Phantom attacks Attacker changes params after inspection
  27. @lizrice Need to make the check at the right place

  28. @lizrice Linux Security Modules • Stable interface • Safe places

    to make checks
  29. @lizrice BPF LSM • Stable interface • Safe places to

    make checks + eBPF benefits • Dynamic • Protect pre-existing processes
  30. @lizrice $ sudo ./chmoddemo & [1] 7631 $ sudo cat

    /sys/kernel/debug/tracing/trace_pipe chmod-7776 [001] d... 38197.342160: bpf_trace_printk: lsm path_chmod liz BPF LSM hook has kernel info populated SEC("lsm/path_chmod") int BPF_PROG(path_chmod, const struct path *path, umode_t mode) { bpf_printk("lsm path_chmod %s\n", path->dentry->d_iname); return 0; } Filename known to kernel
  31. @lizrice BPF LSM • Stable interface • Safe places to

    make checks + eBPF benefits • Dynamic • Protect pre-existing processes But needs kernel 5.7+ & Kubernetes context?
  32. How stable is the Linux kernel?

  33. @lizrice Cilium Tetragon • Safe places to make checks +

    eBPF benefits • Dynamic • Protect pre-existing processes Uses kernel knowledge to hook into sufficiently stable functions Adds Kubernetes context
  34. @lizrice Photo credit: Bibafu A Tetragonisca angustula bee guarding the

    nest-entrance
  35. @lizrice apiVersion: cilium.io/v1alpha1 kind: TracingPolicy metadata: name: "etc-files" spec: kprobes:

    - call: "fd_install" … matchArgs: - index: 1 operator: "Prefix" values: - "/etc/" … Cilium Tetragon tracing policy + Policy “follows” file descriptor through read, write & close events
  36. @lizrice $ kubectl logs ds/tetragon -c export-stdout -f | tetragon

    observe 🚀 process default/xwing /usr/bin/vi /etc/passwd 📬 open default/xwing /usr/bin/vi /etc/passwd 📪 close default/xwing /usr/bin/vi 📬 open default/xwing /usr/bin/vi /etc/passwd 📝 write default/xwing /usr/bin/vi /etc/passwd 1275 bytes 📪 close default/xwing /usr/bin/vi 💥 exit default/xwing /usr/bin/vi /etc/passwd 0 Cilium Tetragon observe Policy events Kubernetes info
  37. @lizrice Combined network and runtime visibility

  38. eBPF preventative runtime security

  39. @lizrice Network policy → eBPF programs drop packets

  40. @lizrice Preventative actions from user space

  41. @lizrice Preventative actions from kernel

  42. @lizrice $ kubectl logs ds/tetragon -c export-stdout -f | tetragon

    observe 🚀 process default/xwing /usr/bin/vi /etc/passwd 📬 open default/xwing /usr/bin/vi /etc/passwd 📪 close default/xwing /usr/bin/vi 📬 open default/xwing /usr/bin/vi /etc/passwd 📝 write default/xwing /usr/bin/vi /etc/passwd 1269 bytes 💥 exit default/xwing /usr/bin/vi /etc/passwd SIGKILL Cilium Tetragon observe Killed before write
  43. eBPF security observability • Dynamic instrumentation - zero app modifications

    • Contextual information, Kubernetes identity-aware • Option for runtime enforcement from the kernel
  44. Thank you! cilium/tetragon @ciliumproject cilium.io | ebpf.io @lizrice