Upgrade to Pro — share decks privately, control downloads, hide ads and more …

eBPF for Security Observability

eBPF for Security Observability

As seen at DevOpsDays Amsterdam and KCD Berlin

Liz Rice

June 24, 2022
Tweet

More Decks by Liz Rice

Other Decks in Technology

Transcript

  1. eBPF for Security Observability Liz Rice | @lizrice Chief Open

    Source Officer, Isovalent
  2. @lizrice

  3. @lizrice What is ? extended Berkeley Packet Filter

  4. @lizrice What is ? Makes the kernel programmable

  5. @lizrice userspace kernel app event system calls eBPF program Run

    custom code in the kernel
  6. @lizrice SEC("kprobe/sys_execve") int hello(void *ctx) { bpf_trace_printk("Hello World!"); return 0;

    } $ sudo ./hello bash-20241 [004] d... 84210.752785: 0: Hello World! bash-20242 [004] d... 84216.321993: 0: Hello World! bash-20243 [004] d... 84225.858880: 0: Hello World! Info about process that called execve syscall + userspace code to load eBPF program eBPF Hello World
  7. Dynamic changes to kernel behaviour

  8. Dynamic tracing tools

  9. @lizrice userspace kernel Tracing tool event eBPF program Use eBPF

    to collect event metrics eBPF Map metrics load Gather & show metrics
  10. @lizrice eBPF tracing tools from iovisor/bcc

  11. @lizrice eBPF tracing - opensnoop ~/bcc/libbpf-tools$ sudo ./opensnoop PID COMM

    FD ERR PATH 5040 node 21 0 /proc/5132/cmdline 5040 node 21 0 /proc/6460/cmdline 5040 node 21 0 /proc/6460/cmdline 6461 opensnoop 18 0 /etc/localtime 5040 node 21 0 /proc/5132/cmdline 5040 node 21 0 /proc/6460/cmdline 5060 node 23 0 /home/liz/.vscode-server/data/User/workspaceStorage/48b53 5040 node 21 0 /proc/5132/cmdline 5040 node 21 0 /proc/6460/cmdline 5040 node 21 0 /proc/5132/cmdline 5040 node 21 0 /proc/6460/cmdline …
  12. eBPF and Kubernetes

  13. @lizrice userspace kernel pod container pod container container One kernel

    per host
  14. @lizrice userspace kernel networking access files create containers One kernel

    per host pod container pod container container
  15. @lizrice userspace kernel app app pods networking access files create

    containers Kernel aware of everything on the host
  16. @lizrice userspace app kernel app pods networking access files create

    containers eBPF programs can be aware of everything
  17. @lizrice $ kubectl gadget trace open NODE NAMESPACE POD CONTAINER

    PID COMM FD ERR PATH kind-2-control-plane default xwing spaceship 361876 vi 3 0 /etc/passwd eBPF tracing on Kubernetes - Inspektor Gadget Kubernetes info
  18. @lizrice eBPF observability tools -

  19. @lizrice eBPF observability tools - Cilium Hubble

  20. eBPF observability

  21. eBPF security observability

  22. @lizrice Security observability

  23. @lizrice Security observability

  24. @lizrice What activity do we care about for security? eBPF

    programs
  25. @lizrice Syscall checks within the kernel

  26. @lizrice TOCTTOU vulnerabilities with syscalls For more details • Leo

    Di Donato & KP Singh at CN eBPF Day 2021 • Rex Guo & Junyuan Zeng at DEFCON 29 on Phantom attacks Attacker changes params after inspection
  27. @lizrice Need to make the check at the right place

  28. @lizrice Linux Security Modules • Stable interface • Safe places

    to make checks
  29. @lizrice BPF LSM • Stable interface • Safe places to

    make checks + eBPF benefits • Dynamic • Protect pre-existing processes
  30. @lizrice $ sudo ./chmoddemo & [1] 7631 $ sudo cat

    /sys/kernel/debug/tracing/trace_pipe chmod-7776 [001] d... 38197.342160: bpf_trace_printk: lsm path_chmod liz BPF LSM hook has kernel info populated SEC("lsm/path_chmod") int BPF_PROG(path_chmod, const struct path *path, umode_t mode) { bpf_printk("lsm path_chmod %s\n", path->dentry->d_iname); return 0; } Filename known to kernel
  31. @lizrice BPF LSM • Stable interface • Safe places to

    make checks + eBPF benefits • Dynamic • Protect pre-existing processes But needs kernel 5.7+ & Kubernetes context?
  32. How stable is the Linux kernel?

  33. @lizrice Cilium Tetragon • Safe places to make checks +

    eBPF benefits • Dynamic • Protect pre-existing processes Uses kernel knowledge to hook into sufficiently stable functions Adds Kubernetes context
  34. @lizrice Photo credit: Bibafu A Tetragonisca angustula bee guarding the

    nest-entrance
  35. @lizrice apiVersion: cilium.io/v1alpha1 kind: TracingPolicy metadata: name: "etc-files" spec: kprobes:

    - call: "fd_install" … matchArgs: - index: 1 operator: "Prefix" values: - "/etc/" … Cilium Tetragon tracing policy + Policy “follows” file descriptor through read, write & close events
  36. @lizrice $ kubectl logs ds/tetragon -c export-stdout -f | tetragon

    observe 🚀 process default/xwing /usr/bin/vi /etc/passwd 📬 open default/xwing /usr/bin/vi /etc/passwd 📪 close default/xwing /usr/bin/vi 📬 open default/xwing /usr/bin/vi /etc/passwd 📝 write default/xwing /usr/bin/vi /etc/passwd 1275 bytes 📪 close default/xwing /usr/bin/vi 💥 exit default/xwing /usr/bin/vi /etc/passwd 0 Cilium Tetragon observe Policy events Kubernetes info
  37. @lizrice Combined network and runtime visibility

  38. eBPF preventative runtime security

  39. @lizrice Network policy → eBPF programs drop packets

  40. @lizrice Preventative actions from user space

  41. @lizrice Preventative actions from kernel

  42. @lizrice $ kubectl logs ds/tetragon -c export-stdout -f | tetragon

    observe 🚀 process default/xwing /usr/bin/vi /etc/passwd 📬 open default/xwing /usr/bin/vi /etc/passwd 📪 close default/xwing /usr/bin/vi 📬 open default/xwing /usr/bin/vi /etc/passwd 📝 write default/xwing /usr/bin/vi /etc/passwd 1269 bytes 💥 exit default/xwing /usr/bin/vi /etc/passwd SIGKILL Cilium Tetragon observe Killed before write
  43. eBPF security observability • Dynamic instrumentation - zero app modifications

    • Contextual information, Kubernetes identity-aware • Option for runtime enforcement from the kernel
  44. Thank you! cilium/tetragon @ciliumproject cilium.io | ebpf.io @lizrice