$30 off During Our Annual Pro Sale. View Details »

eBPF for Security Observability

eBPF for Security Observability

As seen at DevOpsDays Amsterdam and KCD Berlin

Liz Rice

June 24, 2022
Tweet

More Decks by Liz Rice

Other Decks in Technology

Transcript

  1. eBPF for
    Security Observability
    Liz Rice | @lizrice
    Chief Open Source Officer, Isovalent

    View Slide

  2. @lizrice

    View Slide

  3. @lizrice
    What is ?
    extended
    Berkeley
    Packet
    Filter

    View Slide

  4. @lizrice
    What is ?
    Makes the kernel programmable

    View Slide

  5. @lizrice
    userspace
    kernel
    app
    event
    system calls
    eBPF program
    Run custom code in the kernel

    View Slide

  6. @lizrice
    SEC("kprobe/sys_execve")
    int hello(void *ctx)
    {
    bpf_trace_printk("Hello World!");
    return 0;
    }
    $ sudo ./hello
    bash-20241 [004] d... 84210.752785: 0: Hello World!
    bash-20242 [004] d... 84216.321993: 0: Hello World!
    bash-20243 [004] d... 84225.858880: 0: Hello World!
    Info about process that
    called execve syscall
    + userspace code to load eBPF
    program
    eBPF Hello World

    View Slide

  7. Dynamic changes to kernel
    behaviour

    View Slide

  8. Dynamic tracing tools

    View Slide

  9. @lizrice
    userspace
    kernel
    Tracing tool
    event eBPF
    program
    Use eBPF to collect event metrics
    eBPF
    Map
    metrics
    load
    Gather
    & show
    metrics

    View Slide

  10. @lizrice
    eBPF tracing tools from iovisor/bcc

    View Slide

  11. @lizrice
    eBPF tracing - opensnoop
    ~/bcc/libbpf-tools$ sudo ./opensnoop
    PID COMM FD ERR PATH
    5040 node 21 0 /proc/5132/cmdline
    5040 node 21 0 /proc/6460/cmdline
    5040 node 21 0 /proc/6460/cmdline
    6461 opensnoop 18 0 /etc/localtime
    5040 node 21 0 /proc/5132/cmdline
    5040 node 21 0 /proc/6460/cmdline
    5060 node 23 0 /home/liz/.vscode-server/data/User/workspaceStorage/48b53
    5040 node 21 0 /proc/5132/cmdline
    5040 node 21 0 /proc/6460/cmdline
    5040 node 21 0 /proc/5132/cmdline
    5040 node 21 0 /proc/6460/cmdline

    View Slide

  12. eBPF and Kubernetes

    View Slide

  13. @lizrice
    userspace
    kernel
    pod container
    pod container
    container
    One kernel per
    host

    View Slide

  14. @lizrice
    userspace
    kernel
    networking
    access files
    create
    containers
    One kernel per
    host
    pod container
    pod container
    container

    View Slide

  15. @lizrice
    userspace
    kernel
    app
    app
    pods
    networking
    access files
    create
    containers
    Kernel aware of
    everything on
    the host

    View Slide

  16. @lizrice
    userspace
    app
    kernel
    app
    pods
    networking
    access files
    create
    containers
    eBPF programs
    can be aware of
    everything

    View Slide

  17. @lizrice
    $ kubectl gadget trace open
    NODE NAMESPACE POD CONTAINER PID COMM FD ERR PATH
    kind-2-control-plane default xwing spaceship 361876 vi 3 0 /etc/passwd
    eBPF tracing on Kubernetes - Inspektor Gadget
    Kubernetes info

    View Slide

  18. @lizrice
    eBPF observability tools -

    View Slide

  19. @lizrice
    eBPF observability tools - Cilium Hubble

    View Slide

  20. eBPF observability

    View Slide

  21. eBPF security observability

    View Slide

  22. @lizrice
    Security observability

    View Slide

  23. @lizrice
    Security observability

    View Slide

  24. @lizrice
    What activity do we care about for security?
    eBPF programs

    View Slide

  25. @lizrice
    Syscall checks within the kernel

    View Slide

  26. @lizrice
    TOCTTOU vulnerabilities with syscalls
    For more details
    ● Leo Di Donato & KP Singh at CN eBPF Day 2021
    ● Rex Guo & Junyuan Zeng at DEFCON 29 on Phantom attacks
    Attacker changes params
    after inspection

    View Slide

  27. @lizrice
    Need to make the check at the right place

    View Slide

  28. @lizrice
    Linux Security Modules
    ● Stable interface
    ● Safe places to make checks

    View Slide

  29. @lizrice
    BPF LSM
    ● Stable interface
    ● Safe places to make checks
    + eBPF benefits
    ● Dynamic
    ● Protect pre-existing processes

    View Slide

  30. @lizrice
    $ sudo ./chmoddemo &
    [1] 7631
    $ sudo cat /sys/kernel/debug/tracing/trace_pipe
    chmod-7776 [001] d... 38197.342160: bpf_trace_printk: lsm path_chmod liz
    BPF LSM hook has kernel info populated
    SEC("lsm/path_chmod")
    int BPF_PROG(path_chmod, const struct path *path, umode_t mode)
    {
    bpf_printk("lsm path_chmod %s\n", path->dentry->d_iname);
    return 0;
    } Filename known
    to kernel

    View Slide

  31. @lizrice
    BPF LSM
    ● Stable interface
    ● Safe places to make checks
    + eBPF benefits
    ● Dynamic
    ● Protect pre-existing processes
    But needs kernel 5.7+
    & Kubernetes context?

    View Slide

  32. How stable is the Linux kernel?

    View Slide

  33. @lizrice
    Cilium Tetragon
    ● Safe places to make checks
    + eBPF benefits
    ● Dynamic
    ● Protect pre-existing processes
    Uses kernel knowledge to hook into
    sufficiently stable functions
    Adds Kubernetes context

    View Slide

  34. @lizrice
    Photo credit: Bibafu
    A Tetragonisca angustula bee
    guarding the nest-entrance

    View Slide

  35. @lizrice
    apiVersion: cilium.io/v1alpha1
    kind: TracingPolicy
    metadata:
    name: "etc-files"
    spec:
    kprobes:
    - call: "fd_install"

    matchArgs:
    - index: 1
    operator: "Prefix"
    values:
    - "/etc/"

    Cilium Tetragon tracing policy
    + Policy “follows” file descriptor
    through read, write & close
    events

    View Slide

  36. @lizrice
    $ kubectl logs ds/tetragon -c export-stdout -f | tetragon observe
    🚀 process default/xwing /usr/bin/vi /etc/passwd
    📬 open default/xwing /usr/bin/vi /etc/passwd
    📪 close default/xwing /usr/bin/vi
    📬 open default/xwing /usr/bin/vi /etc/passwd
    📝 write default/xwing /usr/bin/vi /etc/passwd 1275 bytes
    📪 close default/xwing /usr/bin/vi
    💥 exit default/xwing /usr/bin/vi /etc/passwd 0
    Cilium Tetragon observe
    Policy events
    Kubernetes info

    View Slide

  37. @lizrice
    Combined network and runtime visibility

    View Slide

  38. eBPF preventative runtime security

    View Slide

  39. @lizrice
    Network policy → eBPF programs drop packets

    View Slide

  40. @lizrice
    Preventative actions from user space

    View Slide

  41. @lizrice
    Preventative actions from kernel

    View Slide

  42. @lizrice
    $ kubectl logs ds/tetragon -c export-stdout -f | tetragon observe
    🚀 process default/xwing /usr/bin/vi /etc/passwd
    📬 open default/xwing /usr/bin/vi /etc/passwd
    📪 close default/xwing /usr/bin/vi
    📬 open default/xwing /usr/bin/vi /etc/passwd
    📝 write default/xwing /usr/bin/vi /etc/passwd 1269 bytes
    💥 exit default/xwing /usr/bin/vi /etc/passwd SIGKILL
    Cilium Tetragon observe
    Killed before write

    View Slide

  43. eBPF security observability
    ● Dynamic instrumentation - zero app modifications
    ● Contextual information, Kubernetes identity-aware
    ● Option for runtime enforcement from the kernel

    View Slide

  44. Thank you!
    cilium/tetragon
    @ciliumproject
    cilium.io | ebpf.io
    @lizrice

    View Slide