As seen at DevOpsDays Amsterdam and KCD Berlin
eBPF forSecurity ObservabilityLiz Rice | @lizriceChief Open Source Officer, Isovalent
View Slide
@lizrice
@lizriceWhat is ?extendedBerkeleyPacketFilter
@lizriceWhat is ?Makes the kernel programmable
@lizriceuserspacekernelappeventsystem callseBPF programRun custom code in the kernel
@lizriceSEC("kprobe/sys_execve")int hello(void *ctx){bpf_trace_printk("Hello World!");return 0;}$ sudo ./hellobash-20241 [004] d... 84210.752785: 0: Hello World!bash-20242 [004] d... 84216.321993: 0: Hello World!bash-20243 [004] d... 84225.858880: 0: Hello World!Info about process thatcalled execve syscall+ userspace code to load eBPFprogrameBPF Hello World
Dynamic changes to kernelbehaviour
Dynamic tracing tools
@lizriceuserspacekernelTracing toolevent eBPFprogramUse eBPF to collect event metricseBPFMapmetricsloadGather& showmetrics
@lizriceeBPF tracing tools from iovisor/bcc
@lizriceeBPF tracing - opensnoop~/bcc/libbpf-tools$ sudo ./opensnoopPID COMM FD ERR PATH5040 node 21 0 /proc/5132/cmdline5040 node 21 0 /proc/6460/cmdline5040 node 21 0 /proc/6460/cmdline6461 opensnoop 18 0 /etc/localtime5040 node 21 0 /proc/5132/cmdline5040 node 21 0 /proc/6460/cmdline5060 node 23 0 /home/liz/.vscode-server/data/User/workspaceStorage/48b535040 node 21 0 /proc/5132/cmdline5040 node 21 0 /proc/6460/cmdline5040 node 21 0 /proc/5132/cmdline5040 node 21 0 /proc/6460/cmdline…
eBPF and Kubernetes
@lizriceuserspacekernelpod containerpod containercontainerOne kernel perhost
@lizriceuserspacekernelnetworkingaccess filescreatecontainersOne kernel perhostpod containerpod containercontainer
@lizriceuserspacekernelappapppodsnetworkingaccess filescreatecontainersKernel aware ofeverything onthe host
@lizriceuserspaceappkernelapppodsnetworkingaccess filescreatecontainerseBPF programscan be aware ofeverything
@lizrice$ kubectl gadget trace openNODE NAMESPACE POD CONTAINER PID COMM FD ERR PATHkind-2-control-plane default xwing spaceship 361876 vi 3 0 /etc/passwdeBPF tracing on Kubernetes - Inspektor GadgetKubernetes info
@lizriceeBPF observability tools -
@lizriceeBPF observability tools - Cilium Hubble
eBPF observability
eBPF security observability
@lizriceSecurity observability
@lizriceWhat activity do we care about for security?eBPF programs
@lizriceSyscall checks within the kernel
@lizriceTOCTTOU vulnerabilities with syscallsFor more details● Leo Di Donato & KP Singh at CN eBPF Day 2021● Rex Guo & Junyuan Zeng at DEFCON 29 on Phantom attacksAttacker changes paramsafter inspection
@lizriceNeed to make the check at the right place
@lizriceLinux Security Modules● Stable interface● Safe places to make checks
@lizriceBPF LSM● Stable interface● Safe places to make checks+ eBPF benefits● Dynamic● Protect pre-existing processes
@lizrice$ sudo ./chmoddemo &[1] 7631$ sudo cat /sys/kernel/debug/tracing/trace_pipechmod-7776 [001] d... 38197.342160: bpf_trace_printk: lsm path_chmod lizBPF LSM hook has kernel info populatedSEC("lsm/path_chmod")int BPF_PROG(path_chmod, const struct path *path, umode_t mode){bpf_printk("lsm path_chmod %s\n", path->dentry->d_iname);return 0;} Filename knownto kernel
@lizriceBPF LSM● Stable interface● Safe places to make checks+ eBPF benefits● Dynamic● Protect pre-existing processesBut needs kernel 5.7+& Kubernetes context?
How stable is the Linux kernel?
@lizriceCilium Tetragon● Safe places to make checks+ eBPF benefits● Dynamic● Protect pre-existing processesUses kernel knowledge to hook intosufficiently stable functionsAdds Kubernetes context
@lizricePhoto credit: BibafuA Tetragonisca angustula beeguarding the nest-entrance
@lizriceapiVersion: cilium.io/v1alpha1kind: TracingPolicymetadata:name: "etc-files"spec:kprobes:- call: "fd_install"…matchArgs:- index: 1operator: "Prefix"values:- "/etc/"…Cilium Tetragon tracing policy+ Policy “follows” file descriptorthrough read, write & closeevents
@lizrice$ kubectl logs ds/tetragon -c export-stdout -f | tetragon observe🚀 process default/xwing /usr/bin/vi /etc/passwd📬 open default/xwing /usr/bin/vi /etc/passwd📪 close default/xwing /usr/bin/vi📬 open default/xwing /usr/bin/vi /etc/passwd📝 write default/xwing /usr/bin/vi /etc/passwd 1275 bytes📪 close default/xwing /usr/bin/vi💥 exit default/xwing /usr/bin/vi /etc/passwd 0Cilium Tetragon observePolicy eventsKubernetes info
@lizriceCombined network and runtime visibility
eBPF preventative runtime security
@lizriceNetwork policy → eBPF programs drop packets
@lizricePreventative actions from user space
@lizricePreventative actions from kernel
@lizrice$ kubectl logs ds/tetragon -c export-stdout -f | tetragon observe🚀 process default/xwing /usr/bin/vi /etc/passwd📬 open default/xwing /usr/bin/vi /etc/passwd📪 close default/xwing /usr/bin/vi📬 open default/xwing /usr/bin/vi /etc/passwd📝 write default/xwing /usr/bin/vi /etc/passwd 1269 bytes💥 exit default/xwing /usr/bin/vi /etc/passwd SIGKILLCilium Tetragon observeKilled before write
eBPF security observability● Dynamic instrumentation - zero app modifications● Contextual information, Kubernetes identity-aware● Option for runtime enforcement from the kernel
Thank you!cilium/tetragon@ciliumprojectcilium.io | ebpf.io@lizrice
lizrice
akki_megane
oogfranz
korodroid
okunokentaro
seike460
ybrliiu
oekazuma
takufujii
tgalopin
line_developers
regonn
mamohacy
samanthasiow
nonsquared
ufuk
trallard
chriscoyier
sonatard
pedronauck
keavy
sugarenia
lauravandoore
kneath
jonyablonski