Introduction to eBPF and related tools

What is eBPF technology and what it means for Cloud Native tools, Yaniv Agman and Itay Shakury
eBPF is a powerful Linux technology that is recently in the spotlight for use cases of security, observability and customizability of the Linux kernel.
In this talk, we will introduce eBPF, and learn about its merits and challenges. We reference several open source tools for the Cloud Native stack that are using it, and present a new project open source project that we wrote to help everyone use eBPF with containers.

Transcript

1. © 2020 Aqua Security Software Ltd., All Rights Reserved Introduction to eBPF and tools Building tomorrow’s infrastructure
2. 2 Yaniv Agman • Security Researcher @ Aqua Security • Play with Linux stuff • Tracee open source (eBPF based)
3. 3 • (e)BPF. You are going to hear this name a lot in the near future • BPF super powers to the help: - performance analysis - tracing (e.g. system calls) - firewalls - enforcing security policies - debugging - reverse engineering - more... eB what?
4. 4 - Code running in the kernel has more privileges, and visibility of the system - To run code in the kernel, one can modify and compile the kernel itself, or write a kernel module and load it in runtime Running code in kernel kernel space user space
5. 5 • A virtual machine which runs in the (Linux) kernel • BPF has its own instruction set, registers (64 bit each), and the ability to call (some) in-kernel functions • BPF let’s you run your own code in kernel space, safely! What is it exactly eBPF? kernel space BPF vm user space user space code
6. 6 Keeping it safe - what we would like - Not crashing the system - No infinite loops - Don’t mess the kernel state We don’t have these guarantees with kernel modules! kernel BPF vm user space user space code load
7. 7 Keeping it safe - how? - No loops - Memory access bound and type checks - No null dereferences - Limited stack size (512k) - More checks... kernel verifier BPF vm user space user space code load
8. 8 Communicating with user space Kernel can store data in ring buffers which can then be read in user space Maps are key-value data structures shared between the user and kernel spaces kernel verifier BPF vm user space user space code maps ring buffers load messages
9. 9 Triggering an event - tracepoints (kernel static) - USDT (user static) - kprobes (kernel dynamic) - uprobes (user dynamic) - Network packets - Perf events This practically means you can trace almost ANY system event you want! kernel verifier BPF vm user space user space code maps ring buffers Probes load messages
10. 10 JIT compiler If BPF JiT is enabled in the kernel, BPF bytecode gets compiled to native machine code to improve performance kernel verifier BPF vm (Native code) user space user space code maps ring buffers Probes load JiT messages
11. 11 Understanding the process and toolchain user space kernel verifier user space code maps ring buffers messages JiT BPF vm (Native code) bpf () syscall Probes (limited) C load - BPF helper functions - get uid, pid, timestamp, read/write to userspace... - Filter events - Dereference kernel structs - careful!
12. 12 Understanding the process and toolchain - cont. user space kernel verifier user space code maps ring buffers messages JiT BPF vm (Native code) bpf () syscall Probes (limited) C load - BPF helper functions - get uid, pid, timestamp, read/write to userspace... - Filter events - Dereference kernel structs - careful! ELF object file ◦ eBPF opcodes ◦ eBPF maps clang llvm kernel headers
13. 13 Understanding the process and toolchain - cont. user space kernel verifier handle events maps ring buffers bpf () syscall messages load JiT BPF vm (Native code) BPF_PROG_LOAD BPF_MAP_CREATE bpf () syscall Attach Probes ELF object file ◦ eBPF opcodes ◦ eBPF maps (limited) C clang llvm kernel headers - BPF helper functions - get uid, pid, timestamp, read/write to userspace... - Filter events - Dereference kernel structs - careful!
14. 14 BCC (BPF Compiler Collection) bcc llvm bpf() python golang compiles eBPF program wrapper for bpf() syscalls C++ language support
15. 15 #!/usr/bin/python from bcc import BPF prog = """ int my_prog(void *ctx) { bpf_trace_printk("Hello world\\n"); return 0; } """ b = BPF(text=prog) b.attach_kprobe(event="sys_clone", fn_name="my_prog") b.trace_print() Hello world
16. 16 From theory to practice
17. 17 • Open Source @ Aqua Security • Kubernetes and Cloud stuff • Organizer of CNCF TLV (previously Kubernetes TLV) • CNCF Ambassador • @itaysk Itay Shakury
18. 18 iovisor ▪ Linux Foundation organization ▪ Linux observability ▪ Brendan Gregg ▪ bcc, tools
19. 19 iovisor ▪ Linux Foundation organization ▪ Linux observability ▪ Brendan Gregg ▪ bcc, tools ▪ https://github.com/iovisor
20. 20 iovisor ▪ Linux Foundation organization ▪ Linux observability ▪ Brendan Gregg ▪ bcc, tools
21. 21 iovisor ▪ LF organization ▪ Linux observability ▪ Brendan Gregg ▪ bcc, tools
22. © 2020 Aqua Security Software Ltd., All Rights Reserved Demo bcc tools
23. 23 bpftrace ▪ Versatile programmable tracer ▪ Domain Specific Language, inspired by awk ▪ https://github.com/iovisor/bpftrace bpftrace -e 'tracepoint:syscalls:sys_enter_* { @[probe] = count(); }' bpftrace bpf (C) BPF vm (Native code)
24. © 2020 Aqua Security Software Ltd., All Rights Reserved Demo bpf trace
25. 25 kubectl-trace ▪ bpftrace meets Kubernetes ▪ https://github.com/iovisor/kubectl-trace kubectl trace run pod/nginx -e 'tracepoint:syscalls:sys_enter_* { @[probe] = count(); }' bpftrace bpf (C) BPF vm (Native code) kubectl
26. 26 ebpf_exporter ▪ eBPF meets Prometheus ▪ C in yaml string ▪ https://github.com/cloudflare/ebpf_exporter yaml bpf (C) BPF vm (Native code) HTTP Server Prometheus
27. © 2020 Aqua Security Software Ltd., All Rights Reserved Demo ebpf_exporter
28. 28 Tracee ▪ A simple and capable tracer ▪ Yaniv Agman, Ultrabox ▪ Container detection, json output ▪ https://github.com/aquasecurity/tracee tracee
29. © 2020 Aqua Security Software Ltd., All Rights Reserved Demo Tracee
30. 30 • Trace all the things • Usability ◦ Obtaining - better packaging ◦ Dependencies - ship precompiled ◦ Linux headers - BTF • Policy ◦ Falco - https://github.com/falcosecurity/falco ◦ OPA - modern general policy language ◦ Tracee + OPA ? What’s next?
31. Thank You!