gobpf - utilizing BPF from Go - Alban Crequy - Kinvolk

BPF is a Linux in-kernel virtual machine that is used for tracing kernel functions, networking, performance analysis and more. In this talk I will give an introduction into eBPF and gobpf, a library under the IO Visor Project, and show how gobpf can be used to access and utilize BPF features from Go programs. We will also look at the design of gobpf and its use of the bcc framework and Cgo.

Transcript

1. gobpf - utilizing BPF from Go GoDays Berlin | 23.01.2020 https://tinyurl.com/godays-gobpf
2. Hi, I'm Alban Alban Crequy CTO, Kinvolk Github: alban Twitter: @albcr Email: alban@kinvolk.io
3. The Kubernetes Linux Experts Engineering services and products for Kubernetes, containers, process management and Linux user-space + kernel Blog: kinvolk.io/blog Github: kinvolk Twitter: kinvolkio Email: hello@kinvolk.io Kinvolk
4. @albcr Plan - Introduction to BPF - BPF Compiler Collection (BCC) - gobpf - Demos - Future features
5. Introduction to BPF
6. @albcr Intro to Berkeley Packet Filter - BPF: bytecode executed in the Linux kernel - Initially for tcpdump (1992) - Extended BPF (2013)
7. @albcr How tcpdump captures network packets Local process (apache) input output routing decision forward post-routing pre-routing Incoming packet tcpdump socket AF_PACKET copy socket AF_INET
8. @albcr tcpdump filters - Not Turing complete, but - Complex and arbitrary filters: tcpdump src 10.0.2.4 and (dst port 3389 or 22) tcpdump portrange 21-23 tcpdump 'tcp[13] & 1!=0' tcpdump 'tcp[32:4] = 0x47455420' (HTTP “GET ”)
9. @albcr tcpdump filters Incoming packet tcpdump socket AF_PACKET Local process (apache) input output routing decision forward post-routing pre-routing bpf filter port=22 socket AF_INET userspace kernel
10. @albcr Classic BPF (1992) -> extended BPF (2013) - Different kinds of BPF programs - Not just for tcpdump anymore - bpf() system call - BPF maps - BPF filesystem: /sys/fs/bpf
11. @albcr (e)BPF in a nutshell
12. @albcr ...attached to different objects Different kinds of BPF programs... Networking Security Tracing sockets (tcpdump) (Landlock), KRSI kprobes cgroups seccomp tracepoints traffic control devices uprobes
13. @albcr The BPF verifier ensures it’s safe SEC("socket/0") int main_prog(struct __sk_buff *skb) { bpf_map_lookup_elem(...) return 0; } input parameters Output: return value BPF helper functions (e.g. lookup maps) no loops x x access to kernel memory
14. @albcr BPF Maps kernel userspace BPF program 1 BPF map Userspace program ∘ Keep context between calls ∘ Report to userspace API: syscall bpf(cmd, ...) - BPF_MAP_CREATE - BPF_MAP_LOOKUP_ELEM - ... API: helper functions - bpf_map_lookup_elem() - bpf_map_update_elem() - bpf_map_delete_elem() BPF program 2
15. @albcr BPF Maps - 25 different types today, including: - Hash table - Array - Queue - Perf ring buffers - Created with: - fd = bpf() system call - Command line: “bpftool map create”, visible on bpf filesystem (/sys/fs/bpf)
16. @albcr Example of code using maps
17. BPF Compiler Collection (BCC)
18. @albcr BCC - “BCC makes BPF programs easier to write, with kernel instrumentation in C (and includes a C wrapper around LLVM), and front-ends in Python and lua.” - Lots of examples and tools - https://github.com/iovisor/bcc - https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md
19. @albcr
20. @albcr BCC tools on github
21. @albcr Example with bashreadline.py
22. gobpf https://github.com/iovisor/gobpf
23. @albcr gobpf history - Some original code from PLUMgrid & IO Visor - In 2016: Kinvolk adds a feature with BPF in Weave Scope (in Golang and not Python) - Grew organically in features - Now 42 contributors, top 4 at Kinvolk - No releases yet. Currently recommend to vendor the library
24. @albcr gobpf contributions graph
25. @albcr gobpf: two libraries Library to create, load and use eBPF programs from Go • use Cgo + the BPF Compiler Collection (bcc) import "github.com/iovisor/gobpf/bcc" or • load and use pre-built elf object files import "github.com/iovisor/gobpf/elf"
26. DEMOS gobpf (bcc)
27. @albcr DEMO gobpf (bcc) • bashreadline.py but in Golang sudo -E go run examples/bcc/bash_readline/bash_readline.go • execsnoop but in Golang go build ./examples/bcc/execsnoop sudo ./execsnoop
28. DEMOS gobpf (elf)
29. @albcr DEMO gobpf (elf): tcptracer-bpf • Add kprobes on kernel functions: ◦ TCP connect ◦ TCP accept ◦ TCP close sudo ./tests/tracer https://github.com/weaveworks/tcptracer-bpf
30. @albcr DEMO gobpf (elf): traceloop • Tracing system calls in cgroups • Flight recorder with overwritable ring buffers sudo ./traceloop cgroups --dump-on-exit \ /sys/fs/cgroup/system.slice/sshd.service https://github.com/kinvolk/traceloop https://github.com/kinvolk/inspektor-gadget #inspektor-gadget on Kubernetes Slack
31. gobpf/elf API
32. @albcr Module Different structs / objects Kprobe Map Map Uprobe CgroupProgram SocketFilter SchedProgram TracepointProgram XDPProgram PerfMap Map • One module per ELF file • ELF files contains ELF sections ◦ For each map ◦ For each BPF program
33. @albcr type Module struct • Constructor func NewModule(fileName string) *Module func NewModuleFromReader(fileReader io.ReaderAt) *Module • Loading func (b *Module) Load(parameters map[string]SectionParams) error - Ring buffer size - Pinning config - Override max_entries Read an ELF file
34. @albcr type Module struct • Attaching func (b *Module) EnableKprobe(secName string, maxactive int) error func (b *Module) EnableKprobes(maxactive int) error func (b *Module) EnableTracepoint(secName string) error ... • Getters func (b *Module) Map(name string) *Map func (b *Module) IterMaps() <-chan *Map func (b *Module) Kprobe(name string) *Kprobe func (b *Module) IterKprobes() <-chan *Kprobe ...
35. @albcr type Map struct • Generic read & write (hash table, array...) func (b *Module) LookupElement(mp *Map, key, value unsafe.Pointer) error func (b *Module) LookupNextElement(mp *Map, key, nextKey, value unsafe.Pointer) (bool, error) func (b *Module) UpdateElement(mp *Map, key, value unsafe.Pointer, flags uint64) error func (b *Module) DeleteElement(mp *Map, key unsafe.Pointer) error • Low-level functions func (m *Map) Fd() int
36. @albcr type PerfMap struct • Helper to read the ring buffer using Go channels • Constructor func InitPerfMap(b *Module, mapName string, receiverChan chan []byte, lostChan chan uint64) (*PerfMap, error) • Ring buffer features func (pm *PerfMap) SetTimestampFunc(timestamp func(*[]byte) uint64) func (pm *PerfMap) PollStart() func (pm *PerfMap) PollStop()
37. @albcr Module Flexible model Kprobe Map Map Kprobe Module Kprobe Map Map Kprobe • Different programs can reference the same map • Including between different modules via pinning
38. @albcr Module “main.o” Real example with traceloop tail_call_enter tracepoint__sys_enter tracepoint__sys_exit tail_call_exit cgroup_map Module “tailcall.o” (1) events tracepoint__sys_enter tracepoint__sys_exit probe_at_sys_exit Module “tailcall.o” (2) events tracepoint__sys_enter tracepoint__sys_exit probe_at_sys_exit ...
39. gobpf/bcc API
40. @albcr type Module struct • Constructor func NewModule(code string, cflags []string) *Module source code in C
41. Future features
42. @albcr Examples of missing features - Better debug information from the verifier - DWARF debug information in ELF files - Access to kernel structs between different versions - BTF (BPF Type Format) - Support for more BPF maps - Example: maps of maps - Support for more BPF programs - Example: Linux Infrared Remote Control (LIRC)
43. Alban Crequy Github: alban Twitter: albcr Email: alban@kinvolk.io Kinvolk Blog: kinvolk.io/blog Github: kinvolk Twitter: kinvolkio Email: hello@kinvolk.io Slides: https://tinyurl.com/godays-gobpf Thank you!