gobpf - utilizing BPF from Go - Alban Crequy - Kinvolk

Slide 1

Slide 1 text

gobpf - utilizing BPF from Go GoDays Berlin | 23.01.2020 https://tinyurl.com/godays-gobpf

Slide 2

Slide 2 text

Hi, I'm Alban Alban Crequy CTO, Kinvolk Github: alban Twitter: @albcr Email: [email protected]

Slide 3

Slide 3 text

The Kubernetes Linux Experts Engineering services and products for Kubernetes, containers, process management and Linux user-space + kernel Blog: kinvolk.io/blog Github: kinvolk Twitter: kinvolkio Email: [email protected] Kinvolk

Slide 4

Slide 4 text

@albcr Plan - Introduction to BPF - BPF Compiler Collection (BCC) - gobpf - Demos - Future features

Slide 5

Slide 5 text

Introduction to BPF

Slide 6

Slide 6 text

@albcr Intro to Berkeley Packet Filter - BPF: bytecode executed in the Linux kernel - Initially for tcpdump (1992) - Extended BPF (2013)

Slide 7

Slide 7 text

@albcr How tcpdump captures network packets Local process (apache) input output routing decision forward post-routing pre-routing Incoming packet tcpdump socket AF_PACKET copy socket AF_INET

Slide 8

Slide 8 text

@albcr tcpdump ﬁlters - Not Turing complete, but - Complex and arbitrary ﬁlters: tcpdump src 10.0.2.4 and (dst port 3389 or 22) tcpdump portrange 21-23 tcpdump 'tcp[13] & 1!=0' tcpdump 'tcp[32:4] = 0x47455420' (HTTP “GET ”)

Slide 9

Slide 9 text

@albcr tcpdump ﬁlters Incoming packet tcpdump socket AF_PACKET Local process (apache) input output routing decision forward post-routing pre-routing bpf filter port=22 socket AF_INET userspace kernel

Slide 10

Slide 10 text

@albcr Classic BPF (1992) -> extended BPF (2013) - Different kinds of BPF programs - Not just for tcpdump anymore - bpf() system call - BPF maps - BPF ﬁlesystem: /sys/fs/bpf

Slide 11

Slide 11 text

@albcr (e)BPF in a nutshell

Slide 12

Slide 12 text

@albcr ...attached to different objects Diﬀerent kinds of BPF programs... Networking Security Tracing sockets (tcpdump) (Landlock), KRSI kprobes cgroups seccomp tracepoints trafﬁc control devices uprobes

Slide 13

Slide 13 text

@albcr The BPF veriﬁer ensures it’s safe SEC("socket/0") int main_prog(struct __sk_buff *skb) { bpf_map_lookup_elem(...) return 0; } input parameters Output: return value BPF helper functions (e.g. lookup maps) no loops x x access to kernel memory

Slide 14

Slide 14 text

@albcr BPF Maps kernel userspace BPF program 1 BPF map Userspace program ∘ Keep context between calls ∘ Report to userspace API: syscall bpf(cmd, ...) - BPF_MAP_CREATE - BPF_MAP_LOOKUP_ELEM - ... API: helper functions - bpf_map_lookup_elem() - bpf_map_update_elem() - bpf_map_delete_elem() BPF program 2

Slide 15

Slide 15 text

@albcr BPF Maps - 25 different types today, including: - Hash table - Array - Queue - Perf ring buffers - Created with: - fd = bpf() system call - Command line: “bpftool map create”, visible on bpf ﬁlesystem (/sys/fs/bpf)

Slide 16

Slide 16 text

@albcr Example of code using maps

Slide 17

Slide 17 text

BPF Compiler Collection (BCC)

Slide 18

Slide 18 text

@albcr BCC - “BCC makes BPF programs easier to write, with kernel instrumentation in C (and includes a C wrapper around LLVM), and front-ends in Python and lua.” - Lots of examples and tools - https://github.com/iovisor/bcc - https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md

Slide 19

Slide 19 text

@albcr

Slide 20

Slide 20 text

@albcr BCC tools on github

Slide 21

Slide 21 text

@albcr Example with bashreadline.py

Slide 22

Slide 22 text

gobpf https://github.com/iovisor/gobpf

Slide 23

Slide 23 text

@albcr gobpf history - Some original code from PLUMgrid & IO Visor - In 2016: Kinvolk adds a feature with BPF in Weave Scope (in Golang and not Python) - Grew organically in features - Now 42 contributors, top 4 at Kinvolk - No releases yet. Currently recommend to vendor the library

Slide 24

Slide 24 text

@albcr gobpf contributions graph

Slide 25

Slide 25 text

@albcr gobpf: two libraries Library to create, load and use eBPF programs from Go ● use Cgo + the BPF Compiler Collection (bcc) import "github.com/iovisor/gobpf/bcc" or ● load and use pre-built elf object ﬁles import "github.com/iovisor/gobpf/elf"

Slide 26

Slide 26 text

DEMOS gobpf (bcc)

Slide 27

Slide 27 text

@albcr DEMO gobpf (bcc) ● bashreadline.py but in Golang sudo -E go run examples/bcc/bash_readline/bash_readline.go ● execsnoop but in Golang go build ./examples/bcc/execsnoop sudo ./execsnoop

Slide 28

Slide 28 text

DEMOS gobpf (elf)

Slide 29

Slide 29 text

@albcr DEMO gobpf (elf): tcptracer-bpf ● Add kprobes on kernel functions: ○ TCP connect ○ TCP accept ○ TCP close sudo ./tests/tracer https://github.com/weaveworks/tcptracer-bpf

Slide 30

Slide 30 text

@albcr DEMO gobpf (elf): traceloop ● Tracing system calls in cgroups ● Flight recorder with overwritable ring buffers sudo ./traceloop cgroups --dump-on-exit \ /sys/fs/cgroup/system.slice/sshd.service https://github.com/kinvolk/traceloop https://github.com/kinvolk/inspektor-gadget #inspektor-gadget on Kubernetes Slack

Slide 31

Slide 31 text

gobpf/elf API

Slide 32

Slide 32 text

@albcr Module Different structs / objects Kprobe Map Map Uprobe CgroupProgram SocketFilter SchedProgram TracepointProgram XDPProgram PerfMap Map ● One module per ELF file ● ELF files contains ELF sections ○ For each map ○ For each BPF program

Slide 33

Slide 33 text

@albcr type Module struct ● Constructor func NewModule(fileName string) *Module func NewModuleFromReader(fileReader io.ReaderAt) *Module ● Loading func (b *Module) Load(parameters map[string]SectionParams) error - Ring buffer size - Pinning conﬁg - Override max_entries Read an ELF ﬁle

Slide 34

Slide 34 text

@albcr type Module struct ● Attaching func (b *Module) EnableKprobe(secName string, maxactive int) error func (b *Module) EnableKprobes(maxactive int) error func (b *Module) EnableTracepoint(secName string) error ... ● Getters func (b *Module) Map(name string) *Map func (b *Module) IterMaps() <-chan *Map func (b *Module) Kprobe(name string) *Kprobe func (b *Module) IterKprobes() <-chan *Kprobe ...

Slide 35

Slide 35 text

@albcr type Map struct ● Generic read & write (hash table, array...) func (b *Module) LookupElement(mp *Map, key, value unsafe.Pointer) error func (b *Module) LookupNextElement(mp *Map, key, nextKey, value unsafe.Pointer) (bool, error) func (b *Module) UpdateElement(mp *Map, key, value unsafe.Pointer, flags uint64) error func (b *Module) DeleteElement(mp *Map, key unsafe.Pointer) error ● Low-level functions func (m *Map) Fd() int

Slide 36

Slide 36 text

@albcr type PerfMap struct ● Helper to read the ring buffer using Go channels ● Constructor func InitPerfMap(b *Module, mapName string, receiverChan chan []byte, lostChan chan uint64) (*PerfMap, error) ● Ring buffer features func (pm *PerfMap) SetTimestampFunc(timestamp func(*[]byte) uint64) func (pm *PerfMap) PollStart() func (pm *PerfMap) PollStop()

Slide 37

Slide 37 text

@albcr Module Flexible model Kprobe Map Map Kprobe Module Kprobe Map Map Kprobe ● Different programs can reference the same map ● Including between different modules via pinning

Slide 38

Slide 38 text

@albcr Module “main.o” Real example with traceloop tail_call_enter tracepoint__sys_enter tracepoint__sys_exit tail_call_exit cgroup_map Module “tailcall.o” (1) events tracepoint__sys_enter tracepoint__sys_exit probe_at_sys_exit Module “tailcall.o” (2) events tracepoint__sys_enter tracepoint__sys_exit probe_at_sys_exit ...

Slide 39

Slide 39 text

gobpf/bcc API

Slide 40

Slide 40 text

@albcr type Module struct ● Constructor func NewModule(code string, cflags []string) *Module source code in C

Slide 41

Slide 41 text

Future features

Slide 42

Slide 42 text

@albcr Examples of missing features - Better debug information from the veriﬁer - DWARF debug information in ELF ﬁles - Access to kernel structs between different versions - BTF (BPF Type Format) - Support for more BPF maps - Example: maps of maps - Support for more BPF programs - Example: Linux Infrared Remote Control (LIRC)

Slide 43

Slide 43 text

Alban Crequy Github: alban Twitter: albcr Email: [email protected] Kinvolk Blog: kinvolk.io/blog Github: kinvolk Twitter: kinvolkio Email: [email protected] Slides: https://tinyurl.com/godays-gobpf Thank you!