$30 off During Our Annual Pro Sale. View Details »

gobpf - utilizing BPF from Go - Alban Crequy - Kinvolk

January 23, 2020

gobpf - utilizing BPF from Go - Alban Crequy - Kinvolk

BPF is a Linux in-kernel virtual machine that is used for tracing kernel functions, networking, performance analysis and more. In this talk I will give an introduction into eBPF and gobpf, a library under the IO Visor Project, and show how gobpf can be used to access and utilize BPF features from Go programs. We will also look at the design of gobpf and its use of the bcc framework and Cgo.


January 23, 2020

More Decks by GoDays

Other Decks in Technology


  1. gobpf - utilizing BPF from Go GoDays Berlin | 23.01.2020

  2. Hi, I'm Alban Alban Crequy CTO, Kinvolk Github: alban Twitter:

    @albcr Email: alban@kinvolk.io
  3. The Kubernetes Linux Experts Engineering services and products for Kubernetes,

    containers, process management and Linux user-space + kernel Blog: kinvolk.io/blog Github: kinvolk Twitter: kinvolkio Email: hello@kinvolk.io Kinvolk
  4. @albcr Plan - Introduction to BPF - BPF Compiler Collection

    (BCC) - gobpf - Demos - Future features
  5. Introduction to BPF

  6. @albcr Intro to Berkeley Packet Filter - BPF: bytecode executed

    in the Linux kernel - Initially for tcpdump (1992) - Extended BPF (2013)
  7. @albcr How tcpdump captures network packets Local process (apache) input

    output routing decision forward post-routing pre-routing Incoming packet tcpdump socket AF_PACKET copy socket AF_INET
  8. @albcr tcpdump filters - Not Turing complete, but - Complex

    and arbitrary filters: tcpdump src and (dst port 3389 or 22) tcpdump portrange 21-23 tcpdump 'tcp[13] & 1!=0' tcpdump 'tcp[32:4] = 0x47455420' (HTTP “GET ”)
  9. @albcr tcpdump filters Incoming packet tcpdump socket AF_PACKET Local process

    (apache) input output routing decision forward post-routing pre-routing bpf filter port=22 socket AF_INET userspace kernel
  10. @albcr Classic BPF (1992) -> extended BPF (2013) - Different

    kinds of BPF programs - Not just for tcpdump anymore - bpf() system call - BPF maps - BPF filesystem: /sys/fs/bpf
  11. @albcr (e)BPF in a nutshell

  12. @albcr ...attached to different objects Different kinds of BPF programs...

    Networking Security Tracing sockets (tcpdump) (Landlock), KRSI kprobes cgroups seccomp tracepoints traffic control devices uprobes
  13. @albcr The BPF verifier ensures it’s safe SEC("socket/0") int main_prog(struct

    __sk_buff *skb) { bpf_map_lookup_elem(...) return 0; } input parameters Output: return value BPF helper functions (e.g. lookup maps) no loops x x access to kernel memory
  14. @albcr BPF Maps kernel userspace BPF program 1 BPF map

    Userspace program ∘ Keep context between calls ∘ Report to userspace API: syscall bpf(cmd, ...) - BPF_MAP_CREATE - BPF_MAP_LOOKUP_ELEM - ... API: helper functions - bpf_map_lookup_elem() - bpf_map_update_elem() - bpf_map_delete_elem() BPF program 2
  15. @albcr BPF Maps - 25 different types today, including: -

    Hash table - Array - Queue - Perf ring buffers - Created with: - fd = bpf() system call - Command line: “bpftool map create”, visible on bpf filesystem (/sys/fs/bpf)
  16. @albcr Example of code using maps

  17. BPF Compiler Collection (BCC)

  18. @albcr BCC - “BCC makes BPF programs easier to write,

    with kernel instrumentation in C (and includes a C wrapper around LLVM), and front-ends in Python and lua.” - Lots of examples and tools - https://github.com/iovisor/bcc - https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md
  19. @albcr

  20. @albcr BCC tools on github

  21. @albcr Example with bashreadline.py

  22. gobpf https://github.com/iovisor/gobpf

  23. @albcr gobpf history - Some original code from PLUMgrid &

    IO Visor - In 2016: Kinvolk adds a feature with BPF in Weave Scope (in Golang and not Python) - Grew organically in features - Now 42 contributors, top 4 at Kinvolk - No releases yet. Currently recommend to vendor the library
  24. @albcr gobpf contributions graph

  25. @albcr gobpf: two libraries Library to create, load and use

    eBPF programs from Go • use Cgo + the BPF Compiler Collection (bcc) import "github.com/iovisor/gobpf/bcc" or • load and use pre-built elf object files import "github.com/iovisor/gobpf/elf"
  26. DEMOS gobpf (bcc)

  27. @albcr DEMO gobpf (bcc) • bashreadline.py but in Golang sudo

    -E go run examples/bcc/bash_readline/bash_readline.go • execsnoop but in Golang go build ./examples/bcc/execsnoop sudo ./execsnoop
  28. DEMOS gobpf (elf)

  29. @albcr DEMO gobpf (elf): tcptracer-bpf • Add kprobes on kernel

    functions: ◦ TCP connect ◦ TCP accept ◦ TCP close sudo ./tests/tracer https://github.com/weaveworks/tcptracer-bpf
  30. @albcr DEMO gobpf (elf): traceloop • Tracing system calls in

    cgroups • Flight recorder with overwritable ring buffers sudo ./traceloop cgroups --dump-on-exit \ /sys/fs/cgroup/system.slice/sshd.service https://github.com/kinvolk/traceloop https://github.com/kinvolk/inspektor-gadget #inspektor-gadget on Kubernetes Slack
  31. gobpf/elf API

  32. @albcr Module Different structs / objects Kprobe Map Map Uprobe

    CgroupProgram SocketFilter SchedProgram TracepointProgram XDPProgram PerfMap Map • One module per ELF file • ELF files contains ELF sections ◦ For each map ◦ For each BPF program
  33. @albcr type Module struct • Constructor func NewModule(fileName string) *Module

    func NewModuleFromReader(fileReader io.ReaderAt) *Module • Loading func (b *Module) Load(parameters map[string]SectionParams) error - Ring buffer size - Pinning config - Override max_entries Read an ELF file
  34. @albcr type Module struct • Attaching func (b *Module) EnableKprobe(secName

    string, maxactive int) error func (b *Module) EnableKprobes(maxactive int) error func (b *Module) EnableTracepoint(secName string) error ... • Getters func (b *Module) Map(name string) *Map func (b *Module) IterMaps() <-chan *Map func (b *Module) Kprobe(name string) *Kprobe func (b *Module) IterKprobes() <-chan *Kprobe ...
  35. @albcr type Map struct • Generic read & write (hash

    table, array...) func (b *Module) LookupElement(mp *Map, key, value unsafe.Pointer) error func (b *Module) LookupNextElement(mp *Map, key, nextKey, value unsafe.Pointer) (bool, error) func (b *Module) UpdateElement(mp *Map, key, value unsafe.Pointer, flags uint64) error func (b *Module) DeleteElement(mp *Map, key unsafe.Pointer) error • Low-level functions func (m *Map) Fd() int
  36. @albcr type PerfMap struct • Helper to read the ring

    buffer using Go channels • Constructor func InitPerfMap(b *Module, mapName string, receiverChan chan []byte, lostChan chan uint64) (*PerfMap, error) • Ring buffer features func (pm *PerfMap) SetTimestampFunc(timestamp func(*[]byte) uint64) func (pm *PerfMap) PollStart() func (pm *PerfMap) PollStop()
  37. @albcr Module Flexible model Kprobe Map Map Kprobe Module Kprobe

    Map Map Kprobe • Different programs can reference the same map • Including between different modules via pinning
  38. @albcr Module “main.o” Real example with traceloop tail_call_enter tracepoint__sys_enter tracepoint__sys_exit

    tail_call_exit cgroup_map Module “tailcall.o” (1) events tracepoint__sys_enter tracepoint__sys_exit probe_at_sys_exit Module “tailcall.o” (2) events tracepoint__sys_enter tracepoint__sys_exit probe_at_sys_exit ...
  39. gobpf/bcc API

  40. @albcr type Module struct • Constructor func NewModule(code string, cflags

    []string) *Module source code in C
  41. Future features

  42. @albcr Examples of missing features - Better debug information from

    the verifier - DWARF debug information in ELF files - Access to kernel structs between different versions - BTF (BPF Type Format) - Support for more BPF maps - Example: maps of maps - Support for more BPF programs - Example: Linux Infrared Remote Control (LIRC)
  43. Alban Crequy Github: alban Twitter: albcr Email: alban@kinvolk.io Kinvolk Blog:

    kinvolk.io/blog Github: kinvolk Twitter: kinvolkio Email: hello@kinvolk.io Slides: https://tinyurl.com/godays-gobpf Thank you!