eBPF’s Abilities and Limitations: The Truth Liz Rice | @lizrice Chief Open Source Officer, Isovalent Emeritus Chair, CNCF TOC CNCF & OpenUK boards John Fastabend | @jrfastab Principal Software Engineer, Isovalent

userspace kernel app event system calls eBPF program eBPF lets us run custom code in the kernel

From ebpf.io

Are these statements true? “Implementing HTTP/2, or terminating TLS, are simply not possible in eBPF: they’re too complex” “eBPF is not Turing complete” Hint: these were all said by people who don’t work on eBPF

Turing completeness This is not a computer science class Can we use eBPF for arbitrarily complex tasks? ● Ability to process arbitrarily large amounts of data ● Unbounded loops ● Memory for storing state

Turing completeness Turing complete examples: All the common general purpose programming languages. C/C++, Java, Python, Go But also lots of other less practical (for programmers) environments. One Instruction Set Computer (OISC), X86 MMU, Conway’s Game of Life Some domains benefit from bounded computation: An HTTP parser guaranteeing no packet can DDOS the environment Syscall/Kernel tracing DSL with bounded/enforced maximum overhead

Conway’s Game Of Life This is not a computer science class Can we use eBPF for Conway’s Game Of Life? From playgameoflife.com

userspace kernel app event system calls eBPF program 🔍 verifier eBPF verifier ensures safety

userspace kernel app event system calls eBPF program 🔍 verifier eBPF verifier core safety principles 1. Read memory only with correct permissions 2. All writes to valid and safe memory 3. Valid in-bounds and well formed control flow 4. BPF execution on-cpu time is bounded: a. program sleeps b. schedules callback, iterators, next event c. program completes 5. Acquire/Release and reference count semantics a. Locks, RCU, socket ref, …

Conway’s Game of Life in eBPF

eBPF game of life program static int do_game(void) { copy_cellmap(); next_generation (); send_update(); return run_next(2); } int nextX(uint cell) { uint x, count; uint k = 0; struct cellmap *m; char *cellp; m = lookup_elem(&b, &k); if (!m) return -1; cellp = (char *)m->temp; for (x = 0; x < W; x++) { if (cell_off > MAX) continue; cell++; } return cell; } int next_generation(void) { uint cell; uint y; cell = 0; for (y=0; y < H; y++) cell = nextX(cell); return 0; }

Are eBPF programs limited to a few instructions? Verifier has limits to ensure verifier completes in reasonable amount of time: 1. (sub)program Instruction limit 2. Verifier step limit a. DFS traversal to check program DAG, unbounded loops, and jumps 4.14 4.19 5.4 5.15 6.1 unpriv 4096 4096 4096 4096 4096 privilege 4096 4096 1M 1M 1M

Consider 1M instruction limit for ‘modern’ stable kernels: ● Doom: ~<100k instructions ● Envoy: ~15M instructions ● Clang: ~26M instructions 4.14 4.19 5.4 5.15 6.1 unpriv 4096 4096 4096 4096 4096 privilege 4096 4096 1M 1M 1M Are eBPF programs limited to a few instructions?

Consider 1M instruction limit for ‘modern’ stable kernels: ● Doom: ~<100k instructions ● Envoy: ~15M instructions ● Clang: ~26M instructions 4.14 4.19 5.4 5.15 6.1 unpriv 4096 4096 4096 4096 4096 privilege 4096 4096 1M 1M 1M 1M << 15M Stay tuned, we will address this shortly But what about loops John? Are eBPF programs limited to a few instructions?

Can eBPF programs only jump forwards? No loops → write out instructions several times → unroll compiler directive Basic math shows this can be problematic: 64 X 64 Grid ->4096 “cells” to visit each iteration 1M / 4K -> 250 iterations before 1M limit int next_generation(void) { uint cell; uint y; cell = 0; for (y=0; y < H; y++) cell = nextX(cell); return 0; }

No loops → write out instructions several times → unroll compiler directive Basic math shows this can be problematic: 64 X 64 Grid ->4096 “cells” to visit each iteration 1M / 4K -> 250 iterations before 1M limit Not even an approximation of a forever tape !!! int next_generation(void) { uint cell; uint y; cell = 0; for (y=0; y < H; y++) cell = nextX(cell); return 0; } Can eBPF programs only jump forwards?

No loops → write out instructions several times → unroll compiler directive → loops Verifier ensures loops terminate in finite steps → bpf_loop(u32 nr, void *callback, void *ctx, u64 flags) Insert “intrinsic” loop logic verifier inserts counter int next_generation(void) { uint cell; uint y; cell = 0; for (y=0; y < H; y++) cell = nextX(cell); return 0; } If we bound our loops or use bpf helper to insert bounds, we can loop for (y=0; y < H; y++) Can eBPF programs only jump forwards?

No loops → write out instructions several times → unroll compiler directive → loops Verifier ensures loops terminate in finite steps → bpf_loop(u32 nr, void *callback, void *ctx, u64 flags) Insert “intrinsic” loop logic verifier inserts counter 4.14 4.19 5.4 5.15 6.1 Loops unroll unroll loop* loops bpf_loop int next_generation(void) { uint cell; uint y; cell = 0; for (y=0; y < H; y++) cell = nextX(cell); return 0; } Can eBPF programs only jump forwards?

No loops → write out instructions several times → unroll compiler directive → loops Verifier ensures loops terminate in finite steps → bpf_loop(u32 nr, void *callback, void *ctx, u64 flags) Insert “intrinsic” loop logic verifier inserts counter 4.14 4.19 5.4 5.15 6.1 Loops unroll unroll loop* loops bpf_loop 6.9 may_goto int next_generation(void) { uint cell; uint y; cell = 0; for (y=0; y < H; y++) cell = nextX(cell); return 0; } Can eBPF programs only jump forwards?

Are eBPF programs limited to a few instructions? Consider 1M instruction limit for ‘modern’ stable kernels: ● Doom: ~<100k instructions ● Envoy: ~15M instructions ● Clang: ~26M instructions 4.14 4.19 5.4 5.15 6.1 privilege 4096 4096 1M 1M 1M 1M << 15M But what about loops John? Verifier has limits to ensure verifier completes in reasonable amount of time: 1. (sub)program Instruction limit 2. Verifier step limit

Are eBPF programs limited to a few M++ instructions? Verifier has limits to ensure verifier completes in reasonable amount of time: 1. (sub)program Instruction limit 2. Verifier step limit subprogram ~function Verifier operates on subprograms. Max subprograms is an internal detail, but (tail calls) limit is 31 pushing to at least 31M instructions Clang: ~26M instructions can back of napkin fit inside a BPF Linux runtime.

Are eBPF programs limited to a few M++ instructions? Verifier has limits to ensure verifier completes in reasonable amount of time: 1. (sub)program Instruction limit 2. Verifier step limit subprogram ~function Verifier operates on subprograms. Max subprograms is an internal detail, but (tail calls) limit is 31 pushing to at least 31M instructions Clang: ~26M instructions can back of napkin fit inside a BPF Linux runtime. Caution 1: Writing C-like code the compiler, verifier and human can all agree on at these scales requires skill and practice. Good news, core group of folks are working hard to make this easier.

Are eBPF programs limited to a few M++ instructions? Verifier has limits to ensure verifier completes in reasonable amount of time: 1. (sub)program Instruction limit 2. Verifier step limit subprogram ~function Verifier operates on subprograms. Max subprograms is an internal detail, but (tail calls) limit is 31 pushing to at least 31M instructions Clang: ~26M instructions can back of napkin fit inside a BPF Linux runtime. Caution 2: This may become addictive and you may try to start trying to port doom, quake, tetris and other artifacts into BPF.

Do eBPF programs have to terminate? Yes and No: BPF execution on-cpu time is bounded the BPF program must release the CPU. How can a BPF program release the CPU: ● Reach end of the BPF program ● Iterator programs are specially designed to walk arbitrary lists of objects v5.8 ● Sleep on page fault or other memory operation. v5.10 ● Timer callback (restart my program at funcA at time X, X may be 0). v5.15 static int do_game(void) { copy_cellmap(); next_generation (); send_update(); return run_next(2); }

Can eBPF programs allocate memory? Maps as memory management system. int nextX(uint cell) { uint x, count; uint k = 0; struct cellmap *m; char *cellp; m = lookup_elem(&b, &k); if (!m) return -1; [...] } BPF array maps can be arbitrarily large as long as they fit inside system memory limits. 6.9 will introduce bpf_arena() a 4GB 32bit pointer vaddr space.

Demo: Conway’s Game of Life in eBPF

eBPF can handle arbitrarily complex tasks

Are these statements true? No! “Implementing HTTP/2, or terminating TLS, are simply not possible in eBPF: they’re too complex” “eBPF is not Turing complete” They might not be implemented today, but they are possible

Thank you ebpf.io isovalent.com Download from isovalent.com Book signing Booth E5 13:00 today