bpftrace: a swiss army knife tracing tool — SUSE Labs Conference 2026

This talk will give a quick introduction to bpftrace, an awk-like scripting language powered by BPF that specializes in tracing. We'll go through its concept, syntax, and provide a few example usage to see it in action, and finally close with a comparison with other existing tracing tools (bcc, ftrace, LTTng, systemtap, to name a few).

Transcript

1. bpftrace - a swiss army knife tracing tool Shung-Hsi Yu SUSE Labs Conference 2026
2. - “Sales pitch” – add bpftrace to your toolkit - little experience with bpftrace on real-world problem - showing on subsystems I know little about - Not the best tool for every job 2 Disclaimer
3. Why use bpftrace when I can use ? 3
4. 4 Introduction to bpftrace
5. - bpftrace - Dtrace for Linux / Systemtap - Awk-like language for tracing 5 What is it?
6. - bpftrace language 6 What is it? probe { action }
7. - bpftrace language 7 What is it? tracepoint:syscalls:sys_enter_openat { printf("openat() called"); }
8. - tracepoint - fprobe (fentry, fexit) - kprobe - uprobe - interval, profile, watchpoint - BEGIN, END 8 Probes
9. - args gives access to tracepoint arguments 9 Built-in variables tracepoint:syscalls:sys_enter_openat { printf("openat() called on %s", str(args.filename)); }
10. - Key-value store for persisting data 10 Maps tracepoint:syscalls:sys_enter_openat { @filenames[str(args.filename)] = true; }
11. 11 Finding correlations
12. - Capture the filename at entry 12 Function entry vs exit tracepoint:syscalls:sys_enter_openat { @filenames[tid] = str(args.filename); }
13. - Print filename with return code 13 Function entry vs exit tracepoint:syscalls:sys_exit_openat { printf("openat(%s) = %d\n", @filenames[tid], args.ret); }
14. - Print filename with return code 14 Function entry vs exit fentry:vfs_read { @f[tid] = args.file; } fexit:vfs_read { printf("...", @f[tid], args.ret); }
15. - Calculate execution time 15 Function entry vs exit fentry:vfs_read { @start[tid] = nsecs; } fexit:vfs_read { printf("Took %d us", nsecs - @start[tid] / 1000); }
16. - Print filename with return code 16 Async submission & completion t:block:block_bio_queue { @start[args.sector] = nsec; } t:block:block_bio_complete { $lat = nsecs - @start[args.sector]; }
17. 17 Filtering
18. - Process name, PID, cgroup 18 Basic filtering t:block:block_bio_queue { if (comm != "git") return; ...
19. - Process name, PID, cgroup 19 Basic filtering t:block:block_bio_queue /comm == "git"/ { ... }
20. - Process name, PID, cgroup 20 Basic filtering t:block:block_bio_queue /pid == 1234/ { ... }
21. - Process name, PID, cgroup 21 Basic filtering t:block:block_bio_queue /cgroup == cgroupid("/sys/fs/cgroup/...")/ { ... }
22. - Looking only at the slow ones 22 Latency t:block:block_bio_complete { $us = nsecs - @start[args.sector]/1000; if ($us > 100) { ...
23. - Current execution environment 23 Context kprobe:do_fault { @faulted[cpu] = true } kretprobe:do_fault { delete(@faulted, cpu); }
24. - Current execution environment 24 Context ... t:kmem:kmalloc /@faulted[cpu]/ { ... // kmalloc called during do_fault
25. 25 Aggregation
26. - Not realistic for frequently called function 26 Per-event output t:kmem:kmalloc /@faulted[cpu]/ { printf("kmalloc called!\n"); }
27. - Count the number times 27 Basic aggregation t:kmem:kmalloc { @c = count(); }
28. - Find the total alloc size 28 Basic aggregation t:kmem:kmalloc { @total_alloc = sum(args.bytes_alloc); }
29. - Find the number of time grouped by alloc size 29 Grouping t:kmem:kmalloc { @by_alloc[args.bytes_alloc] = count(); }
30. - Current execution environment 30 Visualization t:kmem:kmalloc { @ = hist(args.bytes_alloc); }
31. - Current execution environment 31 Aggregate on stack t:kmem:kmalloc { @by_stack[kstack] = sum(args.bytes...); }
32. 32 Live demo
33. 33 Less common uses
34. - Directly reading CPU register value at specific instruction (see “how to use bpftrace probe…”) 34 kprobe:do_truncate+109 { printf(" %x\n", reg("sp")); } Reading local variable
35. - Monitor for memory read/write 35 watchpoint::0x100000000:8:rw { printf(“hit!”\n”); } Watchpoint
36. - Let busy spin, useful for making race condition more reproducible 36 Make things go slower fentry:vfs_write /comm == "dd"/ { for ($x : 1..999999) { @++; }
37. 37 Comparison
38. Hand-ons analysis with ~reproducible issue 1. Fast iteration 2. Probe & read anything* 3. Live data processing - filtering & aggregation 38 Strength of bpftrace
39. 39 Comparison aspects - Safety in production - Ease of use - Performance - Minimal dependencies - Fast iteration - Advance filtering - Always-on / Kdump enhancement - Data dumping - Userspace tracking
40. - SystemTap - kernel module (kprobe) - printk() - dynamic debug - ftrace / trace-cmd - LTTng 40 Similar tools - perf - strace - ltrace - tcpdump - bcc
41. 41 Thank you!