R&D Center, Sony Group Corporation Copyright 2021 Sony Group Corporation Linux Tracing Technologies for Rust Information Exchange Meeting for Container Technologies part 15 Kenta Tada

About me ⚫ Kenta Tada ⚫ System Software Engineer, R&D Center, Sony Group Corporation 2

Agenda ⚫ Linux Tracing Technologies ⚫ Uprobes ⚫ USDT ⚫ Rust with Linux Tracers ⚫ Container Runtime Debug 3

Environment ⚫ Linux Kernel v5.14.9 ⚫ rustc 1.55.0 4

Linux Tracing Technologies 5

Overview 6 https://sched.co/TPGS

Summary ⚫ Data sources • Kprobes • Tracepoints • Uprobes • Uprobes + USDT ⚫ Tools • Perf • BCC(BPF Compiler Collection) • bpftrace • SystemTap 7

Uprobes 8

Uprobes ⚫ Uprobes : User-space Probes ⚫ Uprobes helps us to trace applications without modifying source codes. ⚫ If you investigate the offset of the probe point, you can probe functions. ⚫ Uretprobes provides the probe point at the exit of functions. 9

Uprobes interfaces ⚫ ftrace’s interface • Ex. /sys/kernel/debug/tracing/uprobe_events ⚫ perf_event_open(2) • perf tools • eBPF tools ⚫ uprobe_register() in the kernel • int uprobe_register(struct inode *inode, loff_t offset, struct uprobe_consumer *uc); • Kernel modules can use uprobe_register() to register a probe. – Ex. https://qiita.com/kentaost/items/1c749012d21fb2c8745e 10

How to use uprobe_register() 11 #define DEBUGGEE_FILE "/home/kentaost/debuggee_app" #define DEBUGGEE_FILE_OFFSET (0x526) … (skipped) static int __init init_uprobe_sample(void) { int ret; struct path path; ret = kern_path(DEBUGGEE_FILE, LOOKUP_FOLLOW, &path); … (skipped) debuggee_inode = igrab(path.dentry->d_inode); path_put(&path); ret = uprobe_register(debuggee_inode, DEBUGGEE_FILE_OFFSET, &uc); https://qiita.com/kentaost/items/1c749012d21fb2c8745e ⚫ Uprobes need inode and offset. • Many tools help to transform a file name into inode easily.

How to probe functions ⚫ Breakpoint instruction is used to probe functions. ⚫ After the handlers are executed, kernel will single-step the original instruction. 12 https://dev.framing.life/tracing/uprobes-and-int3-insn/

How to set up a breakpoint for Uprobes ⚫ Existing processes before the prove point is registered • When uprobe_register() is called, register_for_each_vma() inserts breakpoints in existing processes. ⚫ New processes after the prove point is registered • uprobe_mmap() inserts breakpoints in new processes. • Ex. mmap(2) → mmap_region() in the kernel → uprobe_mmap() in the kernel 13

Use Case : SSL sniffer ⚫ Set uprobes on SSL_write() and SSL_read() in TLS Library. 14 https://blog.px.dev/ebpf-openssl-tracing/

Demo : How the breakpoint is installed for Uprobes?? 15

USDT 16

USDT ⚫ USDT : Userland Statically Defined Tracing ⚫ USDT probes provide applications with static tracing markers. ⚫ You need to add markers in the source code manually. ⚫ You need to investigate the offset of markers from .note.stapsdt section when you trace applications. • Finally, Uprobes are used to probe the probe point. 17

USDT interfaces ⚫ Add a marker in your source code • Install a package – Ex. # apt install systemtap-sdt-dev • Write a header in the C source code – #include • Write a provider at the location when you want to probe – DTRACE_PROBE2(provider, name, arg1, arg2) ⚫ When the code is compiled, USDT probes will be nop. 18 [C source code] … DTRACE_PROBE2(…) Compile [ELF Binary] … nop … .note.stapsdt

How to use USDT 1. When the program is loaded at first, the instruction is nop. 2. The tracer program reads the .note.stapsdt section of the tracee program. 3. The tracer program changes the instruction from nop to breakpoint using Uprobes. 19

Example : Slow queries ⚫ Some applications have already supported USDT. • MySQL, PostgreSQL, Node.js and so on. ⚫ dbslower.py from BCC is a tool to investigate slow queries. • query__start • query__done ⚫ dbslower.py tries to use Uprobes only if MySQL is without USDT support. • https://github.com/iovisor/bcc/pull/1239 20

Rust with Linux Tracers 21

What we need to use Uprobes/USDT for Rust ⚫ Mangling ⚫ USDT for Rust 22

Mangling ⚫ Rust symbols are mangled in the binary except using the no_mangle attribute. ⚫ You need to demangle symbols to find the offset of the specified function before using Uprobes. ⚫ Tools • objdump -Ct • https://github.com/rust-lang/rustc-demangle 23

USDT for Rust ⚫ All you need is to inject USDT probes into your ELF binary. ⚫ Some tools have been already provided. • Ex. https://github.com/cuviper/rust-libprobe 24

Container Runtime Debug 25

Container Runtime is complex 26 User space Kernel space cgroup Low-Level Container Runtime systemd cgroupfs driver systemd driver D-Bus High-Level Container RUntime Kubelet cgroupfs driver systemd driver D-Bus

Demo : Trace the value of CPU shares without modification 27

Key takeaways ⚫ Uprobes is a great technology to observe applications. ⚫ USDT enhance the traceability of Uprobes. ⚫ Make Rust libraries more traceable using kernel technologies. 28

SONY is a registered trademark of Sony Group Corporation. Names of Sony products and services are the registered trademarks and/or trademarks of Sony Group Corporation or its Group companies. Other company names and product names are registered trademarks and/or trademarks of the respective companies.