Slide 1

Slide 1 text

R&D Center, Sony Group Corporation Copyright 2021 Sony Group Corporation Linux Tracing Technologies for Rust Information Exchange Meeting for Container Technologies part 15 Kenta Tada

Slide 2

Slide 2 text

About me ⚫ Kenta Tada ⚫ System Software Engineer, R&D Center, Sony Group Corporation 2

Slide 3

Slide 3 text

Agenda ⚫ Linux Tracing Technologies ⚫ Uprobes ⚫ USDT ⚫ Rust with Linux Tracers ⚫ Container Runtime Debug 3

Slide 4

Slide 4 text

Environment ⚫ Linux Kernel v5.14.9 ⚫ rustc 1.55.0 4

Slide 5

Slide 5 text

Linux Tracing Technologies 5

Slide 6

Slide 6 text

Overview 6 https://sched.co/TPGS

Slide 7

Slide 7 text

Summary ⚫ Data sources • Kprobes • Tracepoints • Uprobes • Uprobes + USDT ⚫ Tools • Perf • BCC(BPF Compiler Collection) • bpftrace • SystemTap 7

Slide 8

Slide 8 text

Uprobes 8

Slide 9

Slide 9 text

Uprobes ⚫ Uprobes : User-space Probes ⚫ Uprobes helps us to trace applications without modifying source codes. ⚫ If you investigate the offset of the probe point, you can probe functions. ⚫ Uretprobes provides the probe point at the exit of functions. 9

Slide 10

Slide 10 text

Uprobes interfaces ⚫ ftrace’s interface • Ex. /sys/kernel/debug/tracing/uprobe_events ⚫ perf_event_open(2) • perf tools • eBPF tools ⚫ uprobe_register() in the kernel • int uprobe_register(struct inode *inode, loff_t offset, struct uprobe_consumer *uc); • Kernel modules can use uprobe_register() to register a probe. – Ex. https://qiita.com/kentaost/items/1c749012d21fb2c8745e 10

Slide 11

Slide 11 text

How to use uprobe_register() 11 #define DEBUGGEE_FILE "/home/kentaost/debuggee_app" #define DEBUGGEE_FILE_OFFSET (0x526) … (skipped) static int __init init_uprobe_sample(void) { int ret; struct path path; ret = kern_path(DEBUGGEE_FILE, LOOKUP_FOLLOW, &path); … (skipped) debuggee_inode = igrab(path.dentry->d_inode); path_put(&path); ret = uprobe_register(debuggee_inode, DEBUGGEE_FILE_OFFSET, &uc); https://qiita.com/kentaost/items/1c749012d21fb2c8745e ⚫ Uprobes need inode and offset. • Many tools help to transform a file name into inode easily.

Slide 12

Slide 12 text

How to probe functions ⚫ Breakpoint instruction is used to probe functions. ⚫ After the handlers are executed, kernel will single-step the original instruction. 12 https://dev.framing.life/tracing/uprobes-and-int3-insn/

Slide 13

Slide 13 text

How to set up a breakpoint for Uprobes ⚫ Existing processes before the prove point is registered • When uprobe_register() is called, register_for_each_vma() inserts breakpoints in existing processes. ⚫ New processes after the prove point is registered • uprobe_mmap() inserts breakpoints in new processes. • Ex. mmap(2) → mmap_region() in the kernel → uprobe_mmap() in the kernel 13

Slide 14

Slide 14 text

Use Case : SSL sniffer ⚫ Set uprobes on SSL_write() and SSL_read() in TLS Library. 14 https://blog.px.dev/ebpf-openssl-tracing/

Slide 15

Slide 15 text

Demo : How the breakpoint is installed for Uprobes?? 15

Slide 16

Slide 16 text

USDT 16

Slide 17

Slide 17 text

USDT ⚫ USDT : Userland Statically Defined Tracing ⚫ USDT probes provide applications with static tracing markers. ⚫ You need to add markers in the source code manually. ⚫ You need to investigate the offset of markers from .note.stapsdt section when you trace applications. • Finally, Uprobes are used to probe the probe point. 17

Slide 18

Slide 18 text

USDT interfaces ⚫ Add a marker in your source code • Install a package – Ex. # apt install systemtap-sdt-dev • Write a header in the C source code – #include • Write a provider at the location when you want to probe – DTRACE_PROBE2(provider, name, arg1, arg2) ⚫ When the code is compiled, USDT probes will be nop. 18 [C source code] … DTRACE_PROBE2(…) Compile [ELF Binary] … nop … .note.stapsdt

Slide 19

Slide 19 text

How to use USDT 1. When the program is loaded at first, the instruction is nop. 2. The tracer program reads the .note.stapsdt section of the tracee program. 3. The tracer program changes the instruction from nop to breakpoint using Uprobes. 19

Slide 20

Slide 20 text

Example : Slow queries ⚫ Some applications have already supported USDT. • MySQL, PostgreSQL, Node.js and so on. ⚫ dbslower.py from BCC is a tool to investigate slow queries. • query__start • query__done ⚫ dbslower.py tries to use Uprobes only if MySQL is without USDT support. • https://github.com/iovisor/bcc/pull/1239 20

Slide 21

Slide 21 text

Rust with Linux Tracers 21

Slide 22

Slide 22 text

What we need to use Uprobes/USDT for Rust ⚫ Mangling ⚫ USDT for Rust 22

Slide 23

Slide 23 text

Mangling ⚫ Rust symbols are mangled in the binary except using the no_mangle attribute. ⚫ You need to demangle symbols to find the offset of the specified function before using Uprobes. ⚫ Tools • objdump -Ct • https://github.com/rust-lang/rustc-demangle 23

Slide 24

Slide 24 text

USDT for Rust ⚫ All you need is to inject USDT probes into your ELF binary. ⚫ Some tools have been already provided. • Ex. https://github.com/cuviper/rust-libprobe 24

Slide 25

Slide 25 text

Container Runtime Debug 25

Slide 26

Slide 26 text

Container Runtime is complex 26 User space Kernel space cgroup Low-Level Container Runtime systemd cgroupfs driver systemd driver D-Bus High-Level Container RUntime Kubelet cgroupfs driver systemd driver D-Bus

Slide 27

Slide 27 text

Demo : Trace the value of CPU shares without modification 27

Slide 28

Slide 28 text

Key takeaways ⚫ Uprobes is a great technology to observe applications. ⚫ USDT enhance the traceability of Uprobes. ⚫ Make Rust libraries more traceable using kernel technologies. 28

Slide 29

Slide 29 text

SONY is a registered trademark of Sony Group Corporation. Names of Sony products and services are the registered trademarks and/or trademarks of Sony Group Corporation or its Group companies. Other company names and product names are registered trademarks and/or trademarks of the respective companies.