R&D Center, Sony Group Corporation
Copyright 2021 Sony Group Corporation
Linux Tracing Technologies for Rust
Information Exchange Meeting for Container Technologies part 15
⚫ Kenta Tada
⚫ System Software Engineer, R&D Center, Sony Group
⚫ Linux Tracing Technologies
⚫ Rust with Linux Tracers
⚫ Container Runtime Debug
⚫ Linux Kernel v5.14.9
⚫ rustc 1.55.0
Linux Tracing Technologies
⚫ Data sources
• Uprobes + USDT
• BCC(BPF Compiler Collection)
⚫ Uprobes : User-space Probes
⚫ Uprobes helps us to trace applications without modifying
⚫ If you investigate the offset of the probe point, you can
⚫ Uretprobes provides the probe point at the exit of functions.
⚫ ftrace’s interface
• Ex. /sys/kernel/debug/tracing/uprobe_events
• perf tools
• eBPF tools
⚫ uprobe_register() in the kernel
• int uprobe_register(struct inode *inode, loff_t offset, struct
• Kernel modules can use uprobe_register() to register a probe.
– Ex. https://qiita.com/kentaost/items/1c749012d21fb2c8745e
How to use uprobe_register()
#define DEBUGGEE_FILE "/home/kentaost/debuggee_app"
#define DEBUGGEE_FILE_OFFSET (0x526)
static int __init init_uprobe_sample(void)
struct path path;
ret = kern_path(DEBUGGEE_FILE, LOOKUP_FOLLOW, &path);
debuggee_inode = igrab(path.dentry->d_inode);
ret = uprobe_register(debuggee_inode,
⚫ Uprobes need inode and offset.
• Many tools help to transform a file name into inode easily.
How to probe functions
⚫ Breakpoint instruction is used to probe functions.
⚫ After the handlers are executed, kernel will single-step the
How to set up a breakpoint for Uprobes
⚫ Existing processes before the prove point is registered
• When uprobe_register() is called, register_for_each_vma() inserts
breakpoints in existing processes.
⚫ New processes after the prove point is registered
• uprobe_mmap() inserts breakpoints in new processes.
→ mmap_region() in the kernel
→ uprobe_mmap() in the kernel
Use Case : SSL sniffer
⚫ Set uprobes on SSL_write() and SSL_read() in TLS Library.
Demo : How the breakpoint is installed for Uprobes??
⚫ USDT : Userland Statically Defined Tracing
⚫ USDT probes provide applications with static tracing
⚫ You need to add markers in the source code manually.
⚫ You need to investigate the offset of markers
from .note.stapsdt section when you trace applications.
• Finally, Uprobes are used to probe the probe point.
⚫ Add a marker in your source code
• Install a package
– Ex. # apt install systemtap-sdt-dev
• Write a header in the C source code
• Write a provider at the location when you want to probe
– DTRACE_PROBE2(provider, name, arg1, arg2)
⚫ When the code is compiled, USDT probes will be nop.
[C source code]
How to use USDT
1. When the program is loaded at first, the instruction is nop.
2. The tracer program reads the .note.stapsdt section of the
3. The tracer program changes the instruction from nop to
breakpoint using Uprobes.
Example : Slow queries
⚫ Some applications have already supported USDT.
• MySQL, PostgreSQL, Node.js and so on.
⚫ dbslower.py from BCC is a tool to investigate slow queries.
⚫ dbslower.py tries to use Uprobes only if MySQL is without
Rust with Linux Tracers
What we need to use Uprobes/USDT for Rust
⚫ USDT for Rust
⚫ Rust symbols are mangled in the binary except using the
⚫ You need to demangle symbols to find the offset of the
specified function before using Uprobes.
• objdump -Ct
USDT for Rust
⚫ All you need is to inject USDT probes into your ELF binary.
⚫ Some tools have been already provided.
• Ex. https://github.com/cuviper/rust-libprobe
Container Runtime Debug
Container Runtime is complex
Demo : Trace the value of CPU shares without modification
⚫ Uprobes is a great technology to observe applications.
⚫ USDT enhance the traceability of Uprobes.
⚫ Make Rust libraries more traceable using kernel
SONY is a registered trademark of Sony Group Corporation.
Names of Sony products and services are the registered trademarks and/or trademarks of Sony Group Corporation or its Group companies.
Other company names and product names are registered trademarks and/or trademarks of the respective companies.