Ruby x BPF in Action / RubyKaigi 2022

Google Slide version with demo movies working: https://docs.google.com/presentation/d/1z9fGEL2eordrMifVLh7DcHw8M9tfhJVZgTpkTbMfp-k/edit?usp=sharing

Transcript

1. Ruby x BPF in Action Uchio Kondo from Mirrativ, Inc. @ RubyKaigi 2022.09.09
2. Uchio Kondo Infra & Streaming team @ Mirrativ, Inc Speaker @ RubyKaigi 2016, 2018, 2019, 2021 “Hacker” Supporter @ Engineer’s Café Fukuoka Ruby & Rust enthuasist, Linux freak Live in Fukuoka
3. (After all,) What can BPF do? §1
4. e.g. Visualize “SYN queue” From cloudflare’s blog “SYN packet handling in the wild”: https://blog.cloudflare.com/syn-packet-handling-in-the-wild/
5. 2 servers in different config • Same server (WEBRick 1.7.0, ruby 3.1.2) ◦ • Same bench parameter: ◦ • Different value of net.core.somaxconn ◦ 4,096 vs 500, how this makes effect?
6. Compare results somaxconn = 500 somaxconn = 4096
7. Tool to Visualize “SYN queue”
8. C to Visualize “SYN queue”
9. Let’s watch queue satuation somaxconn = 500 somaxconn = 4096
10. demo
11. Now, you’ve completely understood BPF! …OK. Let me keep going.
12. Quick Introduction to BPF §2
13. Big picture first https://whimsical.com/bpf-ABAvCvJFLcSie2ML9ee2fn
14. #1 History
15. #1 History https://www.tcpdump.org/papers/bpf-usenix93.pdf
16. #1 History https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commi t/?id=bd4cf0ed331a275e9bf5a49e6d0fd55dffc551b8
17. #2 How it works
18. #2 How it works ref. https://speakerdeck.com/chikuwait/learn-ebpf?slide=17 by Yuki Nakata, 2020 emoji from https://github.com/twitter/twemoji/tree/master/assets (*) Very simplified Scripting Bytecode BPF VM BPF Map User Interface Collectiong Kernel Data… … or perf buffer, etc. The Userland The Kingdom of Kernel
19. #2 How it works BTF requires: Kernel version >= 5.6 && CONFIG_DEBUG_INFO_BTF should be enabled
20. #3 Comparison
21. #3 Comparison
22. #3 Comparison
23. #4 (Expanding) Use cases…
24. #4 (Expanding) Use cases… • BPF-based network & security for containers • the de facto Kubernetes threat detection engine using BPF
25. #4 (Expanding) Use cases… rbperf (https://github.com/javierhonduco/rbperf) by Jabier H. Coto with
26. RbBCC and BPF: Observe Everything §3
27. What is RbBCC? • A: BCC for Ruby (libbcc FFI binding for Ruby) • WHAT is BCC? ◦ BPF Compiler Collection: ◦ An SDK to make BPF tools, using Script Languages (Python/Lua supported officially) ◦ But - Ruby is not in its support list, so I’m developping I’m going to show How to use – How to write BPF Ruby codes.
28. RbBCC’s “4 Keys”
29. #1 kprobe • kprobe - (mainly) function trace in Linux Kernel • e.g. __ARCH_sys_execve() ◦ The substantial function called when execve(2) is invoked
30. Example RbBCC Program:
31. Example RbBCC Program: C Part: tracing function run inside kernel
32. Example RbBCC Program: Ruby Part: Load the C part above and Get feedback and data from BPF program inside kernel
33. demo
34. #1 kprobe • ex.2 tcp_v4_conn_request (the first demo)
35. Trace the connect • Also has 2 parts: ◦ BPF DSL in C ◦ Load program & handle data in Ruby
36. Return to first demo
37. #2 tracepoint (for kernel) • Different stuff from Ruby’s TracePoint class • A static entrypoint to trace kernel events • It won’t change in the future version of Linux ◦ kprobe traces an exported symbol of kernel, so it should be changed and maybe unstable.
38. #2 tracepoint (for kernel) • Tracing syscall invocations: • raw_syscalls/sys_enter • raw_syscalls/sys_exit
39. tracepoint demo • Example output (compared with strace -w):
40. • Example: tracing WEBrick (again): ◦ ruby: ◦ ab: • Tracing command: ◦ ruby: ◦ strace: FYI: Performance sideeffect
41. FYI: Performance sideeffect w/ RbBCC w/ strace
42. #3 uprobe • uprobe: A mechanism to attach to user space function calls
43. #3 uprobe • Using uprobe (and USDT afterwards) with ease, build a special Ruby binary with a specific option:
44. #3 uprobe • Tracing rb_str_new(const char *ptr, long len)
45. #3 uprobe • Tracing rb_str_new(const char *ptr, long len)
46. #3 uprobe • Collecting rb_str_new()’s: (function return timestamp - function entry timestamp) • This represents the latency of a function call • function entry = uprobe, function return = uretprobe
47. #3 uprobe • Example of rb_str_new()’s latency histogram: ruby -e ‘p “Hello”’ ruby --disable gems -e ‘p “Hello”’
48. #4 USDT • USDT: Userspace Statically Defined Tracepoint ◦ Probe points that an author of a program embedded in advance ◦ cf. uprobe traces real function call dynamically • USDT for uprobe is just as Tracepoint for kprobe Dynamic Static Kernel space kprobe tracepount User space uprobe USDT
49. #4 USDT • Ruby’s USDT (first for DTrace, but available via BPF in Linux) Japanese article: https://magazine.rubyist.net/articles/0041/0041-200Special-dtrace.html https://rubyreferences.github.io/rubyref/advanced/dtrace.html
50. #4 USDT • Example: USDTs about GC: ◦ usdt:./bin/ruby:ruby:gc__mark__begin ◦ usdt:./bin/ruby:ruby:gc__mark__end ◦ usdt:./bin/ruby:ruby:gc__sweep__begin ◦ usdt:./bin/ruby:ruby:gc__sweep__end • They can be used to trace GC latency: ◦ (mark_end_time - mark_begin_time)
51. #4 USDT • Example: Real-time tracing of RSS, GC mark and sweep statics ◦ Plumping up the Sinatra app process and visualize
52. USDT demo • RSS pumped up and mark proc. took more time
53. Summary: • BPF Observability has 4 keys of tracing source: • RbBCC can access all of four. Just use Ruby (and small C). • Use Ruby to trace Ruby. Dynamic Static Kernel space kprobe tracepount User space uprobe USDT
54. Observability in Action: Improve Gem’s Performance §4
55. Real World Tuning • Well-Done Speedup Contest in RubyKaigi • Theme: JSON parser Ruston ◦ mainly implemented by … Rust. (it’s native gem) ◦ Somewhat slow compared to de-facto json.rb
56. JSON vs Ruston (naive ver.)
57. JSON vs Ruston (naive ver.)
58. So, First command is…?
59. run perf • perf is useful to grasp the overall bottleneck • json’s flamegraph
60. run perf • ruston’s flamegraph • from_iter of Vec<T> • realloc ◦ in vec’s grow
61. Let’s start tracing by BPF • tracing focused function: malloc/free for this time (*) N is limited to 10,000 in solo measurement
62. Use “uprobe” of Rust • tracing focused function: malloc/free for this time
63. Measure first, Then let’s refine codes!
64. Point 1: Reduce iter()/String • Reduce iterator methods on Lex#peek() ◦ peek() is called many times on lexing process…
65. Point 1: Reduce iter()/String • This leads to reduce the usage of String ◦ Use &[u8] instead
66. Point 1: Reduce iter()/String • Then measure! malloc calloc free Ruston Before 750197 22 753491 Ruston After 110197 22 113596 cf. C JSON 20206 10022 34142 (*) N = 10,000
67. Point 2: Reduce realloc • On longer case:
68. Point 2: Reduce realloc • Implement realloc tracer
69. Point 2: Reduce realloc • Try to reduce realloc to allocate in advance ◦ Specify capacity via Vec::with_capacity()
70. Point 2: Reduce realloc • Measure! … The effect seems limited. - To be continued - realloc elapsed(s) longer case Ruston w/o vec capacity 90002 0.080172 Ruston w/ vec capacity 40002 0.071188 cf. C JSON 2 0.052459
71. The result #2 • Comparison before / after all; for case N = 50,000 user system total Ruston Before 0.277292 0.000000 0.277292 Ruston After All 0.051765 0.000000 0.051765 cf. C JSON 0.054263 0.000000 0.054263
72. Lessons learned • Existing tools are useful (e.g. perf, strace, gdb…) • To grasp detailed bottleneck, making simple BPF tool is effective. • uprobe is an entrypoint to x-ray native programs’ performance e.g. C, C++ and Rust (also … Zig?) • Just keep them in mind: measure, reproduce, measure.
73. Conclusion §5
74. BPF for Observability Gives Strong Power to us
75. Observability is Hard, but… Using Ruby, It’s Just a Fun Programming
76. Observability is Your Best Friend!!
77. Enjoy and Measure it!
78. Acknolegements: • The Book “Linux Observability with BPF” ◦ by David Calavera, Lorenzo Fontana ◦ https://www.oreilly.com/library/view/linux-observability-wit h/9781492050193/ • Brendan Gregg for his superb articles: ◦ https://www.brendangregg.com/bpf-performance-tools-b ook.html • Masashi Misono for his Japanese introduction to BPF ◦ https://atmarkit.itmedia.co.jp/ait/articles/2004/09/news006 .html
79. Acknolegements: • RbBCC received Ruby Association Grant in 2019 ◦ report: https://www.ruby.or.jp/ja/news/20200508 ◦ Maintored by Koichi “ko1” Sasada (Cookpad, Inc.) ◦ Given some advices from Ryosuke Matsumoto (Sakura Internet), Takao Shimayoshi and Yoshiaki Kasahara (Kyushu Univ.)
80. Environment of this slide: • Ruby: ◦ 3.1.2 with dtrace enabled • Linux: ◦ CPU: aarch64 ◦ Ubuntu 20.04.1 with kernel 5.8.0-63-generic • Other libraries and softwares: ◦ BCC(libbcc): 0.18.0 built with LLVM 9 ◦ strace: 5.5 (from package manager) ◦ perf: 5.8.18 (from package manager) • Code Examples