$30 off During Our Annual Pro Sale. View Details »

Ruby x BPF in Action / RubyKaigi 2022

KONDO Uchio
September 09, 2022

Ruby x BPF in Action / RubyKaigi 2022

KONDO Uchio

September 09, 2022
Tweet

More Decks by KONDO Uchio

Other Decks in Technology

Transcript

  1. Ruby x BPF in Action
    Uchio Kondo from Mirrativ, Inc. @ RubyKaigi 2022.09.09

    View Slide

  2. Uchio Kondo
    Infra & Streaming team @ Mirrativ, Inc
    Speaker @ RubyKaigi 2016, 2018, 2019, 2021
    “Hacker” Supporter @ Engineer’s Café Fukuoka
    Ruby & Rust enthuasist, Linux freak
    Live in Fukuoka

    View Slide

  3. (After all,)
    What can BPF do?
    §1

    View Slide

  4. e.g. Visualize “SYN queue”
    From cloudflare’s blog “SYN packet handling in the wild”:
    https://blog.cloudflare.com/syn-packet-handling-in-the-wild/

    View Slide

  5. 2 servers in different config
    ● Same server (WEBRick 1.7.0, ruby 3.1.2)

    ● Same bench parameter:

    ● Different value of net.core.somaxconn
    ○ 4,096 vs 500, how this makes effect?

    View Slide

  6. Compare results
    somaxconn = 500 somaxconn = 4096

    View Slide

  7. Tool to Visualize “SYN queue”

    View Slide

  8. C to Visualize “SYN queue”

    View Slide

  9. Let’s watch queue satuation
    somaxconn = 500 somaxconn = 4096

    View Slide

  10. demo

    View Slide

  11. Now, you’ve completely
    understood BPF!
    …OK. Let me keep going.

    View Slide

  12. Quick Introduction to
    BPF
    §2

    View Slide

  13. Big picture first
    https://whimsical.com/bpf-ABAvCvJFLcSie2ML9ee2fn

    View Slide

  14. #1 History

    View Slide

  15. #1 History
    https://www.tcpdump.org/papers/bpf-usenix93.pdf

    View Slide

  16. #1 History
    https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commi
    t/?id=bd4cf0ed331a275e9bf5a49e6d0fd55dffc551b8

    View Slide

  17. #2 How it works

    View Slide

  18. #2 How it works
    ref. https://speakerdeck.com/chikuwait/learn-ebpf?slide=17 by Yuki Nakata, 2020
    emoji from https://github.com/twitter/twemoji/tree/master/assets
    (*) Very simplified
    Scripting Bytecode
    BPF VM
    BPF Map
    User Interface Collectiong
    Kernel
    Data…
    … or perf buffer, etc.
    The Userland The Kingdom of Kernel

    View Slide

  19. #2 How it works
    BTF requires:
    Kernel version >= 5.6 &&
    CONFIG_DEBUG_INFO_BTF
    should be enabled

    View Slide

  20. #3 Comparison

    View Slide

  21. #3 Comparison

    View Slide

  22. #3 Comparison

    View Slide

  23. #4 (Expanding) Use cases…

    View Slide

  24. #4 (Expanding) Use cases…
    ● BPF-based network
    & security for containers
    ● the de facto Kubernetes
    threat detection engine
    using BPF

    View Slide

  25. #4 (Expanding) Use cases…
    rbperf (https://github.com/javierhonduco/rbperf)
    by Jabier H. Coto with

    View Slide

  26. RbBCC and BPF:
    Observe Everything
    §3

    View Slide

  27. What is RbBCC?
    ● A: BCC for Ruby (libbcc FFI binding for Ruby)
    ● WHAT is BCC?
    ○ BPF Compiler Collection:
    ○ An SDK to make BPF tools, using Script Languages
    (Python/Lua supported officially)
    ○ But - Ruby is not in its support list, so I’m developping
    I’m going to show How to use –
    How to write BPF Ruby codes.

    View Slide

  28. RbBCC’s “4 Keys”

    View Slide

  29. #1 kprobe
    ● kprobe - (mainly) function trace in Linux Kernel
    ● e.g. __ARCH_sys_execve()
    ○ The substantial function called when execve(2) is invoked

    View Slide

  30. Example RbBCC Program:

    View Slide

  31. Example RbBCC Program:
    C Part:
    tracing function run inside kernel

    View Slide

  32. Example RbBCC Program:
    Ruby Part:
    Load the C part above and
    Get feedback and data
    from BPF program inside kernel

    View Slide

  33. demo

    View Slide

  34. #1 kprobe
    ● ex.2 tcp_v4_conn_request (the first demo)

    View Slide

  35. Trace the connect
    ● Also has 2 parts:
    ○ BPF DSL in C
    ○ Load program &
    handle data in Ruby

    View Slide

  36. Return to first demo

    View Slide

  37. #2 tracepoint (for kernel)
    ● Different stuff from Ruby’s TracePoint class
    ● A static entrypoint to trace kernel events
    ● It won’t change in the future
    version of Linux
    ○ kprobe traces an
    exported symbol of
    kernel, so it should be
    changed and
    maybe unstable.

    View Slide

  38. #2 tracepoint (for kernel)
    ● Tracing syscall invocations:
    ● raw_syscalls/sys_enter
    ● raw_syscalls/sys_exit

    View Slide

  39. tracepoint demo
    ● Example output (compared with strace -w):

    View Slide

  40. ● Example: tracing WEBrick (again):
    ○ ruby:
    ○ ab:
    ● Tracing command:
    ○ ruby:
    ○ strace:
    FYI: Performance sideeffect

    View Slide

  41. FYI: Performance sideeffect
    w/ RbBCC w/ strace

    View Slide

  42. #3 uprobe
    ● uprobe: A mechanism to attach to user space function calls

    View Slide

  43. #3 uprobe
    ● Using uprobe (and USDT afterwards) with ease,
    build a special Ruby binary with a specific option:

    View Slide

  44. #3 uprobe
    ● Tracing rb_str_new(const char *ptr, long len)

    View Slide

  45. #3 uprobe
    ● Tracing rb_str_new(const char *ptr, long len)

    View Slide

  46. #3 uprobe
    ● Collecting rb_str_new()’s:
    (function return timestamp - function entry timestamp)
    ● This represents the latency of a function call
    ● function entry = uprobe, function return = uretprobe

    View Slide

  47. #3 uprobe
    ● Example of rb_str_new()’s latency histogram:
    ruby -e ‘p “Hello”’
    ruby --disable gems -e ‘p “Hello”’

    View Slide

  48. #4 USDT
    ● USDT: Userspace Statically Defined Tracepoint
    ○ Probe points that an author of a program
    embedded in advance
    ○ cf. uprobe traces real function call dynamically
    ● USDT for uprobe is just as Tracepoint for kprobe
    Dynamic Static
    Kernel space kprobe tracepount
    User space uprobe USDT

    View Slide

  49. #4 USDT
    ● Ruby’s USDT
    (first for DTrace,
    but available via BPF in Linux)
    Japanese article: https://magazine.rubyist.net/articles/0041/0041-200Special-dtrace.html
    https://rubyreferences.github.io/rubyref/advanced/dtrace.html

    View Slide

  50. #4 USDT
    ● Example: USDTs about GC:
    ○ usdt:./bin/ruby:ruby:gc__mark__begin
    ○ usdt:./bin/ruby:ruby:gc__mark__end
    ○ usdt:./bin/ruby:ruby:gc__sweep__begin
    ○ usdt:./bin/ruby:ruby:gc__sweep__end
    ● They can be used to trace GC latency:
    ○ (mark_end_time - mark_begin_time)

    View Slide

  51. #4 USDT
    ● Example: Real-time tracing of RSS, GC mark and sweep statics
    ○ Plumping up the Sinatra
    app process and
    visualize

    View Slide

  52. USDT demo
    ● RSS pumped up and mark proc. took more time

    View Slide

  53. Summary:
    ● BPF Observability has 4 keys of tracing source:
    ● RbBCC can access all of four. Just use Ruby (and small C).
    ● Use Ruby to trace Ruby.
    Dynamic Static
    Kernel space kprobe tracepount
    User space uprobe USDT

    View Slide

  54. Observability in Action:
    Improve Gem’s Performance
    §4

    View Slide

  55. Real World Tuning
    ● Well-Done Speedup Contest in RubyKaigi
    ● Theme: JSON parser Ruston
    ○ mainly implemented by
    … Rust. (it’s native gem)
    ○ Somewhat slow
    compared to
    de-facto json.rb

    View Slide

  56. JSON vs Ruston (naive ver.)

    View Slide

  57. JSON vs Ruston (naive ver.)

    View Slide

  58. So, First command is…?

    View Slide

  59. run perf
    ● perf is useful to grasp
    the overall bottleneck
    ● json’s flamegraph

    View Slide

  60. run perf
    ● ruston’s flamegraph
    ● from_iter of Vec
    ● realloc
    ○ in vec’s grow

    View Slide

  61. Let’s start tracing by BPF
    ● tracing focused function: malloc/free for this time
    (*) N is limited to 10,000 in solo measurement

    View Slide

  62. Use “uprobe” of Rust
    ● tracing focused function: malloc/free for this time

    View Slide

  63. Measure first,
    Then let’s refine codes!

    View Slide

  64. Point 1: Reduce iter()/String
    ● Reduce iterator methods on Lex#peek()
    ○ peek() is called many times on lexing process…

    View Slide

  65. Point 1: Reduce iter()/String
    ● This leads to reduce the usage of String
    ○ Use &[u8] instead

    View Slide

  66. Point 1: Reduce iter()/String
    ● Then measure!
    malloc calloc free
    Ruston Before 750197 22 753491
    Ruston After 110197 22 113596
    cf. C JSON 20206 10022 34142
    (*) N = 10,000

    View Slide

  67. Point 2: Reduce realloc
    ● On longer case:

    View Slide

  68. Point 2: Reduce realloc
    ● Implement realloc tracer

    View Slide

  69. Point 2: Reduce realloc
    ● Try to reduce realloc to allocate in advance
    ○ Specify capacity via Vec::with_capacity()

    View Slide

  70. Point 2: Reduce realloc
    ● Measure! … The effect seems limited. - To be continued -
    realloc elapsed(s)
    longer case
    Ruston
    w/o vec capacity
    90002 0.080172
    Ruston
    w/ vec capacity
    40002 0.071188
    cf. C JSON 2 0.052459

    View Slide

  71. The result #2
    ● Comparison before / after all; for case N = 50,000
    user system total
    Ruston Before 0.277292 0.000000 0.277292
    Ruston After All 0.051765 0.000000 0.051765
    cf. C JSON 0.054263 0.000000 0.054263

    View Slide

  72. Lessons learned
    ● Existing tools are useful (e.g. perf, strace, gdb…)
    ● To grasp detailed bottleneck,
    making simple BPF tool is effective.
    ● uprobe is an entrypoint to x-ray native programs’ performance
    e.g. C, C++ and Rust (also … Zig?)
    ● Just keep them in mind:
    measure, reproduce, measure.

    View Slide

  73. Conclusion
    §5

    View Slide

  74. BPF for Observability Gives
    Strong Power to us

    View Slide

  75. Observability is Hard, but…
    Using Ruby,
    It’s Just a Fun Programming

    View Slide

  76. Observability is
    Your Best Friend!!

    View Slide

  77. Enjoy and Measure it!

    View Slide

  78. Acknolegements:
    ● The Book “Linux Observability with BPF”
    ○ by David Calavera, Lorenzo Fontana
    ○ https://www.oreilly.com/library/view/linux-observability-wit
    h/9781492050193/
    ● Brendan Gregg for his superb articles:
    ○ https://www.brendangregg.com/bpf-performance-tools-b
    ook.html
    ● Masashi Misono for his Japanese introduction to BPF
    ○ https://atmarkit.itmedia.co.jp/ait/articles/2004/09/news006
    .html

    View Slide

  79. Acknolegements:
    ● RbBCC received Ruby Association Grant in 2019
    ○ report: https://www.ruby.or.jp/ja/news/20200508
    ○ Maintored by Koichi “ko1” Sasada (Cookpad, Inc.)
    ○ Given some advices from
    Ryosuke Matsumoto (Sakura Internet),
    Takao Shimayoshi and Yoshiaki Kasahara (Kyushu Univ.)

    View Slide

  80. Environment of this slide:
    ● Ruby:
    ○ 3.1.2 with dtrace enabled
    ● Linux:
    ○ CPU: aarch64
    ○ Ubuntu 20.04.1 with kernel 5.8.0-63-generic
    ● Other libraries and softwares:
    ○ BCC(libbcc): 0.18.0 built with LLVM 9
    ○ strace: 5.5 (from package manager)
    ○ perf: 5.8.18 (from package manager)
    ● Code Examples

    View Slide