The Ruby One-Binary Tool, Enhanced with Kompo

Transcript

1. RubyKaigi 2025 ahogappa The Ruby One-Binary Tool, Enhanced with Kompo
2. About me • Sho Hirano(@ahogappa) • STORES, Inc. • I'm interested in `require`, executable binaries, and linkers. 2
3. I want to talk about kompo's updates and implementation.
4. What is Kompo? • A tool that packs Ruby into a single binary • Combines all Ruby scripts and Gems into one binary • Based on the concept of “run easily, anywhere, and fast enough” • 梱包(こんぽう) 4
5. Why do we want a one binary? • When distributing a Ruby project, it might not run because of version mismatches or missing libraries. • I want it to work by handing out just one file ◦ for example, when distributing a Ruby game. • Back in 2023, no one‑binary tool met my needs: ◦ easy to use ◦ cross‑platform ◦ compatible with the latest Ruby 5
6. Kompo's goal, or its enhanced point as the tool • Can be turned into one binary easily • Supports multiple platforms • Requires no additional installation • Runs at a decent speed 6
7. What is a one binary Ruby? 7 Ruby main.rb a.rb b.rb main.rb a.rb b.rb main.rb a.rb b.rb
8. Kompo so far • Achieves this by overriding `require`: ◦ When a Ruby file is loaded, it is fetched from the embedded data instead of the filesystem • Statically links native extensions into the binary • For example, it could run programs built with Sinatra + SQLite ◦ Cannot yet run large Ruby projects like Rails 8
9. Kompo so far 9 baz.rb Ruby main.rb Kernel#require Kernel#require_relative Kernel#autoload
10. Kompo so far 10 Ruby foo.rb bar.rb baz.rb main.rb Kernel#require Kernel#require_relative Kernel#autoload libkompo.a
11. Problems with Kompo • The original design had fundamental flaws. 11
12. Problems with Kompo 12 autoload :Foo, './foo.rb' p Foo # => Foo class Foo; end
13. Problems with Kompo 13 def require(file) eval File.read(file) end autoload :Foo, './foo.rb' p Foo # => ??? class Foo; end
14. Problems with Kompo 14 def require(file) eval File.read(file) # => uninitialized constant Foo end autoload :Foo, './foo.rb' p Foo class Foo; end
15. Problems with Kompo 15 def require(file) Object.send(:remove_const, :Foo) 　eval File.read(file) end autoload :Foo, './foo.rb' p Foo # => Foo class Foo; end Object.constants # => [..., :Foo, ...]
16. Problems with Kompo 16 def require(file) Object.send(:remove_const, :Foo) 　eval File.read(file) end autoload :Foo, './foo.rb' p Foo::Bar # => uninitialized constant Foo::Bar # => 🤯 class Foo autoload :Bar, './bar.rb' end class Foo class Bar; end end
17. Problems with Kompo • The original design had fundamental flaws. • Users could not freely access files in one binary. ◦ for example, they could not use File.read. • Because it specialized only in require, it could embed only Ruby scripts, even though programs may also need YAML, CSV, and other files. • It could not handle directories, so iterating with `Dir.children` was impossible. 17
18. Kompo couldn't freely handle data with a hierarchical structure.
19. How to solve this • Construct a virtual filesystem inside the one binary. • When Ruby calls functions such as read or open, allow it to choose whether to load data from the physical file system or from the embedded binary. 19
20. Build a file system within the one binary • Retrieve the data embedded in the binary when libc's read or read is called. • Supporting only read‑only functions is sufficient; covering every feature is unnecessary. 20
21. Methods considered • Using FUSE ◦ Requires additional installation on the user's side • Using SquashFS ◦ Also requires installation by the user 21
22. Allow the user to choose the read source • When Ruby calls read, it must be able to choose whether to call libc's read or kompo's read. ◦ A straightforward implementation would lead to recursion. • I want to link libc's read while also linking a custom read that performs additional processing. ◦ In other words, we want to intercept the call. 22 CRuby code read(fd); kompo code ssize_t read(int fd); { … read(fd); … } Usually, libc's read is called. This read refers to itself, not to libc.
23. Methods considered • Patch libc ◦ Hard to maintain • Use the linker option `--wrap` ◦ It is a GNU extension, so some linkers do not support it • Hook the system call ◦ Difficult to implement 23
24. Kompo's solution • Build our own filesystem ◦ Read‑only is enough ◦ Simulate the filesystem • Dynamically getting a function address using `dlsym(RTLD_NEXT)` ◦ The implementation stays very simple ◦ Like `--wrap`, it is an extension, but easier to compile 24
25. Usage in kompo 25 1:pub static READ_HANDLE: std::sync::LazyLock< 2: unsafe extern "C-unwind" fn( 3: fd: libc::c_int, 4: buf: *mut libc::c_void, 5: count: libc::size_t, 6: ) -> libc::ssize_t, 7:> = std::sync::LazyLock::new(|| unsafe { 8: let handle = libc::dlsym(libc::RTLD_NEXT, b"read\0".as_ptr() as _); 9: std::mem::transmute::< 10: *mut libc::c_void, 11: unsafe extern "C-unwind" fn( 12: fd: libc::c_int, 13: buf: *mut libc::c_void, 14: count: libc::size_t, 15: ) -> libc::ssize_t, 16: >(handle) 17:}); Store libc's `read` function in a static variable.
26. Demo 26 • (WIP)kompo: ◦ https://github.com/ahogappa/kompo/tree/feature/use_rtld_next • kompo-vfs: ◦ https://github.com/ahogappa/kompo-vfs
27. 27 libc functions in ruby - open(2) - read(2) - stat(2) - opendir(2) - readdir(2) - chdir(2) - getcwd(2) … physical file system - /real/b/c.rb - /real/b/d.rb …
28. in ruby script File.open “/real/b/c.rb” 28 libc functions in ruby - open(2) - read(2) - stat(2) - opendir(2) - readdir(2) - chdir(2) - getcwd(2) … physical file system - /real/b/c.rb - /real/b/d.rb …
29. 29 in ruby script require “/real/b/c.rb” physical file system - /real/b/c.rb - /real/b/d.rb … libc functions in ruby - open(2) - read(2) - stat(2) - opendir(2) - readdir(2) - chdir(2) - getcwd(2) …
30. 30 libc functions in ruby - open(2) - read(2) - stat(2) - opendir(2) - readdir(2) - chdir(2) - getcwd(2) … physical file system - /real/b/c.rb - /real/b/d.rb …
31. 31 libc functions in ruby - open(2) - read(2) - stat(2) - opendir(2) - readdir(2) - chdir(2) - getcwd(2) … kompo wrap - Access binary data under these conditions: - Absolute path: Check the embedded file system. - Relative path: Consider the current directory and check if the file exists in the binary or file system. physical file system - /real/b/c.rb - /real/b/d.rb …
32. 32 libc functions in ruby - open(2) - read(2) - stat(2) - opendir(2) - readdir(2) - chdir(2) - getcwd(2) … kompo wrap - Access binary data under these conditions: - Absolute path: Check the embedded file system. - Relative path: Consider the current directory and check if the file exists in the binary or file system. physical file system - /real/b/c.rb - /real/b/d.rb … virtual file system - /kompo/b/c.rb - /kompo/b/d.rb …
33. 33 in ruby script File.open “/kompo/b/c.rb” libc functions in ruby - open(2) - read(2) - stat(2) - opendir(2) - readdir(2) - chdir(2) - getcwd(2) … physical file system - /real/b/c.rb - /real/b/d.rb … virtual file system - /kompo/b/c.rb - /kompo/b/d.rb … kompo wrap - Access binary data under these conditions: - Absolute path: Check the embedded file system. - Relative path: Consider the current directory and check if the file exists in the binary or file system.
34. 34 in ruby script File.open “./c.rb” /real/b current directory libc functions in ruby - open(2) - read(2) - stat(2) - opendir(2) - readdir(2) - chdir(2) - getcwd(2) … physical file system - /real/b/c.rb - /real/b/d.rb … virtual file system - /kompo/b/c.rb - /kompo/b/d.rb … kompo wrap - Access binary data under these conditions: - Absolute path: Check the embedded file system. - Relative path: Consider the current directory and check if the file exists in the binary or file system.
35. 35 in ruby script Dir.chdir “/kompo/b” /kompo/b current directory libc functions in ruby - open(2) - read(2) - stat(2) - opendir(2) - readdir(2) - chdir(2) - getcwd(2) … physical file system - /real/b/c.rb - /real/b/d.rb … virtual file system - /kompo/b/c.rb - /kompo/b/d.rb … kompo wrap - Access binary data under these conditions: - Absolute path: Check the embedded file system. - Relative path: Consider the current directory and check if the file exists in the binary or file system.
36. 36 in ruby script File.open “./c.rb” /kompo/b libc functions in ruby - open(2) - read(2) - stat(2) - opendir(2) - readdir(2) - chdir(2) - getcwd(2) … physical file system - /real/b/c.rb - /real/b/d.rb … virtual file system - /kompo/b/c.rb - /kompo/b/d.rb … kompo wrap - Access binary data under these conditions: - Absolute path: Check the embedded file system. - Relative path: Consider the current directory and check if the file exists in the binary or file system. current directory
37. Difference from Kompo’s goal • Can be turned into one binary easily ◦ Support coming soon • Cross-platform support ◦ Not yet available • Requires no additional installation ◦ Already achieved • Runs at a decent speed ◦ Needs refactoring 37
38. Conclusion • The previous kompo approach sometimes failed to create a one binary. • I tried various implementations and found a workable method. • With this implementation, I managed to make Rails run as one binary. • From now on, I want to focus on turning it into a gem and adding cross-platform support. 38
39. Thank you!!