RubyConf 2017: Packing your Ruby application into a single executable

Recent languages like Go compiles a project into a nice executable, why can't good ol' Ruby? We have built an packer for Ruby to do just that. It is 100% open-source, and can produce executables for Windows, macOS and Linux individually. By packing, distributing Ruby apps are made extremely easy, additionally with intellectual property protection. Auto-updating is also made easy, in that the executable only needs to download and replace itself. So, how we did it? How to use it? What goes under the hood? What future will this bring to Ruby? That's what will be unraveled in this talk!

Transcript

1. Packing your Ruby application into a single executable Minqi Pan
2. I’m Minqi Pan
3. Beijing
4. Hacker of Ruby/C++
5. Node.js Collaborator
7. github.com/pmq20 twitter @psvr
9. go build yours.go
12. Before
20. Problems • Slow installation. Tons of files to download. Great- Wall’d in China. Remembered to use sudo? • Error-prone. Failed compiling native modules? Should I care about post-installations notices? • Ruby runtime version? Wanted to use lonely operator reliably? Coexist with multiple Rubies?
21. Updating?
27. Problems • No version checks. New versions missed without your attentions. • Cumbersome to update. Multiple steps needed.
28. After
33. Updating?
37. Introducing
38. github.com/pmq20/ ruby-packer
39. enclose.io/rubyc
41. Example 1. Producing a single Ruby interpreter executable
42. Example 1. Producing a single Ruby interpreter executable
43. Example 2: Compiling a CLI tool
44. Example 2: Compiling a CLI tool
46. Example 3: Compiling a Rails application
47. Example 3: Compiling a Rails application
48. Example 3: Compiling a Rails application
49. Example 3: Compiling a Rails application
50. Example 3: Compiling a Rails application
51. Example 4: Compiling a Gem
52. Auto-updating
53. https://github.com/pmq20/libautoupdate libautoupdate https://github.com/pmq20/ruby-packer Ruby Packer
54. Under the Hood
55. Introducing… “/__enclose_io_memfs__” a “mounted” disk in RAM for your project
59. (in example 1)
61. (in example 2)
64. The entire Ruby stdlib is in your exe
65. The Idea System calls on paths starting with
 /__enclose_io_memfs__
 are redirected to the RAM while others to the FS
66. /__enclose_io_memfs__/… others
67. Where’s your project? (in example 2)
68. Where’s your project? (in example 2)
69. Where’s your project? (in example 2)
70. Where’s your project? (in example 2)
71. Hard-code an Entrance
72. ruby ~/your_project/bin/your_cli argv[1] preset to
 /__enclose_io_memfs__/local/bin/your_cli ./a.out
73. But there are so many API’s • require, load, require_relative • File.read, File.open, Dir.open • File.readlink, File.stat, File.lstat • …
74. It’s hard to hack them one by one
75. It’s hard to maintain the hacks one by one
76. Yes, we hacked Ruby
77. But in a very minimal way
78. But in a very minimal way
79. That’s because… https://github.com/pmq20/libsquash libsquash https://github.com/pmq20/ruby-packer Ruby Packer
80. Dave Vasilevsky github.com/vasi Author of squashfuse
81. Shengyuan Liu github.com/SounderLiu Co-author of libsquash
82. Introducing SquashFS
83. SquashFS • a compressed read-only file system • used by the Live CD versions of Arch Linux, Debian, Fedora, Gentoo, Mint, Salix, Ubuntu • used on OpenWrt and DD-WRT router firmware
84. a project 148M after squashing 16M mksquashfs
85. SquashFS • Introduced in 2009 with Linux 2.6.29 • File format very stabilized • Unsquashfs and mksquashfs have win32 ports; 7-Zip on win32 also supports SquashFS • Part of kernel; GPL Licensed
86. Introducing libsquash
87. https://github.com/pmq20/libsquash libsquash • MIT licensed • 100% User-land Code • Embeddable, 1 dep. only • Compiles on 3 platforms, even Windows XP with VC++ 2010 • Introduces VFD - virtual file descriptor, intercepting system calls unobtrusively libsquash
88. API of libsquash mirroring system calls
89. Virtual File Descriptor generated by a duplicating file descriptor 0
90. https://github.com/pmq20/libsquash libsquash
91. File Descriptors
 generated by libsquash others https://github.com/pmq20/libsquash libsquash
92. Use libsquash unobtrusively Just include a header and it’s done!
93. Use libsquash unobtrusively Win32 API works as well
94. What about Native Extensions?
95. What about Native Extensions? • Libsquash Intercepts dlopen(), LoadLibraryExW() • Dynamic library files inside the pack are extracted to temporary files • dlopen / LoadLibraryExW redirects the request to the temporary files • temporary files are deleted on exit
96. What about Rails?
97. But what about Rails? • SquashFS is read-only, so your project root is read- only • Rails creates tmp/ or log/ and writes to it, and tmp/ or log/ is in your project root • Rails has config files in your project root
98. Solution: writable root • redirect `mkdir()` inside the memfs to a temporary directory • redirect `open()` with `O_CREAT` inside the memfs to a temporary directory • redirect `CreateFileW()` with writing inside the memfs to a temporary directory • removes the temporary directory and files at exit
99. ENCLOSE_IO_WORKDIR
100. ENCLOSE_IO_WORKDIR
101. ENCLOSE_IO_WORKDIR
102. yours/*
103. yours/* yours.squashfs mksquashfs
104. yours/* yours.squashfs libsquash, libautoupdate compile
105. yours/* yours.squashfs libsquash, libautoupdate Ruby Runtime compile
106. yours/* yours.squashfs libsquash, libautoupdate Ruby Runtime
107. yours/* yours.squashfs libsquash, libautoupdate Ruby Runtime
108. yours/* yours.squashfs libsquash, libautoupdate Ruby Runtime yours.exe Statically Link Statically Link
109. yours.exe Distribute and Enjoy
110. github.com/pmq20 twitter @psvr
111. Download & Install
112. git clone
 https://github.com/ pmq20/ruby-packer.git
113. add bin/rubyc to
 $PATH or %PATH%
114. or
115. http://enclose.io/rubyc
117. Windows • SquashFS Tools 4.3 • Visual Studio 2015 Update 3, all editions including the Community edition (remember to select
 “Common Tools for Visual C++ 2015" feature during installation). • Ruby
118. macOS • SquashFS Tools 4.3: brew install squashfs • Xcode, You also need to install the Command Line Tools via Xcode. • Ruby
119. Linux • SquashFS Tools 4.3: sudo yum install squashfs- tools or sudo apt-get install squashfs-tools • gcc or clang • GNU Make • Ruby
120. Tips
121. Use --tmpdir=/a/fixed/location
122. Use Windows
 with a big Virtual Memory
123. Use older Linux, e.g. CentOS release 5.8
 with gcc and g++ 4.8
 (possibly from devtoolset-2 of slc5-devtoolset)
124. Use older Mac, e.g. Mac OS X 10.7 Lion with Xcode 4.6.3
125. Check dependencies using Dependency Walker, otool, ldd
 before releasing
126. See Also
127. “Real” Compiling KEVIN DEISZ @ RubyConf 2017 Compiling Ruby
128. “Real” Compiling Koichi Sasada @ RailConf 2016 Precompiling Ruby scripts
 Myth and Fact
129. github.com/pmq20 twitter @psvr