RubyConf 2017: Packing your Ruby application into a single executable

8002c84eb4c18170632f8fb7efb09288?s=47 Minqi Pan
November 15, 2017

RubyConf 2017: Packing your Ruby application into a single executable

Recent languages like Go compiles a project into a nice executable, why can't good ol' Ruby? We have built an packer for Ruby to do just that. It is 100% open-source, and can produce executables for Windows, macOS and Linux individually. By packing, distributing Ruby apps are made extremely easy, additionally with intellectual property protection. Auto-updating is also made easy, in that the executable only needs to download and replace itself. So, how we did it? How to use it? What goes under the hood? What future will this bring to Ruby? That's what will be unraveled in this talk!

8002c84eb4c18170632f8fb7efb09288?s=128

Minqi Pan

November 15, 2017
Tweet

Transcript

  1. Packing your Ruby application into a single executable Minqi Pan

  2. I’m Minqi Pan

  3. Beijing

  4. Hacker of Ruby/C++

  5. Node.js Collaborator

  6. None
  7. github.com/pmq20 twitter @psvr

  8. None
  9. go build yours.go

  10. None
  11. None
  12. Before

  13. None
  14. None
  15. None
  16. None
  17. None
  18. None
  19. None
  20. Problems • Slow installation. Tons of files to download. Great-

    Wall’d in China. Remembered to use sudo? • Error-prone. Failed compiling native modules? Should I care about post-installations notices? • Ruby runtime version? Wanted to use lonely operator reliably? Coexist with multiple Rubies?
  21. Updating?

  22. None
  23. None
  24. None
  25. None
  26. None
  27. Problems • No version checks. New versions missed without your

    attentions. • Cumbersome to update. Multiple steps needed.
  28. After

  29. None
  30. None
  31. None
  32. None
  33. Updating?

  34. None
  35. None
  36. None
  37. Introducing

  38. github.com/pmq20/ ruby-packer

  39. enclose.io/rubyc

  40. None
  41. Example 1. Producing a single Ruby interpreter executable

  42. Example 1. Producing a single Ruby interpreter executable

  43. Example 2: Compiling a CLI tool

  44. Example 2: Compiling a CLI tool

  45. None
  46. Example 3: Compiling a Rails application

  47. Example 3: Compiling a Rails application

  48. Example 3: Compiling a Rails application

  49. Example 3: Compiling a Rails application

  50. Example 3: Compiling a Rails application

  51. Example 4: Compiling a Gem

  52. Auto-updating

  53. https://github.com/pmq20/libautoupdate libautoupdate https://github.com/pmq20/ruby-packer Ruby Packer

  54. Under the Hood

  55. Introducing… “/__enclose_io_memfs__” a “mounted” disk in RAM for your project

  56. None
  57. None
  58. None
  59. (in example 1)

  60. None
  61. (in example 2)

  62. None
  63. None
  64. The entire Ruby stdlib is in your exe

  65. The Idea System calls on paths starting with
 /__enclose_io_memfs__
 are

    redirected to the RAM while others to the FS
  66. /__enclose_io_memfs__/… others

  67. Where’s your project? (in example 2)

  68. Where’s your project? (in example 2)

  69. Where’s your project? (in example 2)

  70. Where’s your project? (in example 2)

  71. Hard-code an Entrance

  72. ruby ~/your_project/bin/your_cli argv[1] preset to
 /__enclose_io_memfs__/local/bin/your_cli ./a.out

  73. But there are so many API’s • require, load, require_relative

    • File.read, File.open, Dir.open • File.readlink, File.stat, File.lstat • …
  74. It’s hard to hack them one by one

  75. It’s hard to maintain the hacks one by one

  76. Yes, we hacked Ruby

  77. But in a very minimal way

  78. But in a very minimal way

  79. That’s because… https://github.com/pmq20/libsquash libsquash https://github.com/pmq20/ruby-packer Ruby Packer

  80. Dave Vasilevsky github.com/vasi Author of squashfuse

  81. Shengyuan Liu github.com/SounderLiu Co-author of libsquash

  82. Introducing SquashFS

  83. SquashFS • a compressed read-only file system • used by

    the Live CD versions of Arch Linux, Debian, Fedora, Gentoo, Mint, Salix, Ubuntu • used on OpenWrt and DD-WRT router firmware
  84. a project 148M after squashing 16M mksquashfs

  85. SquashFS • Introduced in 2009 with Linux 2.6.29 • File

    format very stabilized • Unsquashfs and mksquashfs have win32 ports; 7-Zip on win32 also supports SquashFS • Part of kernel; GPL Licensed
  86. Introducing libsquash

  87. https://github.com/pmq20/libsquash libsquash • MIT licensed • 100% User-land Code •

    Embeddable, 1 dep. only • Compiles on 3 platforms, even Windows XP with VC++ 2010 • Introduces VFD - virtual file descriptor, intercepting system calls unobtrusively libsquash
  88. API of libsquash mirroring system calls

  89. Virtual File Descriptor generated by a duplicating file descriptor 0

  90. https://github.com/pmq20/libsquash libsquash

  91. File Descriptors
 generated by libsquash others https://github.com/pmq20/libsquash libsquash

  92. Use libsquash unobtrusively Just include a header and it’s done!

  93. Use libsquash unobtrusively Win32 API works as well

  94. What about Native Extensions?

  95. What about Native Extensions? • Libsquash Intercepts dlopen(), LoadLibraryExW() •

    Dynamic library files inside the pack are extracted to temporary files • dlopen / LoadLibraryExW redirects the request to the temporary files • temporary files are deleted on exit
  96. What about Rails?

  97. But what about Rails? • SquashFS is read-only, so your

    project root is read- only • Rails creates tmp/ or log/ and writes to it, and tmp/ or log/ is in your project root • Rails has config files in your project root
  98. Solution: writable root • redirect `mkdir()` inside the memfs to

    a temporary directory • redirect `open()` with `O_CREAT` inside the memfs to a temporary directory • redirect `CreateFileW()` with writing inside the memfs to a temporary directory • removes the temporary directory and files at exit
  99. ENCLOSE_IO_WORKDIR

  100. ENCLOSE_IO_WORKDIR

  101. ENCLOSE_IO_WORKDIR

  102. yours/*

  103. yours/* yours.squashfs mksquashfs

  104. yours/* yours.squashfs libsquash, libautoupdate compile

  105. yours/* yours.squashfs libsquash, libautoupdate Ruby Runtime compile

  106. yours/* yours.squashfs libsquash, libautoupdate Ruby Runtime

  107. yours/* yours.squashfs libsquash, libautoupdate Ruby Runtime

  108. yours/* yours.squashfs libsquash, libautoupdate Ruby Runtime yours.exe Statically Link Statically

    Link
  109. yours.exe Distribute and Enjoy

  110. github.com/pmq20 twitter @psvr

  111. Download & Install

  112. git clone
 https://github.com/ pmq20/ruby-packer.git

  113. add bin/rubyc to
 $PATH or %PATH%

  114. or

  115. http://enclose.io/rubyc

  116. None
  117. Windows • SquashFS Tools 4.3 • Visual Studio 2015 Update

    3, all editions including the Community edition (remember to select
 “Common Tools for Visual C++ 2015" feature during installation). • Ruby
  118. macOS • SquashFS Tools 4.3: brew install squashfs • Xcode,

    You also need to install the Command Line Tools via Xcode. • Ruby
  119. Linux • SquashFS Tools 4.3: sudo yum install squashfs- tools

    or sudo apt-get install squashfs-tools • gcc or clang • GNU Make • Ruby
  120. Tips

  121. Use --tmpdir=/a/fixed/location

  122. Use Windows
 with a big Virtual Memory

  123. Use older Linux, e.g. CentOS release 5.8
 with gcc and

    g++ 4.8
 (possibly from devtoolset-2 of slc5-devtoolset)
  124. Use older Mac, e.g. Mac OS X 10.7 Lion with

    Xcode 4.6.3
  125. Check dependencies using Dependency Walker, otool, ldd
 before releasing

  126. See Also

  127. “Real” Compiling KEVIN DEISZ @ RubyConf 2017 Compiling Ruby

  128. “Real” Compiling Koichi Sasada @ RailConf 2016 Precompiling Ruby scripts


    Myth and Fact
  129. github.com/pmq20 twitter @psvr