RubyConf 2017: Packing your Ruby application into a single executable

Recent languages like Go compiles a project into a nice executable, why can't good ol' Ruby? We have built an packer for Ruby to do just that. It is 100% open-source, and can produce executables for Windows, macOS and Linux individually. By packing, distributing Ruby apps are made extremely easy, additionally with intellectual property protection. Auto-updating is also made easy, in that the executable only needs to download and replace itself. So, how we did it? How to use it? What goes under the hood? What future will this bring to Ruby? That's what will be unraveled in this talk!

Transcript

1. Packing your Ruby
   application into a single
   executable
   Minqi Pan
   

   View Slide

2. I’m Minqi Pan
   

   View Slide

3. Beijing
   

   View Slide

4. Hacker of Ruby/C++
   

   View Slide

5. Node.js Collaborator
   

   View Slide

6. View Slide

7. github.com/pmq20
   twitter
   @psvr
   

   View Slide

8. View Slide

9. go build yours.go
   

   View Slide

10. View Slide

11. View Slide

12. Before
    

    View Slide

13. View Slide

14. View Slide

15. View Slide

16. View Slide

17. View Slide

18. View Slide

19. View Slide

20. Problems
    • Slow installation. Tons of files to download. Great-
    Wall’d in China. Remembered to use sudo?
    • Error-prone. Failed compiling native modules?
    Should I care about post-installations notices?
    • Ruby runtime version? Wanted to use lonely
    operator reliably? Coexist with multiple Rubies?
    

    View Slide

21. Updating?
    

    View Slide

22. View Slide

23. View Slide

24. View Slide

25. View Slide

26. View Slide

27. Problems
    • No version checks. New versions missed without
    your attentions.
    • Cumbersome to update. Multiple steps needed.
    

    View Slide

28. After
    

    View Slide

29. View Slide

30. View Slide

31. View Slide

32. View Slide

33. Updating?
    

    View Slide

34. View Slide

35. View Slide

36. View Slide

37. Introducing
    

    View Slide

38. github.com/pmq20/
    ruby-packer
    

    View Slide

39. enclose.io/rubyc
    

    View Slide

40. View Slide

41. Example 1. Producing a single
    Ruby interpreter executable
    

    View Slide

42. Example 1. Producing a single
    Ruby interpreter executable
    

    View Slide

43. Example 2: Compiling a CLI
    tool
    

    View Slide

44. Example 2: Compiling a CLI
    tool
    

    View Slide

45. View Slide

46. Example 3: Compiling a
    Rails application
    

    View Slide

47. Example 3: Compiling a
    Rails application
    

    View Slide

48. Example 3: Compiling a
    Rails application
    

    View Slide

49. Example 3: Compiling a
    Rails application
    

    View Slide

50. Example 3: Compiling a
    Rails application
    

    View Slide

51. Example 4: Compiling a
    Gem
    

    View Slide

52. Auto-updating
    

    View Slide

53. https://github.com/pmq20/libautoupdate
    libautoupdate
    https://github.com/pmq20/ruby-packer
    Ruby Packer
    

    View Slide

54. Under the Hood
    

    View Slide

55. Introducing…
    “/__enclose_io_memfs__”
    a “mounted” disk in RAM for your project
    

    View Slide

56. View Slide

57. View Slide

58. View Slide

59. (in example 1)
    

    View Slide

60. View Slide

61. (in example 2)
    

    View Slide

62. View Slide

63. View Slide

64. The entire
    Ruby stdlib is
    in your exe
    

    View Slide

65. The Idea
    System calls on paths starting with

    /__enclose_io_memfs__

    are redirected to the RAM while others to the FS
    

    View Slide

66. /__enclose_io_memfs__/… others
    

    View Slide

67. Where’s your project?
    (in example 2)
    

    View Slide

68. Where’s your project?
    (in example 2)
    

    View Slide

69. Where’s your project?
    (in example 2)
    

    View Slide

70. Where’s your project?
    (in example 2)
    

    View Slide

71. Hard-code an Entrance
    

    View Slide

72. ruby ~/your_project/bin/your_cli
    argv[1] preset to

    /__enclose_io_memfs__/local/bin/your_cli
    ./a.out
    

    View Slide

73. But there are so many API’s
    • require, load, require_relative
    • File.read, File.open, Dir.open
    • File.readlink, File.stat, File.lstat
    • …
    

    View Slide

74. It’s hard to hack them
    one by one
    

    View Slide

75. It’s hard to maintain the
    hacks one by one
    

    View Slide

76. Yes, we hacked Ruby
    

    View Slide

77. But in a very minimal way
    

    View Slide

78. But in a very minimal way
    

    View Slide

79. That’s because…
    https://github.com/pmq20/libsquash
    libsquash
    https://github.com/pmq20/ruby-packer
    Ruby Packer
    

    View Slide

80. Dave Vasilevsky
    github.com/vasi
    Author of
    squashfuse
    

    View Slide

81. Shengyuan Liu
    github.com/SounderLiu
    Co-author of
    libsquash
    

    View Slide

82. Introducing SquashFS
    

    View Slide

83. SquashFS
    • a compressed read-only file system
    • used by the Live CD versions of Arch
    Linux, Debian, Fedora, Gentoo, Mint,
    Salix, Ubuntu
    • used on OpenWrt and DD-WRT router
    firmware
    

    View Slide

84. a project
    148M
    after squashing
    16M
    mksquashfs
    

    View Slide

85. SquashFS
    • Introduced in 2009 with Linux 2.6.29
    • File format very stabilized
    • Unsquashfs and mksquashfs have win32 ports;
    7-Zip on win32 also supports SquashFS
    • Part of kernel; GPL Licensed
    

    View Slide

86. Introducing libsquash
    

    View Slide

87. https://github.com/pmq20/libsquash
    libsquash
    • MIT licensed
    • 100% User-land Code
    • Embeddable, 1 dep. only
    • Compiles on 3 platforms, even
    Windows XP with VC++ 2010
    • Introduces VFD - virtual file
    descriptor, intercepting
    system calls unobtrusively
    libsquash
    

    View Slide

88. API of libsquash
    mirroring system calls
    

    View Slide

89. Virtual File Descriptor
    generated by a duplicating file descriptor 0
    

    View Slide

90. https://github.com/pmq20/libsquash
    libsquash
    

    View Slide

91. File Descriptors

    generated by libsquash others
    https://github.com/pmq20/libsquash
    libsquash
    

    View Slide

92. Use libsquash unobtrusively
    Just include a header and it’s done!
    

    View Slide

93. Use libsquash unobtrusively
    Win32 API works as well
    

    View Slide

94. What about Native
    Extensions?
    

    View Slide

95. What about Native
    Extensions?
    • Libsquash Intercepts dlopen(), LoadLibraryExW()
    • Dynamic library files inside the pack are extracted
    to temporary files
    • dlopen / LoadLibraryExW redirects the request to
    the temporary files
    • temporary files are deleted on exit
    

    View Slide

96. What about Rails?
    

    View Slide

97. But what about Rails?
    • SquashFS is read-only, so your project root is read-
    only
    • Rails creates tmp/ or log/ and writes to it, and tmp/
    or log/ is in your project root
    • Rails has config files in your project root
    

    View Slide

98. Solution: writable root
    • redirect `mkdir()` inside the memfs to a temporary
    directory
    • redirect `open()` with `O_CREAT` inside the memfs
    to a temporary directory
    • redirect `CreateFileW()` with writing inside the
    memfs to a temporary directory
    • removes the temporary directory and files at exit
    

    View Slide

99. ENCLOSE_IO_WORKDIR
    

    View Slide

100. ENCLOSE_IO_WORKDIR
     

     View Slide

101. ENCLOSE_IO_WORKDIR
     

     View Slide

102. yours/*
     

     View Slide

103. yours/*
     yours.squashfs
     mksquashfs
     

     View Slide

104. yours/*
     yours.squashfs
     libsquash, libautoupdate compile
     

     View Slide

105. yours/*
     yours.squashfs
     libsquash, libautoupdate
     Ruby Runtime compile
     

     View Slide

106. yours/*
     yours.squashfs
     libsquash, libautoupdate
     Ruby Runtime
     

     View Slide

107. yours/*
     yours.squashfs
     libsquash, libautoupdate
     Ruby Runtime
     

     View Slide

108. yours/*
     yours.squashfs
     libsquash, libautoupdate
     Ruby Runtime
     yours.exe
     Statically Link
     Statically Link
     

     View Slide

109. yours.exe
     Distribute and Enjoy
     

     View Slide

110. github.com/pmq20
     twitter
     @psvr
     

     View Slide

111. Download & Install
     

     View Slide

112. git clone

     https://github.com/
     pmq20/ruby-packer.git
     

     View Slide

113. add bin/rubyc to

     $PATH or %PATH%
     

     View Slide

114. or
     

     View Slide

115. http://enclose.io/rubyc
     

     View Slide

116. View Slide

117. Windows
     • SquashFS Tools 4.3
     • Visual Studio 2015 Update 3, all editions including
     the Community edition (remember to select

     “Common Tools for Visual C++ 2015" feature
     during installation).
     • Ruby
     

     View Slide

118. macOS
     • SquashFS Tools 4.3: brew install squashfs
     • Xcode, You also need to install the Command Line
     Tools via Xcode.
     • Ruby
     

     View Slide

119. Linux
     • SquashFS Tools 4.3: sudo yum install squashfs-
     tools or sudo apt-get install squashfs-tools
     • gcc or clang
     • GNU Make
     • Ruby
     

     View Slide

120. Tips
     

     View Slide

121. Use --tmpdir=/a/fixed/location
     

     View Slide

122. Use Windows

     with a big Virtual Memory
     

     View Slide

123. Use older Linux, e.g. CentOS release 5.8

     with gcc and g++ 4.8

     (possibly from devtoolset-2 of slc5-devtoolset)
     

     View Slide

124. Use older Mac, e.g. Mac OS X 10.7 Lion
     with Xcode 4.6.3
     

     View Slide

125. Check dependencies using
     Dependency Walker, otool, ldd

     before releasing
     

     View Slide

126. See Also
     

     View Slide

127. “Real” Compiling
     KEVIN DEISZ @ RubyConf 2017
     Compiling Ruby
     

     View Slide

128. “Real” Compiling
     Koichi Sasada @ RailConf 2016
     Precompiling Ruby scripts

     Myth and Fact
     

     View Slide

129. github.com/pmq20
     twitter
     @psvr
     

     View Slide