[RubyKaigi 2020] The why's and how's of transpiling Ruby

Transpiling is a source-to-source compiling. Why might we need it in Ruby? Compatibility and experiments.

Ruby is evolving fast nowadays. The latest MRI release introduced, for example, the pattern matching syntax. Unfortunately, not everyone is ready to use it yet: gems authors have to support older versions, Ruby implementations are lagging. And it's still experimental, which raises the question: how to evaluate proposals? By backporting them to older Rubies!

I want to discuss these problems and share the story of the Ruby transpiler — Ruby Next. A decent amount of Ruby hackery is guaranteed.

Transcript

1. Vladimir Dementyev Evil Martians The why’s and how’s of transpiling Ruby
2. palkan_tula palkan RubyKaigi‘20 Ruby 2
3. palkan_tula palkan RubyKaigi‘20 Transpiler 3 Illustration from Ruby Weekly #477 Source-to-source compiler
4. palkan_tula palkan RubyKaigi‘20 The most popular transpiler 4
5. palkan_tula palkan RubyKaigi‘20 Yet another transpiler 5
6. palkan_tula palkan RubyKaigi‘20 6 Ruby vs. Transpiling
7. palkan_tula palkan RubyKaigi‘20 Ruby 3.02.8 7
8. palkan_tula palkan RubyKaigi‘20 Ruby is evolving faster than ever 8
9. palkan_tula palkan RubyKaigi‘20 Ruby 2.8 9 def greet(val) = case val in hello: hello if hello =~ /human/i '' in hello: 'martian' '' end greet(hello: 'martian') => greeting puts greeting # =>
10. palkan_tula palkan RubyKaigi‘20 Ruby 2.8 10 def greet(val) = case val in hello: hello if hello =~ /human/i '' in hello: 'martian' '' end greet(hello: 'martian') => greeting puts greeting # => Pattern matching (2.7)
11. palkan_tula palkan RubyKaigi‘20 Ruby 2.8 11 def greet(val) = case val in hello: hello if hello =~ /human/i '' in hello: 'martian' '' end greet(hello: 'martian') => greeting puts greeting # => Endless method (2.8)
12. palkan_tula palkan RubyKaigi‘20 Ruby 2.8 12 def greet(val) = case val in hello: hello if hello =~ /human/i '' in hello: 'martian' '' end greet(hello: 'martian') => greeting puts greeting # => Rightward assignment (2.8)
13. palkan_tula palkan RubyKaigi‘20 How to use upcoming features today? 13
14. palkan_tula palkan RubyKaigi‘20 Transpiler for Ruby 14
15. palkan_tula palkan RubyKaigi‘20 $ ruby -v
 2.5.3p105
 $ ruby -ruby-next -e "
 case {hello: 'こんにちは'}
 in hello: hello if hello =~ /[\p{Hiragana}]+/
 puts '私はウラジミールです'
 in hello: hello if hello =~ /[a-z]+/i
 puts 'My name is Vladimir'
 end"
 私はウラジミールです 15 tl;dr
16. palkan_tula palkan RubyKaigi‘20 About me 16 github.com/palkan
17. palkan_tula palkan RubyKaigi‘20 evilmartians.com 17
18. palkan_tula palkan RubyKaigi‘20 18 evilmartians.com
19. palkan_tula palkan RubyKaigi‘20 evilmartians.com 19 Ruby Next Transpiler for Ruby
20. palkan_tula palkan RubyKaigi‘20 evl.ms/blog 20
21. palkan_tula palkan RubyKaigi‘20 Why transpiling Ruby? 21
22. palkan_tula palkan RubyKaigi‘20 Backporting Why transpiling? 22
23. palkan_tula palkan RubyKaigi‘20 stats.rubygems.org 23
24. palkan_tula palkan RubyKaigi‘20 Gem authors should stick to older versions 24
25. palkan_tula palkan RubyKaigi‘20 Story: Hanami::API 25
26. palkan_tula palkan RubyKaigi‘20 26 Story: Hanami::API
27. palkan_tula palkan RubyKaigi‘20 Backporting Interoperability Why transpiling? 27
28. palkan_tula palkan RubyKaigi‘20 JRuby 9.3 — 2.6 TruffleRuby — 2.6 mruby ~ 2.7 (no pattern matching) Opal, RubyMotion, Artichoke — ??? Syntax support 28
29. palkan_tula palkan RubyKaigi‘20 Backporting Interoperability ... Why transpiling? 29
30. palkan_tula palkan RubyKaigi‘20 How to transpile Ruby? 30
31. palkan_tula palkan RubyKaigi‘20 Transpiler is a source-to- source compiler 31
32. palkan_tula palkan RubyKaigi‘20 Compilers 32 @pgurtovaya
33. palkan_tula palkan RubyKaigi‘20 Parse Analyze/optimize Generate Transpiling 33
34. palkan_tula palkan RubyKaigi‘20 Transpiling 34 source code new source code AST new AST
35. palkan_tula palkan RubyKaigi‘20 Ripper RubyVM::AbstractSyntaxTree Parser Source to AST 35
36. palkan_tula palkan RubyKaigi‘20 Written in pure Ruby Version-independent Bullet-proofed (e.g., by RuboCop) Parser 36
37. palkan_tula palkan RubyKaigi‘20 github.com/whitequark/parser 37
38. palkan_tula palkan RubyKaigi‘20 def transpile(source) rewriters.inject(source) do |src, rewriter| buffer = Parser ::Source ::Buffer.new("<dynamic>") buffer.source = src rewriter.new.rewrite(buffer, parse(src)) end end 38
39. palkan_tula palkan RubyKaigi‘20 module Rewriters class ArgsForward < Base def on_forward_args(node) replace(node.loc.expression, "(* __rest __, & __block __") end def on_send(node) return super(node) unless node.children[2]&.type == :forwarded_args replace(node.children[2].loc.expression, "* __rest __, & __block __") end end end 39 Rewriter
40. palkan_tula palkan RubyKaigi‘20 Ruby Next Transpiling 40 source code new source code AST in-place rewrite unparse bits of AST
41. palkan_tula palkan RubyKaigi‘20 41 def fizzbuzz(num) case [num % 3, num % 5] in [0, 0] then 'FizzBuzz' in [0, _] then 'Fizz' in [_, 0] then 'Buzz' end end Example
42. palkan_tula palkan RubyKaigi‘20 42 def fizzbuzz(num) rems = [num % 3, num % 5] if rems == [0, 0] 'FizzBuzz' elsif rems[0] == 0 'Fizz' elsif rems[1] == 1 'Buzz' else raise "NoMatchingPattern" end end def fizzbuzz(num) case [num % 3, num % 5] in [0, 0] then 'FizzBuzz' in [0, _] then 'Fizz' in [_, 0] then 'Buzz' end end 2.7 2.5 "Transpile" by hand
43. palkan_tula palkan RubyKaigi‘20 43 def fizzbuzz(num) case; when ( __m __ = [(num % 3), (num % 5)]) && false when ( __p_1 __ = ( __m __.respond_to?(:deconstruct) && ((( __m_arr __ = __m __.deconstruct) || true) && ((Array === __m_arr __) || Kernel.raise(TypeError, "#deconstruct must return Array"))))) && (( __p_2 __ = (2 == __m_arr __.size)) && ((0 === __m_arr __[0]) && (0 === __m_arr __[1]))) then 'FizzBuzz' when __p_1 __ && ( __p_2 __ && (0 === __m_arr __[0])) then 'Fizz' when __p_1 __ && ( __p_2 __ && (0 === __m_arr __[1])) then 'Buzz'; else; Kernel.raise(NoMatchingPatternError, __m __.inspect) end end def fizzbuzz(num) case [num % 3, num % 5] in [0, 0] then 'FizzBuzz' in [0, _] then 'Fizz' in [_, 0] then 'Buzz' end end For humans (2.7) For machines (2.5) Transpile with Ruby Next
44. palkan_tula palkan RubyKaigi‘20 Parse Analyze/optimize Generate Transpiling 44
45. palkan_tula palkan RubyKaigi‘20 module Rewriters class PatternMatching < Base def on_case_match(node) # ~800 LOC end end end 45 Pattern matching
46. palkan_tula palkan RubyKaigi‘20 def call(val) status, headers, body = 200, {}, "" case val in [String => body] [status, headers, [body]] in [Integer => status] [status, headers, [body]] in [Integer, String] => response [response[0], headers, [response[1]]] in [Integer, Hash, String] => response headers.merge!(response[1]) [response[0], headers, [response[2]]] end end 46 Comparison: transpiled (last pattern): 1162200.7 i/s baseline (last pattern): 799739.5 i/s - 1.45x slower Transpiled != Slow
47. palkan_tula palkan RubyKaigi‘20 47
48. palkan_tula palkan RubyKaigi‘20 How to integrate a transpiler into an interpreted language? 48
49. palkan_tula palkan RubyKaigi‘20 Gems: transpile at “build”/ release time Apps/Scripts: transpile at runtime 49
50. palkan_tula palkan RubyKaigi‘20 $LOAD_PATH: where to search for “features” $LOADED_FEATURES: required files Two pillars of require 50
51. palkan_tula palkan RubyKaigi‘20 Add transpiled files to releases No additional runtime deps* Nextify gems 51 * polyfills might still be required
52. palkan_tula palkan RubyKaigi‘20 $ ruby-next nextify ./lib -V
 Generated: ./lib/.rbnext/2.7/rubanok/rule.rb
 Generated: ./lib/.rbnext/2.8/rubanok/dsl/matching.rb 52
53. palkan_tula palkan RubyKaigi‘20 # lib/my_gem.rb require "ruby-next/language/setup" RubyNext ::Language.setup_gem_load_path 53
54. palkan_tula palkan RubyKaigi‘20 def setup_gem_load_path lib_dir = File.dirname(caller_locations(1, 1).first.path) current_index = $LOAD_PATH.index(lib_dir) next_dir = File.join(lib_dir, ".rbnext") if File.exist?(next_dir) $LOAD_PATH.insert current_index, next_dir current_index += 1 end end 54
55. palkan_tula palkan RubyKaigi‘20 Hijack Kernel#require & co Reimplement require mechanism Runtime 55
56. palkan_tula palkan RubyKaigi‘20 module Kernel module_function alias_method :require_without_ruby_next, :require def require(path) RubyNext.require(path) rescue => e warn "Failed to require ' #{path}': #{e.message}" require_without_ruby_next(path) end end 56 Kernel#require *Don't try this at home
57. palkan_tula palkan RubyKaigi‘20 module RubyNext module_function def require(path) realpath = resolve_feature_path(path) return false if $LOADED_FEATURES.include?(realpath) File.read(realpath) .then(&Language.:transform).then do |source| RubyVM ::InstructionSequence.compile(code, filepath).then(&:eval) $LOADED_FEATURES << path true end end end 57 RubyNext.require
58. palkan_tula palkan RubyKaigi‘20 How to use a transpiler with compilable Rubies? 58
59. palkan_tula palkan RubyKaigi‘20 Transpile source files before compiling Replace target files with the transpiled versions at build time mruby 59
60. palkan_tula palkan RubyKaigi‘20 desc "transpile source code with ruby-next" task rbnext: [] do Dir.chdir(APP_ROOT) do sh "ruby-next nextify -V" end end Rake ::Task["compile"].enhance [:rbnext] 60 Nextify before compiling
61. palkan_tula palkan RubyKaigi‘20 using(Module.new do refine MRuby ::Gem ::Specification do def setup_ruby_next!(next_dir: ".rbnext") lib_root = File.join(@dir, @mrblib_dir) next_root = Pathname.new(next_dir).absolute? ? next_dir : File.join(lib_root, next_dir) Dir.glob(" #{next_root}/**/*.rb").each do |next_file| orig_file = next_file.sub(next_root, lib_root) index = @rbfiles.index(orig_file) || raise "Source file not found for: #{next_file}" @rbfiles[index] = next_file end end end end) MRuby ::Gem ::Specification.new("acli") do |spec| # ... spec.setup_ruby_next! end 61 Update rbfiles
62. palkan_tula palkan RubyKaigi‘20 Backporting Interoperability Evolution Why transpiling? 62
63. palkan_tula palkan RubyKaigi‘20 Recent Ruby versions added experimental features 63
64. palkan_tula palkan RubyKaigi‘20 We need an experiment 64
65. palkan_tula palkan RubyKaigi‘20 The best way to taste new features is to start using them every day 65
66. palkan_tula palkan RubyKaigi‘20 66 Example: Hash shorthand
67. palkan_tula palkan RubyKaigi‘20 67 Hash shorthand
68. palkan_tula palkan RubyKaigi‘20 $ ruby -v
 2.5.3p105
 $ ruby -ruby-next -e "
 event = "RubyKaigi" year = 2020 p {event, year}
 "
 {event: "RubyKaigi", year: 2020} 68 Ruby Next 0.10.0
69. palkan_tula palkan RubyKaigi‘20 Share your opinion, let's make future Ruby together! 69
70. palkan_tula palkan RubyKaigi‘20 github.com/ruby-next/ruby-next 70
71. Thank you! evilmartians.com github.com/ruby-next evilmartians palkan_tula Vladimir Dementyev