[RubyKaigi 2020] The why's and how's of transpiling Ruby

52cc8a838bf44a589d2572833b2dd1b9?s=47 Vlad Dem
September 04, 2020

[RubyKaigi 2020] The why's and how's of transpiling Ruby

Transpiling is a source-to-source compiling. Why might we need it in Ruby? Compatibility and experiments.

Ruby is evolving fast nowadays. The latest MRI release introduced, for example, the pattern matching syntax. Unfortunately, not everyone is ready to use it yet: gems authors have to support older versions, Ruby implementations are lagging. And it's still experimental, which raises the question: how to evaluate proposals? By backporting them to older Rubies!

I want to discuss these problems and share the story of the Ruby transpiler — Ruby Next. A decent amount of Ruby hackery is guaranteed.

52cc8a838bf44a589d2572833b2dd1b9?s=128

Vlad Dem

September 04, 2020
Tweet

Transcript

  1. Vladimir Dementyev Evil Martians The why’s and how’s of transpiling

    Ruby
  2. palkan_tula palkan RubyKaigi‘20 Ruby 2

  3. palkan_tula palkan RubyKaigi‘20 Transpiler 3 Illustration from Ruby Weekly #477

    Source-to-source compiler
  4. palkan_tula palkan RubyKaigi‘20 The most popular transpiler 4

  5. palkan_tula palkan RubyKaigi‘20 Yet another transpiler 5

  6. palkan_tula palkan RubyKaigi‘20 6 Ruby vs. Transpiling

  7. palkan_tula palkan RubyKaigi‘20 Ruby 3.02.8 7

  8. palkan_tula palkan RubyKaigi‘20 Ruby is evolving faster than ever 8

  9. palkan_tula palkan RubyKaigi‘20 Ruby 2.8 9 def greet(val) = case

    val in hello: hello if hello =~ /human/i '' in hello: 'martian' '' end greet(hello: 'martian') => greeting puts greeting # =>
  10. palkan_tula palkan RubyKaigi‘20 Ruby 2.8 10 def greet(val) = case

    val in hello: hello if hello =~ /human/i '' in hello: 'martian' '' end greet(hello: 'martian') => greeting puts greeting # => Pattern matching (2.7)
  11. palkan_tula palkan RubyKaigi‘20 Ruby 2.8 11 def greet(val) = case

    val in hello: hello if hello =~ /human/i '' in hello: 'martian' '' end greet(hello: 'martian') => greeting puts greeting # => Endless method (2.8)
  12. palkan_tula palkan RubyKaigi‘20 Ruby 2.8 12 def greet(val) = case

    val in hello: hello if hello =~ /human/i '' in hello: 'martian' '' end greet(hello: 'martian') => greeting puts greeting # => Rightward assignment (2.8)
  13. palkan_tula palkan RubyKaigi‘20 How to use upcoming features today? 13

  14. palkan_tula palkan RubyKaigi‘20 Transpiler for Ruby 14

  15. palkan_tula palkan RubyKaigi‘20 $ ruby -v
 2.5.3p105
 $ ruby -ruby-next

    -e "
 case {hello: 'こんにちは'}
 in hello: hello if hello =~ /[\p{Hiragana}]+/
 puts '私はウラジミールです'
 in hello: hello if hello =~ /[a-z]+/i
 puts 'My name is Vladimir'
 end"
 私はウラジミールです 15 tl;dr
  16. palkan_tula palkan RubyKaigi‘20 About me 16 github.com/palkan

  17. palkan_tula palkan RubyKaigi‘20 evilmartians.com 17

  18. palkan_tula palkan RubyKaigi‘20 18 evilmartians.com

  19. palkan_tula palkan RubyKaigi‘20 evilmartians.com 19 Ruby Next Transpiler for Ruby

  20. palkan_tula palkan RubyKaigi‘20 evl.ms/blog 20

  21. palkan_tula palkan RubyKaigi‘20 Why transpiling Ruby? 21

  22. palkan_tula palkan RubyKaigi‘20 Backporting Why transpiling? 22

  23. palkan_tula palkan RubyKaigi‘20 stats.rubygems.org 23

  24. palkan_tula palkan RubyKaigi‘20 Gem authors should stick to older versions

    24
  25. palkan_tula palkan RubyKaigi‘20 Story: Hanami::API 25

  26. palkan_tula palkan RubyKaigi‘20 26 Story: Hanami::API

  27. palkan_tula palkan RubyKaigi‘20 Backporting Interoperability Why transpiling? 27

  28. palkan_tula palkan RubyKaigi‘20 JRuby 9.3 — 2.6 TruffleRuby — 2.6

    mruby ~ 2.7 (no pattern matching) Opal, RubyMotion, Artichoke — ??? Syntax support 28
  29. palkan_tula palkan RubyKaigi‘20 Backporting Interoperability ... Why transpiling? 29

  30. palkan_tula palkan RubyKaigi‘20 How to transpile Ruby? 30

  31. palkan_tula palkan RubyKaigi‘20 Transpiler is a source-to- source compiler 31

  32. palkan_tula palkan RubyKaigi‘20 Compilers 32 @pgurtovaya

  33. palkan_tula palkan RubyKaigi‘20 Parse Analyze/optimize Generate Transpiling 33

  34. palkan_tula palkan RubyKaigi‘20 Transpiling 34 source code new source code

    AST new AST
  35. palkan_tula palkan RubyKaigi‘20 Ripper RubyVM::AbstractSyntaxTree Parser Source to AST 35

  36. palkan_tula palkan RubyKaigi‘20 Written in pure Ruby Version-independent Bullet-proofed (e.g.,

    by RuboCop) Parser 36
  37. palkan_tula palkan RubyKaigi‘20 github.com/whitequark/parser 37

  38. palkan_tula palkan RubyKaigi‘20 def transpile(source) rewriters.inject(source) do |src, rewriter| buffer

    = Parser ::Source ::Buffer.new("<dynamic>") buffer.source = src rewriter.new.rewrite(buffer, parse(src)) end end 38
  39. palkan_tula palkan RubyKaigi‘20 module Rewriters class ArgsForward < Base def

    on_forward_args(node) replace(node.loc.expression, "(* __rest __, & __block __") end def on_send(node) return super(node) unless node.children[2]&.type == :forwarded_args replace(node.children[2].loc.expression, "* __rest __, & __block __") end end end 39 Rewriter
  40. palkan_tula palkan RubyKaigi‘20 Ruby Next Transpiling 40 source code new

    source code AST in-place rewrite unparse bits of AST
  41. palkan_tula palkan RubyKaigi‘20 41 def fizzbuzz(num) case [num % 3,

    num % 5] in [0, 0] then 'FizzBuzz' in [0, _] then 'Fizz' in [_, 0] then 'Buzz' end end Example
  42. palkan_tula palkan RubyKaigi‘20 42 def fizzbuzz(num) rems = [num %

    3, num % 5] if rems == [0, 0] 'FizzBuzz' elsif rems[0] == 0 'Fizz' elsif rems[1] == 1 'Buzz' else raise "NoMatchingPattern" end end def fizzbuzz(num) case [num % 3, num % 5] in [0, 0] then 'FizzBuzz' in [0, _] then 'Fizz' in [_, 0] then 'Buzz' end end 2.7 2.5 "Transpile" by hand
  43. palkan_tula palkan RubyKaigi‘20 43 def fizzbuzz(num) case; when ( __m

    __ = [(num % 3), (num % 5)]) && false when ( __p_1 __ = ( __m __.respond_to?(:deconstruct) && ((( __m_arr __ = __m __.deconstruct) || true) && ((Array === __m_arr __) || Kernel.raise(TypeError, "#deconstruct must return Array"))))) && (( __p_2 __ = (2 == __m_arr __.size)) && ((0 === __m_arr __[0]) && (0 === __m_arr __[1]))) then 'FizzBuzz' when __p_1 __ && ( __p_2 __ && (0 === __m_arr __[0])) then 'Fizz' when __p_1 __ && ( __p_2 __ && (0 === __m_arr __[1])) then 'Buzz'; else; Kernel.raise(NoMatchingPatternError, __m __.inspect) end end def fizzbuzz(num) case [num % 3, num % 5] in [0, 0] then 'FizzBuzz' in [0, _] then 'Fizz' in [_, 0] then 'Buzz' end end For humans (2.7) For machines (2.5) Transpile with Ruby Next
  44. palkan_tula palkan RubyKaigi‘20 Parse Analyze/optimize Generate Transpiling 44

  45. palkan_tula palkan RubyKaigi‘20 module Rewriters class PatternMatching < Base def

    on_case_match(node) # ~800 LOC end end end 45 Pattern matching
  46. palkan_tula palkan RubyKaigi‘20 def call(val) status, headers, body = 200,

    {}, "" case val in [String => body] [status, headers, [body]] in [Integer => status] [status, headers, [body]] in [Integer, String] => response [response[0], headers, [response[1]]] in [Integer, Hash, String] => response headers.merge!(response[1]) [response[0], headers, [response[2]]] end end 46 Comparison: transpiled (last pattern): 1162200.7 i/s baseline (last pattern): 799739.5 i/s - 1.45x slower Transpiled != Slow
  47. palkan_tula palkan RubyKaigi‘20 47

  48. palkan_tula palkan RubyKaigi‘20 How to integrate a transpiler into an

    interpreted language? 48
  49. palkan_tula palkan RubyKaigi‘20 Gems: transpile at “build”/ release time Apps/Scripts:

    transpile at runtime 49
  50. palkan_tula palkan RubyKaigi‘20 $LOAD_PATH: where to search for “features” $LOADED_FEATURES:

    required files Two pillars of require 50
  51. palkan_tula palkan RubyKaigi‘20 Add transpiled files to releases No additional

    runtime deps* Nextify gems 51 * polyfills might still be required
  52. palkan_tula palkan RubyKaigi‘20 $ ruby-next nextify ./lib -V
 Generated: ./lib/.rbnext/2.7/rubanok/rule.rb


    Generated: ./lib/.rbnext/2.8/rubanok/dsl/matching.rb 52
  53. palkan_tula palkan RubyKaigi‘20 # lib/my_gem.rb require "ruby-next/language/setup" RubyNext ::Language.setup_gem_load_path 53

  54. palkan_tula palkan RubyKaigi‘20 def setup_gem_load_path lib_dir = File.dirname(caller_locations(1, 1).first.path) current_index

    = $LOAD_PATH.index(lib_dir) next_dir = File.join(lib_dir, ".rbnext") if File.exist?(next_dir) $LOAD_PATH.insert current_index, next_dir current_index += 1 end end 54
  55. palkan_tula palkan RubyKaigi‘20 Hijack Kernel#require & co Reimplement require mechanism

    Runtime 55
  56. palkan_tula palkan RubyKaigi‘20 module Kernel module_function alias_method :require_without_ruby_next, :require def

    require(path) RubyNext.require(path) rescue => e warn "Failed to require ' #{path}': #{e.message}" require_without_ruby_next(path) end end 56 Kernel#require *Don't try this at home
  57. palkan_tula palkan RubyKaigi‘20 module RubyNext module_function def require(path) realpath =

    resolve_feature_path(path) return false if $LOADED_FEATURES.include?(realpath) File.read(realpath) .then(&Language.:transform).then do |source| RubyVM ::InstructionSequence.compile(code, filepath).then(&:eval) $LOADED_FEATURES << path true end end end 57 RubyNext.require
  58. palkan_tula palkan RubyKaigi‘20 How to use a transpiler with compilable

    Rubies? 58
  59. palkan_tula palkan RubyKaigi‘20 Transpile source files before compiling Replace target

    files with the transpiled versions at build time mruby 59
  60. palkan_tula palkan RubyKaigi‘20 desc "transpile source code with ruby-next" task

    rbnext: [] do Dir.chdir(APP_ROOT) do sh "ruby-next nextify -V" end end Rake ::Task["compile"].enhance [:rbnext] 60 Nextify before compiling
  61. palkan_tula palkan RubyKaigi‘20 using(Module.new do refine MRuby ::Gem ::Specification do

    def setup_ruby_next!(next_dir: ".rbnext") lib_root = File.join(@dir, @mrblib_dir) next_root = Pathname.new(next_dir).absolute? ? next_dir : File.join(lib_root, next_dir) Dir.glob(" #{next_root}/**/*.rb").each do |next_file| orig_file = next_file.sub(next_root, lib_root) index = @rbfiles.index(orig_file) || raise "Source file not found for: #{next_file}" @rbfiles[index] = next_file end end end end) MRuby ::Gem ::Specification.new("acli") do |spec| # ... spec.setup_ruby_next! end 61 Update rbfiles
  62. palkan_tula palkan RubyKaigi‘20 Backporting Interoperability Evolution Why transpiling? 62

  63. palkan_tula palkan RubyKaigi‘20 Recent Ruby versions added experimental features 63

  64. palkan_tula palkan RubyKaigi‘20 We need an experiment 64

  65. palkan_tula palkan RubyKaigi‘20 The best way to taste new features

    is to start using them every day 65
  66. palkan_tula palkan RubyKaigi‘20 66 Example: Hash shorthand

  67. palkan_tula palkan RubyKaigi‘20 67 Hash shorthand

  68. palkan_tula palkan RubyKaigi‘20 $ ruby -v
 2.5.3p105
 $ ruby -ruby-next

    -e "
 event = "RubyKaigi" year = 2020 p {event, year}
 "
 {event: "RubyKaigi", year: 2020} 68 Ruby Next 0.10.0
  69. palkan_tula palkan RubyKaigi‘20 Share your opinion, let's make future Ruby

    together! 69
  70. palkan_tula palkan RubyKaigi‘20 github.com/ruby-next/ruby-next 70

  71. Thank you! evilmartians.com github.com/ruby-next evilmartians palkan_tula Vladimir Dementyev