Vladimir Dementyev Evil Martians The why’s and how’s of transpiling Ruby

palkan_tula palkan RubyKaigi‘20 Ruby 2

palkan_tula palkan RubyKaigi‘20 Transpiler 3 Illustration from Ruby Weekly #477 Source-to-source compiler

palkan_tula palkan RubyKaigi‘20 The most popular transpiler 4

palkan_tula palkan RubyKaigi‘20 Yet another transpiler 5

palkan_tula palkan RubyKaigi‘20 6 Ruby vs. Transpiling

palkan_tula palkan RubyKaigi‘20 Ruby 3.02.8 7

palkan_tula palkan RubyKaigi‘20 Ruby is evolving faster than ever 8

palkan_tula palkan RubyKaigi‘20 Ruby 2.8 9 def greet(val) = case val in hello: hello if hello =~ /human/i '' in hello: 'martian' '' end greet(hello: 'martian') => greeting puts greeting # =>

palkan_tula palkan RubyKaigi‘20 Ruby 2.8 10 def greet(val) = case val in hello: hello if hello =~ /human/i '' in hello: 'martian' '' end greet(hello: 'martian') => greeting puts greeting # => Pattern matching (2.7)

palkan_tula palkan RubyKaigi‘20 Ruby 2.8 11 def greet(val) = case val in hello: hello if hello =~ /human/i '' in hello: 'martian' '' end greet(hello: 'martian') => greeting puts greeting # => Endless method (2.8)

palkan_tula palkan RubyKaigi‘20 Ruby 2.8 12 def greet(val) = case val in hello: hello if hello =~ /human/i '' in hello: 'martian' '' end greet(hello: 'martian') => greeting puts greeting # => Rightward assignment (2.8)

palkan_tula palkan RubyKaigi‘20 How to use upcoming features today? 13

palkan_tula palkan RubyKaigi‘20 Transpiler for Ruby 14

palkan_tula palkan RubyKaigi‘20 $ ruby -v
 2.5.3p105
 $ ruby -ruby-next -e "
 case {hello: 'こんにちは'}
 in hello: hello if hello =~ /[\p{Hiragana}]+/
 puts '私はウラジミールです'
 in hello: hello if hello =~ /[a-z]+/i
 puts 'My name is Vladimir'
 end"
 私はウラジミールです 15 tl;dr

palkan_tula palkan RubyKaigi‘20 About me 16 github.com/palkan

palkan_tula palkan RubyKaigi‘20 evilmartians.com 17

palkan_tula palkan RubyKaigi‘20 18 evilmartians.com

palkan_tula palkan RubyKaigi‘20 evilmartians.com 19 Ruby Next Transpiler for Ruby

palkan_tula palkan RubyKaigi‘20 evl.ms/blog 20

palkan_tula palkan RubyKaigi‘20 Why transpiling Ruby? 21

palkan_tula palkan RubyKaigi‘20 Backporting Why transpiling? 22

palkan_tula palkan RubyKaigi‘20 stats.rubygems.org 23

palkan_tula palkan RubyKaigi‘20 Gem authors should stick to older versions 24

palkan_tula palkan RubyKaigi‘20 Story: Hanami::API 25

palkan_tula palkan RubyKaigi‘20 26 Story: Hanami::API

palkan_tula palkan RubyKaigi‘20 Backporting Interoperability Why transpiling? 27

palkan_tula palkan RubyKaigi‘20 JRuby 9.3 — 2.6 TruffleRuby — 2.6 mruby ~ 2.7 (no pattern matching) Opal, RubyMotion, Artichoke — ??? Syntax support 28

palkan_tula palkan RubyKaigi‘20 Backporting Interoperability ... Why transpiling? 29

palkan_tula palkan RubyKaigi‘20 How to transpile Ruby? 30

palkan_tula palkan RubyKaigi‘20 Transpiler is a source-to- source compiler 31

palkan_tula palkan RubyKaigi‘20 Compilers 32 @pgurtovaya

palkan_tula palkan RubyKaigi‘20 Parse Analyze/optimize Generate Transpiling 33

palkan_tula palkan RubyKaigi‘20 Transpiling 34 source code new source code AST new AST

palkan_tula palkan RubyKaigi‘20 Ripper RubyVM::AbstractSyntaxTree Parser Source to AST 35

palkan_tula palkan RubyKaigi‘20 Written in pure Ruby Version-independent Bullet-proofed (e.g., by RuboCop) Parser 36

palkan_tula palkan RubyKaigi‘20 github.com/whitequark/parser 37

palkan_tula palkan RubyKaigi‘20 def transpile(source) rewriters.inject(source) do |src, rewriter| buffer = Parser ::Source ::Buffer.new("") buffer.source = src rewriter.new.rewrite(buffer, parse(src)) end end 38

palkan_tula palkan RubyKaigi‘20 module Rewriters class ArgsForward < Base def on_forward_args(node) replace(node.loc.expression, "(* __rest __, & __block __") end def on_send(node) return super(node) unless node.children[2]&.type == :forwarded_args replace(node.children[2].loc.expression, "* __rest __, & __block __") end end end 39 Rewriter

palkan_tula palkan RubyKaigi‘20 Ruby Next Transpiling 40 source code new source code AST in-place rewrite unparse bits of AST

palkan_tula palkan RubyKaigi‘20 41 def fizzbuzz(num) case [num % 3, num % 5] in [0, 0] then 'FizzBuzz' in [0, _] then 'Fizz' in [_, 0] then 'Buzz' end end Example

palkan_tula palkan RubyKaigi‘20 42 def fizzbuzz(num) rems = [num % 3, num % 5] if rems == [0, 0] 'FizzBuzz' elsif rems[0] == 0 'Fizz' elsif rems[1] == 1 'Buzz' else raise "NoMatchingPattern" end end def fizzbuzz(num) case [num % 3, num % 5] in [0, 0] then 'FizzBuzz' in [0, _] then 'Fizz' in [_, 0] then 'Buzz' end end 2.7 2.5 "Transpile" by hand

palkan_tula palkan RubyKaigi‘20 43 def fizzbuzz(num) case; when ( __m __ = [(num % 3), (num % 5)]) && false when ( __p_1 __ = ( __m __.respond_to?(:deconstruct) && ((( __m_arr __ = __m __.deconstruct) || true) && ((Array === __m_arr __) || Kernel.raise(TypeError, "#deconstruct must return Array"))))) && (( __p_2 __ = (2 == __m_arr __.size)) && ((0 === __m_arr __[0]) && (0 === __m_arr __[1]))) then 'FizzBuzz' when __p_1 __ && ( __p_2 __ && (0 === __m_arr __[0])) then 'Fizz' when __p_1 __ && ( __p_2 __ && (0 === __m_arr __[1])) then 'Buzz'; else; Kernel.raise(NoMatchingPatternError, __m __.inspect) end end def fizzbuzz(num) case [num % 3, num % 5] in [0, 0] then 'FizzBuzz' in [0, _] then 'Fizz' in [_, 0] then 'Buzz' end end For humans (2.7) For machines (2.5) Transpile with Ruby Next

palkan_tula palkan RubyKaigi‘20 Parse Analyze/optimize Generate Transpiling 44

palkan_tula palkan RubyKaigi‘20 module Rewriters class PatternMatching < Base def on_case_match(node) # ~800 LOC end end end 45 Pattern matching

palkan_tula palkan RubyKaigi‘20 def call(val) status, headers, body = 200, {}, "" case val in [String => body] [status, headers, [body]] in [Integer => status] [status, headers, [body]] in [Integer, String] => response [response[0], headers, [response[1]]] in [Integer, Hash, String] => response headers.merge!(response[1]) [response[0], headers, [response[2]]] end end 46 Comparison: transpiled (last pattern): 1162200.7 i/s baseline (last pattern): 799739.5 i/s - 1.45x slower Transpiled != Slow

palkan_tula palkan RubyKaigi‘20 47

palkan_tula palkan RubyKaigi‘20 How to integrate a transpiler into an interpreted language? 48

palkan_tula palkan RubyKaigi‘20 Gems: transpile at “build”/ release time Apps/Scripts: transpile at runtime 49

palkan_tula palkan RubyKaigi‘20 $LOAD_PATH: where to search for “features” $LOADED_FEATURES: required files Two pillars of require 50

palkan_tula palkan RubyKaigi‘20 Add transpiled files to releases No additional runtime deps* Nextify gems 51 * polyfills might still be required

palkan_tula palkan RubyKaigi‘20 $ ruby-next nextify ./lib -V
 Generated: ./lib/.rbnext/2.7/rubanok/rule.rb
 Generated: ./lib/.rbnext/2.8/rubanok/dsl/matching.rb 52

palkan_tula palkan RubyKaigi‘20 # lib/my_gem.rb require "ruby-next/language/setup" RubyNext ::Language.setup_gem_load_path 53

palkan_tula palkan RubyKaigi‘20 def setup_gem_load_path lib_dir = File.dirname(caller_locations(1, 1).first.path) current_index = $LOAD_PATH.index(lib_dir) next_dir = File.join(lib_dir, ".rbnext") if File.exist?(next_dir) $LOAD_PATH.insert current_index, next_dir current_index += 1 end end 54

palkan_tula palkan RubyKaigi‘20 Hijack Kernel#require & co Reimplement require mechanism Runtime 55

palkan_tula palkan RubyKaigi‘20 module Kernel module_function alias_method :require_without_ruby_next, :require def require(path) RubyNext.require(path) rescue => e warn "Failed to require ' #{path}': #{e.message}" require_without_ruby_next(path) end end 56 Kernel#require *Don't try this at home

palkan_tula palkan RubyKaigi‘20 module RubyNext module_function def require(path) realpath = resolve_feature_path(path) return false if $LOADED_FEATURES.include?(realpath) File.read(realpath) .then(&Language.:transform).then do |source| RubyVM ::InstructionSequence.compile(code, filepath).then(&:eval) $LOADED_FEATURES << path true end end end 57 RubyNext.require

palkan_tula palkan RubyKaigi‘20 How to use a transpiler with compilable Rubies? 58

palkan_tula palkan RubyKaigi‘20 Transpile source files before compiling Replace target files with the transpiled versions at build time mruby 59

palkan_tula palkan RubyKaigi‘20 desc "transpile source code with ruby-next" task rbnext: [] do Dir.chdir(APP_ROOT) do sh "ruby-next nextify -V" end end Rake ::Task["compile"].enhance [:rbnext] 60 Nextify before compiling

palkan_tula palkan RubyKaigi‘20 using(Module.new do refine MRuby ::Gem ::Specification do def setup_ruby_next!(next_dir: ".rbnext") lib_root = File.join(@dir, @mrblib_dir) next_root = Pathname.new(next_dir).absolute? ? next_dir : File.join(lib_root, next_dir) Dir.glob(" #{next_root}/**/*.rb").each do |next_file| orig_file = next_file.sub(next_root, lib_root) index = @rbfiles.index(orig_file) || raise "Source file not found for: #{next_file}" @rbfiles[index] = next_file end end end end) MRuby ::Gem ::Specification.new("acli") do |spec| # ... spec.setup_ruby_next! end 61 Update rbfiles

palkan_tula palkan RubyKaigi‘20 Backporting Interoperability Evolution Why transpiling? 62

palkan_tula palkan RubyKaigi‘20 Recent Ruby versions added experimental features 63

palkan_tula palkan RubyKaigi‘20 We need an experiment 64

palkan_tula palkan RubyKaigi‘20 The best way to taste new features is to start using them every day 65

palkan_tula palkan RubyKaigi‘20 66 Example: Hash shorthand

palkan_tula palkan RubyKaigi‘20 67 Hash shorthand

palkan_tula palkan RubyKaigi‘20 $ ruby -v
 2.5.3p105
 $ ruby -ruby-next -e "
 event = "RubyKaigi" year = 2020 p {event, year}
 "
 {event: "RubyKaigi", year: 2020} 68 Ruby Next 0.10.0

palkan_tula palkan RubyKaigi‘20 Share your opinion, let's make future Ruby together! 69

palkan_tula palkan RubyKaigi‘20 github.com/ruby-next/ruby-next 70

Thank you! evilmartians.com github.com/ruby-next evilmartians palkan_tula Vladimir Dementyev