Evaluate Ruby Without Ruby Takashi Kokubun / Cookpad Inc. November 10th, 2016 RubyConf 2016

5BLBTIJ,PLVCVO!LLVCVO %FWFMPQFS1SPEVDUJWJUZ(SPVQ $PPLQBE*OD IUUQTHJUIVCDPNLLVCVO

Agenda • What’s Ruby DSL? • Why did we want to evaluate Ruby DSL without MRI? • Introduction to mruby • 2 ways to build portable CLI apps configured by Ruby DSL

What’s Ruby DSL? What’s Ruby DSL? Why Ruby DSL without MRI? Introduction to mruby Building portable CLI apps

Ruby DSL • Domain Specific Language is a language specialized to a particular application domain • “Ruby DSL” is a DSL defined in terms of Ruby language

Example Ruby DSL for server provisioning package 'nginx' do action :install end service 'nginx' do action [:enable, :start] end

Example Ruby DSL for server provisioning package 'nginx' do action :install end service 'nginx' do action [:enable, :start] end CentOS yum -y install nginx Ubuntu apt-get -y install nginx

Example Ruby DSL for server provisioning package 'nginx' do action :install end service 'nginx' do action [:enable, :start] end CentOS chkconfig nginx on service nginx start Ubuntu systemctl enable nginx systemctl start nginx

Why Ruby DSL? package 'nginx' do action :install end service 'nginx' do action [:enable, :start] end • Abstraction for a domain- specific purpose • Happy to write Ruby! • It’s boring to write this in JSON

How is Ruby DSL implemented? • Have you ever implemented Ruby DSL?

How is Ruby DSL implemented? Expected behavior script = <<-EOS package 'nginx' EOS dsl = DSL.new dsl.instance_eval(script) dsl.result #=> [["package", "nginx"]]

How is Ruby DSL implemented? Expected behavior Implementation class DSL def package(name, &block) result << ['package', name] end def result; @result ||= []; end end script = <<-EOS package 'nginx' EOS dsl = DSL.new dsl.instance_eval(script) dsl.result #=> [["package", "nginx"]]

How is Ruby DSL implemented? Expected behavior class DSL def package(name, &block) result << ['package', name] end def result; @result ||= []; end end Implementation script = <<-EOS package 'nginx' do action :install end EOS dsl = DSL.new dsl.instance_eval(script) dsl.result #=> [["package", "nginx", {:action=>[:install]}]]

How is Ruby DSL implemented? Expected behavior class DSL def package(name, &block) pkg = Package.new pkg.instance_exec(&block) result << ['package', name, pkg.attrs] end def result; @result ||= []; end end class Package def action(actions) attrs[:action] = Array(actions) end def attrs; @attrs ||= {}; end end Implementation script = <<-EOS package 'nginx' do action :install end EOS dsl = DSL.new dsl.instance_eval(script) dsl.result #=> [["package", "nginx", {:action=>[:install]}]]

How is Ruby DSL implemented? • Usually Ruby DSL is implemented with MRI • Using instance_eval, instance_exec, …etc • But we wanted to evaluate Ruby DSL without MRI • Why? Let’s have a look.

Why Ruby DSL without MRI? What’s Ruby DSL? Why Ruby DSL without MRI? Introduction to mruby Building portable CLI apps

Ruby DSL use case at Cookpad • Development environment bootstrap for macOS and Linux • We wanted to use Ruby DSL to simply describe operations for multiple OS distributions • For maintenance cost, we used the same Ruby DSL as a server provisioning tool we develop and use, which is Itamae • https://github.com/itamae-kitchen/itamae

Ruby DSL use case at Cookpad package 'memcached' package 'postgresql'

Ruby DSL use case at Cookpad package 'memcached' package 'postgresql' macOS brew install memcached brew install postgresql

Ruby DSL use case at Cookpad package 'memcached' package 'postgresql' macOS Ubuntu apt-get install -y memcached apt-get install -y postgresql brew install memcached brew install postgresql

Problem to bootstrap development environment using MRI • There are difficulties to use MRI on an initial environment • In macOS, we don’t want to use end-of-life Ruby, which is shipped with latest macOS • In Linux, most Linux distro doesn't ship MRI out of the box • We don’t want to manually install MRI prior to bootstrap

Problem to bootstrap development environment using MRI • If you use MRI for user’s environment, you have to care about how a gem is set up everytime you run Ruby DSL • Are rbenv and MRIs already installed? • Is the gem installed to MRI currently selected by rbenv? • Packaging ruby, gem, … to run just one gem is a little overkill

How can we remove dependency of MRI? • All these problems are resolved if it doesn’t depend on MRI • Can we embed Ruby VM into our software? • OK, it’s time to use mruby!

Introduction to mruby What’s Ruby DSL? Why Ruby DSL without MRI? Introduction to mruby Building portable CLI apps

mruby • Lightweight implementation of Ruby • Embeddable in: • Hardware • Software

Embedding mruby into software • You can evaluate Ruby scripts from any software by embedding mruby • It allows us to build a software evaluating Ruby DSL and independent to MRI

How to embed and use mruby 1. git clone https://github.com/mruby/mruby; cd mruby; make 2. Link mruby/build/host/lib/libmruby.a to your software 3. Call mruby’s C API • Or use its binding for a language you use

How to embed and use mruby #include #include #include #include int main(void) { mrb_state *mrb = mrb_open(); mrb_value str = mrb_load_string(mrb, "%w[hello world].join"); // Prints "helloworld\n" printf("%s\n", mrb_string_value_cstr(mrb, &str)); return 0; }

Now we can embed mruby into our software. Is it enough? • No, our use case is development environment bootstrap and we should also care about dependencies other than MRI • We want to build it as a “single binary” • There are some good ways to do that

Building portable CLI apps configured by Ruby DSL What’s Ruby DSL? Why Ruby DSL without MRI? Introduction to mruby Building portable CLI apps

2 ways to build portable CLI apps configured by Ruby DSL • go-mruby • Use mruby only for evaluating Ruby DSL and implement others in Go language • mruby-cli • Implement both Ruby DSL and others using mruby

2 ways to build portable CLI apps configured by Ruby DSL • I experimentally re-implemented our configuration management tool “Itamae” in both ways • Let’s compare these methods!

https://github.com/mitchellh/go-mruby 1. go-mruby

Example 1: Itamae Go • Itamae Go • Alternative implementation of “Itamae”, a configuration management tool inspired by Chef • Single binary, built with Go language and go-mruby • https://github.com/k0kubun/itamae-go

How to run Ruby scripts from Go 1. git clone https://github.com/mruby/mruby; cd mruby; make 2. git clone https://github.com/mitchellh/go-mruby; cp /path/to/mruby/build/host/libmruby.a ./go-mruby 3. import "github.com/mitchellh/go-mruby" and call mruby APIs

// Evaluate Ruby from Go (mitchellh/go-mruby) mrb := mruby.NewMrb() // Call mrb_load_string str := mrb.LoadString("%w[hello world].join") // Prints "helloworld\n" fmt.Println(mrb.StringValue(str)) How to run Ruby scripts from Go

How to implement Ruby DSL in Go • To implement IO operations in Go, it is necessary to get data for Go from an evaluation result of Ruby DSL • I implemented method for Ruby DSL in Go language

How to implement Ruby DSL in Go func Package(mrb *mruby.Mrb, self *mruby.MrbValue) (mruby.Value, mruby.Value) { return mruby.Nil, nil } // In another place cDSL.DefineMethod( “package", Package, mruby.ArgsReq(1))

How to implement Ruby DSL in Go func Package(mrb *mruby.Mrb, self *mruby.MrbValue) (mruby.Value, mruby.Value) { args := mrb.GetArgs() pkg := PackageResource{Name: args[0].String()} resources = append(resources, pkg) return mruby.Nil, nil } // In another place cDSL.DefineMethod( “package", Package, mruby.ArgsReq(1)) type PackageResource struct { Name string }

How to implement Ruby DSL in Go func Package(mrb *mruby.Mrb, self *mruby.MrbValue) (mruby.Value, mruby.Value) { args := mrb.GetArgs() pkg := PackageResource{Name: args[0].String()} if len(args) > 1 { context, _ := mrb.LoadString("Package.new") context.CallBlock("instance_eval", args[1]) attrs, _ := context.Call("attributes") action, _ := mrb.LoadString(":action") value, _ := attrs.Call("[]", action) pkg.Action = value.Array() } resources = append(resources, pkg) return mruby.Nil, nil } // In another place cDSL.DefineMethod( “package", Package, mruby.ArgsReq(1)) type PackageResource struct { Name string Action []string }

It was a little hard to maintain • Writing bridge to connect Ruby and Go world is really boring • Checking errors for all Ruby method calls • Required to use 3 languages: Ruby, Go, C (for mrbgem) • You need to prepare environment for cross compiling by yourself

Why did we need Go language? • Purposes • To make it as a single binary • To implement IO operations for development environment bootstrap • Can't we achieve them without Go language? • Try mruby-cli!

https://github.com/hone/mruby-cli 2. mruby-cli

Example 2: MItamae • MItamae • Alternative implementation of Itamae and Itamae Go, which is actively maintained • Single binary, built with mruby-cli and written in pure Ruby • https://github.com/k0kubun/mitamae

What is mruby-cli? • A cross-compile platform to build CLI apps using mruby • https://github.com/hone/mruby-cli • Eric gave a talk "Building maintainable command-line tools with MRuby" this morning • If you've missed it, watch the video later

How to use mruby-cli 1. Download mruby-cli command from https://github.com/hone/mruby-cli/releases 2. Generate boilerplate by mruby-cli -s app_name 3. cd app_name; docker-compose run compile • You can get binaries in ./mruby/build/*/bin/app_name

How to implement Ruby DSL with mruby-cli • Almost the same way as MRI • Required to add some “mrbgems” to call methods which is not provided by default • mruby-eval for instance_eval • mruby-object-ext for instance_exec

What is mrbgem? • It’s like rubygem for MRI • Unlike rubygem, it’s compiled together when building mruby • So we can use it without adding dependency to a binary • Some of built-in features in MRI are separated to mrbgem in mruby

Porting existing stdlibs and rubygems to mrbgems • It’s easy when they are written in Ruby • Most features required for Ruby programming except keyword arguments are available at existing mrbgems • mrbgems I created to implement MItamae: • mruby-shellwords • mruby-hashie

Implementing IO operations with mruby • There are already IO.popen and Kernel.` (`command`) • But I wanted to capture stdout, stderr and status separately • There were no Open3 and Kernel.#spawn • It was necessary to write C to implement Kernel.#spawn

Incomplete implementation of spawn to implement Open3.capture3 static mrb_value mrb_open3_spawn(mrb_state *mrb, mrb_value self) { const char **cmd; pid_t pid; mrb_value *argv; mrb_int argc, out_dst, err_dst; mrb_get_args(mrb, "*", &argv, &argc); cmd = mrb_str_buf_to_cstr_buf(mrb, argv, argc-1); open3_spawn_process_options(mrb, argv[argc-1], &out_dst, &err_dst); pid = fork(); if (pid == 0) { dup2(out_dst, STDOUT_FILENO); dup2(err_dst, STDERR_FILENO); execvp(cmd[0], cmd); } return mrb_fixnum_value(pid); }

Finally I could built mruby-open3 that has only Open3.capture3 module Open3 # @param [Array] - command to execute # @return [String, String, Process::Status] - stdout, stderr, status def capture3(*cmd) out_r, out_w = IO.pipe err_r, err_w = IO.pipe pid = spawn(*cmd, { 'out' => out_w.to_i, 'err' => err_w.to_i }) # … snip … _, status = Process.waitpid2(pid) [stdout, stderr, status] end module_function :capture3 end

Isn’t it hard to implement apps with mruby-cli? • Yes, it’s hard a little when necessary mrbgems are absent • But there are great merits to build apps using mruby-cli

Merits to build CLI apps using mruby-cli • You can distribute apps easily • Not depending on users' environment • Cross-compilable

Merits to build CLI apps using mruby-cli • Fast startup on command execution $ time itamae version Itamae v1.9.9 itamae version 0.54s user 0.22s system 98% cpu 0.776 total $ time mitamae version MItamae v0.6.0 mitamae version 0.02s user 0.00s system 85% cpu 0.023 total on 1.3 GHz Intel Core m7

Merits to build CLI apps using mruby-cli • Maintainable • No bridging code • Happy to write all implementation in Ruby!

Conclusion • To simplify development environment bootstrap, I made a configuration management tool configured by Ruby DSL as a single binary using mruby • https://github.com/k0kubun/mitamae • There are 2 ways to build a tool configured by Ruby DSL as a single binary • If there’s a reason to use Go language, use go-mruby • Otherwise mruby-cli is recommended