Evaluate Ruby Without Ruby

Evaluate Ruby Without Ruby

RubyConf 2016 at Cincinnati
http://rubyconf.org

08d5432a5bc31e6d9edec87b94cb1db1?s=128

Takashi Kokubun

November 10, 2016
Tweet

Transcript

  1. Evaluate Ruby Without Ruby Takashi Kokubun / Cookpad Inc. November

    10th, 2016 RubyConf 2016
  2. 5BLBTIJ,PLVCVO!LLVCVO %FWFMPQFS1SPEVDUJWJUZ(SPVQ $PPLQBE*OD IUUQTHJUIVCDPNLLVCVO

  3. Agenda • What’s Ruby DSL? • Why did we want

    to evaluate Ruby DSL without MRI? • Introduction to mruby • 2 ways to build portable CLI apps configured by Ruby DSL
  4. What’s Ruby DSL? What’s Ruby DSL? Why Ruby DSL without

    MRI? Introduction to mruby Building portable CLI apps
  5. Ruby DSL • Domain Specific Language is a language specialized

    to a particular application domain • “Ruby DSL” is a DSL defined in terms of Ruby language
  6. Example Ruby DSL for server provisioning package 'nginx' do action

    :install end service 'nginx' do action [:enable, :start] end
  7. Example Ruby DSL for server provisioning package 'nginx' do action

    :install end service 'nginx' do action [:enable, :start] end CentOS yum -y install nginx Ubuntu apt-get -y install nginx
  8. Example Ruby DSL for server provisioning package 'nginx' do action

    :install end service 'nginx' do action [:enable, :start] end CentOS chkconfig nginx on service nginx start Ubuntu systemctl enable nginx systemctl start nginx
  9. Why Ruby DSL? package 'nginx' do action :install end service

    'nginx' do action [:enable, :start] end • Abstraction for a domain- specific purpose • Happy to write Ruby! • It’s boring to write this in JSON
  10. How is Ruby DSL implemented? • Have you ever implemented

    Ruby DSL?
  11. How is Ruby DSL implemented? Expected behavior script = <<-EOS

    package 'nginx' EOS dsl = DSL.new dsl.instance_eval(script) dsl.result #=> [["package", "nginx"]]
  12. How is Ruby DSL implemented? Expected behavior Implementation class DSL

    def package(name, &block) result << ['package', name] end def result; @result ||= []; end end script = <<-EOS package 'nginx' EOS dsl = DSL.new dsl.instance_eval(script) dsl.result #=> [["package", "nginx"]]
  13. How is Ruby DSL implemented? Expected behavior class DSL def

    package(name, &block) result << ['package', name] end def result; @result ||= []; end end Implementation script = <<-EOS package 'nginx' do action :install end EOS dsl = DSL.new dsl.instance_eval(script) dsl.result #=> [["package", "nginx", {:action=>[:install]}]]
  14. How is Ruby DSL implemented? Expected behavior class DSL def

    package(name, &block) pkg = Package.new pkg.instance_exec(&block) result << ['package', name, pkg.attrs] end def result; @result ||= []; end end class Package def action(actions) attrs[:action] = Array(actions) end def attrs; @attrs ||= {}; end end Implementation script = <<-EOS package 'nginx' do action :install end EOS dsl = DSL.new dsl.instance_eval(script) dsl.result #=> [["package", "nginx", {:action=>[:install]}]]
  15. How is Ruby DSL implemented? • Usually Ruby DSL is

    implemented with MRI • Using instance_eval, instance_exec, …etc • But we wanted to evaluate Ruby DSL without MRI • Why? Let’s have a look.
  16. Why Ruby DSL without MRI? What’s Ruby DSL? Why Ruby

    DSL without MRI? Introduction to mruby Building portable CLI apps
  17. Ruby DSL use case at Cookpad • Development environment bootstrap

    for macOS and Linux • We wanted to use Ruby DSL to simply describe operations for multiple OS distributions • For maintenance cost, we used the same Ruby DSL as a server provisioning tool we develop and use, which is Itamae • https://github.com/itamae-kitchen/itamae
  18. Ruby DSL use case at Cookpad package 'memcached' package 'postgresql'

  19. Ruby DSL use case at Cookpad package 'memcached' package 'postgresql'

    macOS brew install memcached brew install postgresql
  20. Ruby DSL use case at Cookpad package 'memcached' package 'postgresql'

    macOS Ubuntu apt-get install -y memcached apt-get install -y postgresql brew install memcached brew install postgresql
  21. Problem to bootstrap development environment using MRI • There are

    difficulties to use MRI on an initial environment • In macOS, we don’t want to use end-of-life Ruby, which is shipped with latest macOS • In Linux, most Linux distro doesn't ship MRI out of the box • We don’t want to manually install MRI prior to bootstrap
  22. Problem to bootstrap development environment using MRI • If you

    use MRI for user’s environment, you have to care about how a gem is set up everytime you run Ruby DSL • Are rbenv and MRIs already installed? • Is the gem installed to MRI currently selected by rbenv? • Packaging ruby, gem, … to run just one gem is a little overkill
  23. How can we remove dependency of MRI? • All these

    problems are resolved if it doesn’t depend on MRI • Can we embed Ruby VM into our software? • OK, it’s time to use mruby!
  24. Introduction to mruby What’s Ruby DSL? Why Ruby DSL without

    MRI? Introduction to mruby Building portable CLI apps
  25. mruby • Lightweight implementation of Ruby • Embeddable in: •

    Hardware • Software
  26. Embedding mruby into software • You can evaluate Ruby scripts

    from any software by embedding mruby • It allows us to build a software evaluating Ruby DSL and independent to MRI
  27. How to embed and use mruby 1. git clone https://github.com/mruby/mruby;

    cd mruby; make 2. Link mruby/build/host/lib/libmruby.a to your software 3. Call mruby’s C API • Or use its binding for a language you use
  28. How to embed and use mruby #include <stdio.h> #include <mruby.h>

    #include <mruby/compile.h> #include <mruby/string.h> int main(void) { mrb_state *mrb = mrb_open(); mrb_value str = mrb_load_string(mrb, "%w[hello world].join"); // Prints "helloworld\n" printf("%s\n", mrb_string_value_cstr(mrb, &str)); return 0; }
  29. Now we can embed mruby into our software. Is it

    enough? • No, our use case is development environment bootstrap and we should also care about dependencies other than MRI • We want to build it as a “single binary” • There are some good ways to do that
  30. Building portable CLI apps configured by Ruby DSL What’s Ruby

    DSL? Why Ruby DSL without MRI? Introduction to mruby Building portable CLI apps
  31. 2 ways to build portable CLI apps configured by Ruby

    DSL • go-mruby • Use mruby only for evaluating Ruby DSL and implement others in Go language • mruby-cli • Implement both Ruby DSL and others using mruby
  32. 2 ways to build portable CLI apps configured by Ruby

    DSL • I experimentally re-implemented our configuration management tool “Itamae” in both ways • Let’s compare these methods!
  33. https://github.com/mitchellh/go-mruby 1. go-mruby

  34. Example 1: Itamae Go • Itamae Go • Alternative implementation

    of “Itamae”, a configuration management tool inspired by Chef • Single binary, built with Go language and go-mruby • https://github.com/k0kubun/itamae-go
  35. How to run Ruby scripts from Go 1. git clone

    https://github.com/mruby/mruby; cd mruby; make 2. git clone https://github.com/mitchellh/go-mruby; cp /path/to/mruby/build/host/libmruby.a ./go-mruby 3. import "github.com/mitchellh/go-mruby" and call mruby APIs
  36. // Evaluate Ruby from Go (mitchellh/go-mruby) mrb := mruby.NewMrb() //

    Call mrb_load_string str := mrb.LoadString("%w[hello world].join") // Prints "helloworld\n" fmt.Println(mrb.StringValue(str)) How to run Ruby scripts from Go
  37. How to implement Ruby DSL in Go • To implement

    IO operations in Go, it is necessary to get data for Go from an evaluation result of Ruby DSL • I implemented method for Ruby DSL in Go language
  38. How to implement Ruby DSL in Go func Package(mrb *mruby.Mrb,

    self *mruby.MrbValue) (mruby.Value, mruby.Value) { return mruby.Nil, nil } // In another place cDSL.DefineMethod( “package", Package, mruby.ArgsReq(1))
  39. How to implement Ruby DSL in Go func Package(mrb *mruby.Mrb,

    self *mruby.MrbValue) (mruby.Value, mruby.Value) { args := mrb.GetArgs() pkg := PackageResource{Name: args[0].String()} resources = append(resources, pkg) return mruby.Nil, nil } // In another place cDSL.DefineMethod( “package", Package, mruby.ArgsReq(1)) type PackageResource struct { Name string }
  40. How to implement Ruby DSL in Go func Package(mrb *mruby.Mrb,

    self *mruby.MrbValue) (mruby.Value, mruby.Value) { args := mrb.GetArgs() pkg := PackageResource{Name: args[0].String()} if len(args) > 1 { context, _ := mrb.LoadString("Package.new") context.CallBlock("instance_eval", args[1]) attrs, _ := context.Call("attributes") action, _ := mrb.LoadString(":action") value, _ := attrs.Call("[]", action) pkg.Action = value.Array() } resources = append(resources, pkg) return mruby.Nil, nil } // In another place cDSL.DefineMethod( “package", Package, mruby.ArgsReq(1)) type PackageResource struct { Name string Action []string }
  41. It was a little hard to maintain • Writing bridge

    to connect Ruby and Go world is really boring • Checking errors for all Ruby method calls • Required to use 3 languages: Ruby, Go, C (for mrbgem) • You need to prepare environment for cross compiling by yourself
  42. Why did we need Go language? • Purposes • To

    make it as a single binary • To implement IO operations for development environment bootstrap • Can't we achieve them without Go language? • Try mruby-cli!
  43. https://github.com/hone/mruby-cli 2. mruby-cli

  44. Example 2: MItamae • MItamae • Alternative implementation of Itamae

    and Itamae Go, which is actively maintained • Single binary, built with mruby-cli and written in pure Ruby • https://github.com/k0kubun/mitamae
  45. What is mruby-cli? • A cross-compile platform to build CLI

    apps using mruby • https://github.com/hone/mruby-cli • Eric gave a talk "Building maintainable command-line tools with MRuby" this morning • If you've missed it, watch the video later
  46. How to use mruby-cli 1. Download mruby-cli command from https://github.com/hone/mruby-cli/releases

    2. Generate boilerplate by mruby-cli -s app_name 3. cd app_name; docker-compose run compile • You can get binaries in ./mruby/build/*/bin/app_name
  47. How to implement Ruby DSL with mruby-cli • Almost the

    same way as MRI • Required to add some “mrbgems” to call methods which is not provided by default • mruby-eval for instance_eval • mruby-object-ext for instance_exec
  48. What is mrbgem? • It’s like rubygem for MRI •

    Unlike rubygem, it’s compiled together when building mruby • So we can use it without adding dependency to a binary • Some of built-in features in MRI are separated to mrbgem in mruby
  49. Porting existing stdlibs and rubygems to mrbgems • It’s easy

    when they are written in Ruby • Most features required for Ruby programming except keyword arguments are available at existing mrbgems • mrbgems I created to implement MItamae: • mruby-shellwords • mruby-hashie
  50. Implementing IO operations with mruby • There are already IO.popen

    and Kernel.` (`command`) • But I wanted to capture stdout, stderr and status separately • There were no Open3 and Kernel.#spawn • It was necessary to write C to implement Kernel.#spawn
  51. Incomplete implementation of spawn to implement Open3.capture3 static mrb_value mrb_open3_spawn(mrb_state

    *mrb, mrb_value self) { const char **cmd; pid_t pid; mrb_value *argv; mrb_int argc, out_dst, err_dst; mrb_get_args(mrb, "*", &argv, &argc); cmd = mrb_str_buf_to_cstr_buf(mrb, argv, argc-1); open3_spawn_process_options(mrb, argv[argc-1], &out_dst, &err_dst); pid = fork(); if (pid == 0) { dup2(out_dst, STDOUT_FILENO); dup2(err_dst, STDERR_FILENO); execvp(cmd[0], cmd); } return mrb_fixnum_value(pid); }
  52. Finally I could built mruby-open3 that has only Open3.capture3 module

    Open3 # @param [Array<String>] - command to execute # @return [String, String, Process::Status] - stdout, stderr, status def capture3(*cmd) out_r, out_w = IO.pipe err_r, err_w = IO.pipe pid = spawn(*cmd, { 'out' => out_w.to_i, 'err' => err_w.to_i }) # … snip … _, status = Process.waitpid2(pid) [stdout, stderr, status] end module_function :capture3 end
  53. Isn’t it hard to implement apps with mruby-cli? • Yes,

    it’s hard a little when necessary mrbgems are absent • But there are great merits to build apps using mruby-cli
  54. Merits to build CLI apps using mruby-cli • You can

    distribute apps easily • Not depending on users' environment • Cross-compilable
  55. Merits to build CLI apps using mruby-cli • Fast startup

    on command execution $ time itamae version Itamae v1.9.9 itamae version 0.54s user 0.22s system 98% cpu 0.776 total $ time mitamae version MItamae v0.6.0 mitamae version 0.02s user 0.00s system 85% cpu 0.023 total on 1.3 GHz Intel Core m7
  56. Merits to build CLI apps using mruby-cli • Maintainable •

    No bridging code • Happy to write all implementation in Ruby!
  57. Conclusion • To simplify development environment bootstrap, I made a

    configuration management tool configured by Ruby DSL as a single binary using mruby • https://github.com/k0kubun/mitamae • There are 2 ways to build a tool configured by Ruby DSL as a single binary • If there’s a reason to use Go language, use go-mruby • Otherwise mruby-cli is recommended