Leveraging Rust with mruby: Loving our fellow Windows® users - FOSDEM 2017

Even if mruby core works on Windows, any mruby app is may not work if the libraries it depends on don’t compile and run on Windows. Using Rust, we can work towards building an ecosystem that is Windows compatible. This talk walks through building mruby libraries that do that.

Unlike some other languages like JavaScript, Java, or Python, Ruby is not known for it’s ability to run well on Windows. The Windows environment is often treated as a second or third class citizen even though there are many potential users. Part of this is due to Ruby’s strong POSIX oriented standard library. This has mostly been ok since most Ruby applications are hosted and run on UNIX servers.

mruby is a lightweight Ruby that can be linked and embedded inside other applications. Being embeddable means that the runtime host could be Windows. mruby also has no Operating System specific standard library. Since the ecosystem is young, there’s potential for an ecosystem that is Windows friendly. But writing cross platform native extensions in C for mruby is still difficult.

Rust is a modern systems programming language that is fast and built to prevent segfaults and guarantee thread safety. It’s more than just a compiler, it’s a platform that has a great dependency manager and cares about cross compilation out of the box. This makes it a great candidate over C to write native extensions.

In this talk, we’ll go over some basic Rust and how we can integrate this to write mruby libraries that compile on Linux, OS X, and Windows. It’ll dig into not only writing the library code, but also the mruby C API as well as the mruby/Rust binding layer that makes this all possible.

Video: https://fosdem.org/2017/schedule/event/ruby_leverging_rust_to_build_a_windows_friendly_ecosystem_for_mruby/

Transcript

1. Leveraging Rust with mruby Loving our fellow Windows® users + <3
2. Terence Lee @hone02
5. Austin, TX
6. Austin, TX
7. Leveraging Rust with mruby Loving our fellow Windows® users
8. Agenda • Ruby on Windows • Intro to mruby & rust • mruby + rust • Future?
9. Ruby on Windows
10. Heroku CLI
13. Python https://www.python.org/downloads/windows/
14. Installing Ruby on Windows • chruby ◦ Requirements: bash >= 3 or zsh • RVM ◦ https://rvm.io/support/faq#does-rvm-work-on-windows-will-it-in-the-future ◦ Does RVM work on windows? Will it in the future? ▪ NO. If you would like to manage multiple versions of ruby on windows please use pik which is an excellent tool by Gordon Thiesfeld. You can find it on GitHub. There are plans to include windows support in RVM 2.0 => https://www.bountysource.com/fundraisers/489-rvm-2-0. • ruby-build ◦ bash script
15. RubyInstaller
16. Extensions on Windows is Hard
17. Extensions on Windows is Hard Nokogiri
18. Ruby is missing a Windows Ecosystem
19. mruby
20. mruby is the lightweight implementation of the Ruby language complying with part of the ISO standard. mruby can be linked and embedded within your application.
21. Differences with MRI
22. no built in File Socket I/O
23. not threadsafe
24. no thread no fork
25. subset of Ruby 2.1
26. procs blocks DHH freedom patching meta programming literals
27. mrubygems?
28. mrbgem.rake MRuby::Gem::Specification.new('hello') do |spec| spec.license = 'MIT' spec.authors = ['Terence Lee'] spec.summary = 'hello world' spec.bins = ['hello'] spec.add_dependency 'mruby-print', :core => 'mruby-print' spec.add_dependency 'mruby-mtest', :mgem => 'mruby-mtest' spec.add_dependency 'mruby-yaml', :github => 'hone/mruby-yaml' end
29. core mrbgems ├── [4.0K] mruby-array-ext ├── [4.0K] mruby-bin-debugger ├── [4.0K] mruby-bin-mirb ├── [4.0K] mruby-bin-mrbc ├── [4.0K] mruby-bin-mruby ├── [4.0K] mruby-bin-mruby-config ├── [4.0K] mruby-bin-strip ├── [4.0K] mruby-compiler ├── [4.0K] mruby-enumerator
30. mgem list
31. mrbgems & Windows?
32. mruby-socket
34. Still making the same mistakes
36. Rust is a systems programming language that runs blazingly fast, prevents segfaults, and guarantees thread safety.
37. Enabler
38. issues even for seasoned maintainers C Extensions are Hard
39. If you have successfully compiled a Rust program, it will not segfault at runtime.
40. Ownership
41. Zero Cost Abstractions
42. Zero Cost Abstractions pub fn main() { let array = [1,2,3]; let mut sum = 0; for i in 0..3 { sum += array[i] } println!("{:?}", sum); }
43. Zero Cost Abstractions pub fn main() { let array = [1,2,3]; let sum = array.iter().fold(0, |sum, &val| sum + val); println!("{:?}", sum); }
44. Rust 1.0: Stability as a Deliverable (2014) https://blog.rust-lang.org/2014/10/30/Stability.html 6 week release cycle Code compiles on Rust stable 1.0 should compile with Rust stable 1.x
45. Rust Cross-compilation https://blog.rust-lang.org/2016/05/13/rustup.html rustup - a toolchain manager for Rust. "rustup works the same on Windows as it does on Unix" - rustup README Targets: $ rustup target list | wc -l 36 Including: • x86_64-apple-darwin (installed) • x86_64-pc-windows-gnu (installed) • x86_64-unknown-linux-gnu (default) • x86_64-unknown-linux-musl (installed)
46. Shipping Rust in Firefox https://hacks.mozilla.org/2016/07/shipping-rust-in-firefox/ "Starting with Firefox 48, Mozilla is shipping its first production Rust code, with more to come!" -Dave Herman, Director of Strategy at Mozilla Research
47. Cargo
48. Cargo.toml [package] name = "hello_world" version = "0.1.0" authors = ["Your Name <you@example.com>"] [dependencies] time = "0.1.12" regex = "0.1.41"
49. cargo build $ cargo build Updating registry `https://github.com/rust-lang/crates.io-index` Downloading memchr v0.1.5 Downloading libc v0.1.10 Downloading regex-syntax v0.2.1 Downloading memchr v0.1.5 Downloading aho-corasick v0.3.0 Downloading regex v0.1.41 Compiling memchr v0.1.5 Compiling libc v0.1.10 Compiling regex-syntax v0.2.1 Compiling memchr v0.1.5 Compiling aho-corasick v0.3.0 Compiling regex v0.1.41 Compiling hello_world v0.1.0 (file:///path/to/project/hello_world)
50. Cargo.lock [root] name = "hello_world" version = "0.1.0" dependencies = [ "regex 0.1.80 (registry+https://github.com/rust-lang/crates.io-index)", "time 0.1.36 (registry+https://github.com/rust-lang/crates.io-index)", ] [[package]] name = "aho-corasick" version = "0.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" dependencies = [ "memchr 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)", ]
51. mruby + rust
52. Creating a mrbgem with C
53. ├── [4.0K] bintest │ └── [ 437] mruby-hello-world.rb ├── [2.3K] build_config.rb ├── [ 298] mrbgem.rake ├── [4.0K] mrblib │ └── [ 42] mruby-hello-world.rb ├── [1.9K] Rakefile ├── [4.0K] src │ └── [ 363] hello_world_gem.c ├── [4.0K] test │ └── [ 119] test_mruby-hello-world.rb └── [4.0K] tools └── [4.0K] mruby-hello-world └── [ 755] mruby-hello-world.c
54. ├── [4.0K] bintest │ └── [ 437] mruby-hello-world.rb ├── [2.3K] build_config.rb ├── [ 298] mrbgem.rake ├── [4.0K] mrblib │ └── [ 42] mruby-hello-world.rb ├── [1.9K] Rakefile ├── [4.0K] src │ └── [ 363] hello_world_gem.c ├── [4.0K] test │ └── [ 119] test_mruby-hello-world.rb └── [4.0K] tools └── [4.0K] mruby-hello-world └── [ 755] mruby-hello-world.c
55. hello_world_gem.c #include "mruby.h" mrb_value c_hello(mrb_state * mrb, mrb_value self) { return mrb_str_new_cstr(mrb, "Hello"); } void mrb_mruby_hello_world_gem_init(mrb_state * mrb) { struct RClass *klass = mrb_define_module(mrb, "Hello"); mrb_define_class_method(mrb, klass, "hello", c_hello, MRB_ARGS_NONE()); } void mrb_mruby_hello_world_gem_final(mrb_state * mrb) { }
56. hello_world_gem.c #include "mruby.h" mrb_value c_hello(mrb_state * mrb, mrb_value self) { return mrb_str_new_cstr(mrb, "Hello"); } void mrb_mruby_hello_world_gem_init(mrb_state * mrb) { struct RClass *klass = mrb_define_module(mrb, "Hello"); mrb_define_class_method(mrb, klass, "hello", c_hello, MRB_ARGS_NONE()); } void mrb_mruby_hello_world_gem_final(mrb_state * mrb) { }
57. hello_world_gem.c #include "mruby.h" mrb_value c_hello(mrb_state * mrb, mrb_value self) { return mrb_str_new_cstr(mrb, "Hello"); } void mrb_mruby_hello_world_gem_init(mrb_state * mrb) { struct RClass *klass = mrb_define_module(mrb, "Hello"); mrb_define_class_method(mrb, klass, "hello", c_hello, MRB_ARGS_NONE()); } void mrb_mruby_hello_world_gem_final(mrb_state * mrb) { }
58. hello_world_gem.c #include "mruby.h" mrb_value c_hello(mrb_state * mrb, mrb_value self) { return mrb_str_new_cstr(mrb, "Hello"); } void mrb_mruby_hello_world_gem_init(mrb_state * mrb) { struct RClass *klass = mrb_define_module(mrb, "Hello"); mrb_define_class_method(mrb, klass, "hello", c_hello, MRB_ARGS_NONE()); } void mrb_mruby_hello_world_gem_final(mrb_state * mrb) { }
59. ├── [4.0K] bintest │ └── [ 437] mruby-hello-world.rb ├── [2.3K] build_config.rb ├── [ 298] mrbgem.rake ├── [4.0K] mrblib │ └── [ 42] mruby-hello-world.rb ├── [1.9K] Rakefile ├── [4.0K] src │ └── [ 363] hello_world_gem.c ├── [4.0K] test │ └── [ 119] test_mruby-hello-world.rb └── [4.0K] tools └── [4.0K] mruby-hello-world └── [ 755] mruby-hello-world.c
60. mruby-hello-world.rb def __main__(argv) puts Hello.hello end
61. Rust FFI https://blog.rust-lang.org/2015/04/24/Rust-Once-Run-Everywhere.html > Rust makes it easy to communicate with C APIs without overhead, and to leverage its ownership system to provide much stronger safety guarantees for those APIs at the same time. #[no_mangle] pub extern "C" fn double_input(input: i32) -> i32 { input * 2 }
62. ├── [4.0K] bintest │ └── [ 437] mruby-rust-hello-world.rb ├── [2.3K] build_config.rb ├── [2.4K] mrbgem.rake ├── [4.0K] mrblib │ ├── [4.0K] mruby-rust-hello-world │ └── [ 41] mruby-rust-hello-world.rb ├── [1.9K] Rakefile ├── [4.0K] rust │ ├── [ 460] Cargo.lock │ ├── [ 196] Cargo.toml │ └── [4.0K] src │ └── [ 710] lib.rs ├── [4.0K] test │ └── [ 119] test_mruby-rust.rb └── [4.0K] tools └── [4.0K] mruby-rust-hello-world └── [ 759] mruby-rust-hello-world.c
63. lib.rs #[no_mangle] pub extern "C" fn mrb_rust_hello(mrb: *mut mrb_state, selfie: mrb_value) -> mrb_value { unsafe { mrb_str_new_cstr(mrb, cstr!("Hello")) } } #[no_mangle] pub extern "C" fn mrb_mruby_rust_hello_world_gem_init(mrb: *mut mrb_state) { unsafe { let klass = mrb_define_module(mrb, cstr!("Rust")); mrb_define_class_method(mrb, klass, cstr!("hello"), mrb_rust_hello as mrb_func_t, MRB_ARGS_NONE()); } } #[no_mangle] pub extern "C" fn mrb_mruby_rust_hello_world_gem_final(mrb: *mut mrb_state) { }
64. Cargo.toml [package] name = "hello" version = "0.1.0" authors = ["Terence Lee <hone02@gmail.com>"] [dependencies] mferuby = { git = "https://github.com/hone/mferuby.git" } [lib] crate-type = ["staticlib"]
65. mferuby Rust bindings for the mruby C API
66. libmruby-sys crate ├── [4.0K] crates │ ├── [4.0K] libmruby-sys │ │ ├── [ 150] Cargo.toml │ │ └── [4.0K] src │ │ └── [5.1K] lib.rs │ └── [4.0K] mferuby │ ├── [ 203] Cargo.toml │ └── [4.0K] src │ ├── [ 588] lib.rs │ └── [ 815] mrb.rs low level rust binding to mruby C API
67. crates/libmruby-sys/src/lib.rs extern { #[link_name = "tmrb_nil_value"] pub fn nil() -> mrb_value; #[link_name = "tmrb_true_value"] pub fn mrb_true() -> mrb_value; #[link_name = "tmrb_false_value"] pub fn mrb_false() -> mrb_value; #[link_name = "tmrb_fixnum_value"] pub fn fixnum(i: c_int) -> mrb_value; #[link_name = "tmrb_float_value"] pub fn float(mrb: *mut mrb_state, f: c_double) -> mrb_value; pub fn mrb_open() -> *mut mrb_state; pub fn mrb_define_module(mrb: *mut mrb_state, name: *const c_char) -> *mut RClass; pub fn mrb_define_class(mrb: *mut mrb_state, name: *const c_char, super_class: *mut RClass) -> *mut RClass; pub fn mrb_define_class_under(mrb: *mut mrb_state, outer: *mut RClass, name: *const c_char, super_class: *mut RClass) -> *mut
68. mferuby crate ├── [4.0K] crates │ ├── [4.0K] libmruby-sys │ │ ├── [ 150] Cargo.toml │ │ └── [4.0K] src │ │ └── [5.1K] lib.rs │ └── [4.0K] mferuby │ ├── [ 203] Cargo.toml │ └── [4.0K] src │ ├── [ 588] lib.rs │ └── [ 815] mrb.rs high level rust API on top of libmruby-sys
69. crates/mferuby/src/lib.rs pub fn mruby_str_to_rust_string(mruby_string: sys::mrb_value) -> Result<String, std::str::Utf8Error> { let size = unsafe { sys::RSTRING_LEN(mruby_string) }; let ptr = unsafe { sys::RSTRING_PTR(mruby_string) }; let slice = unsafe { std::slice::from_raw_parts(ptr as *const u8, size as usize) }; let s = try!(std::str::from_utf8(slice)); Ok(s.to_string()) }
70. mferuby-runtime mrbgem ├── [2.3K] build_config.rb ├── [ 412] mrbgem.rake ├── [1.9K] Rakefile ├── [4.0K] src │ ├── [1.4K] mferuby_runtime.c │ ├── [ 138] mferuby_runtime_gem.c │ └── [ 37] mruby.rs └── [4.0K] test └── [ 131] test_mferuby-runtime.rb compiles against mruby and exposes non-public C API to rust
71. src/mferuby_runtime.c #include <mruby/compile.h> #include <mruby/value.h> mrb_value tmrb_float_value(struct mrb_state *mrb, mrb_float f) { return mrb_float_value(mrb, f); } mrb_value tmrb_fixnum_value(mrb_int i) { return mrb_fixnum_value(i); } mrb_value tmrb_nil_value() { return mrb_nil_value(); } mrb_value tmrb_true_value() { return mrb_true_value(); }
72. mruby-docopt
73. cargo help
74. CPP, still Hard
75. class CommandLine USAGE = <<USAGE Heroku Docker Ruby Util. Usage: ruby-util detect-ruby <gemfile-path> ruby-util detect-gem <profiled-path> <gem> ruby-util install-ruby <gemfile-path> <install-path> <profiled-path> Options: -h --help Show this message USAGE def run(argv) options = Docopt.parse(USAGE, argv) if options["detect-ruby"] puts CLI::DetectRuby.new(File.read(options["<gemfile-path>"])).ruby_ver sion
76. mruby-docopt pub extern "C" fn parse(mrb: *mut sys::mrb_state, this: sys::mrb_value) -> sys::mrb_value { let mrb_obj = mferuby::Mrb::new(mrb); let mut usage: sys::mrb_value = unsafe { mem::uninitialized() }; let mut argv: sys::mrb_value = unsafe { mem::uninitialized() }; unsafe { sys::mrb_get_args(mrb, cstr!("SA"), &mut usage, &mut argv); } let rust_usage = mferuby::mruby_str_to_rust_string(usage).unwrap(); let argc = unsafe { sys::RARRAY_LEN(argv) }; let mut vec_args: Vec<String> = vec![]; for i in 0..argc { let element = unsafe { sys::mrb_ary_ref(mrb, argv, i) }; vec_args.push(mferuby::mruby_str_to_rust_string(element).unwrap()) ;
77. mruby-docopt pub extern "C" fn parse(mrb: *mut sys::mrb_state, this: sys::mrb_value) -> sys::mrb_value { let mrb_obj = mferuby::Mrb::new(mrb); let mut usage: sys::mrb_value = unsafe { mem::uninitialized() }; let mut argv: sys::mrb_value = unsafe { mem::uninitialized() }; unsafe { sys::mrb_get_args(mrb, cstr!("SA"), &mut usage, &mut argv); } let rust_usage = mferuby::mruby_str_to_rust_string(usage).unwrap(); let argc = unsafe { sys::RARRAY_LEN(argv) }; let mut vec_args: Vec<String> = vec![]; for i in 0..argc { let element = unsafe { sys::mrb_ary_ref(mrb, argv, i) }; vec_args.push(mferuby::mruby_str_to_rust_string(element).unwrap()) ;
78. mruby-docopt pub extern "C" fn parse(mrb: *mut sys::mrb_state, this: sys::mrb_value) -> sys::mrb_value { let mrb_obj = mferuby::Mrb::new(mrb); let mut usage: sys::mrb_value = unsafe { mem::uninitialized() }; let mut argv: sys::mrb_value = unsafe { mem::uninitialized() }; unsafe { sys::mrb_get_args(mrb, cstr!("SA"), &mut usage, &mut argv); } let rust_usage = mferuby::mruby_str_to_rust_string(usage).unwrap(); let argc = unsafe { sys::RARRAY_LEN(argv) }; let mut vec_args: Vec<String> = vec![]; for i in 0..argc { let element = unsafe { sys::mrb_ary_ref(mrb, argv, i) }; vec_args.push(mferuby::mruby_str_to_rust_string(element).unwrap()) ;
79. let result = docopt::Docopt::new(rust_usage) .and_then(|d| d.help(false).argv(vec_args.into_iter()).parse()); match result { Ok(args) => { let args = Box::new(args); let klass = unsafe { sys::mrb_class_get_under(mrb, sys::mrb_module_get(mrb, cstr!("Docopt")), cstr!("Options")) }; unsafe { sys::mrb_obj_value(mrb_obj.data_object_alloc::<docopt::ArgvMap>(kl ass, args, &docopt_option_type)) } }, Err(e) => unsafe { println!("ERROR: {:?}", e); sys::nil() }, } }
80. let result = docopt::Docopt::new(rust_usage) .and_then(|d| d.help(false).argv(vec_args.into_iter()).parse()); match result { Ok(args) => { let args = Box::new(args); let klass = unsafe { sys::mrb_class_get_under(mrb, sys::mrb_module_get(mrb, cstr!("Docopt")), cstr!("Options")) }; unsafe { sys::mrb_obj_value(mrb_obj.data_object_alloc::<docopt::ArgvMap>(kl ass, args, &docopt_option_type)) } }, Err(e) => unsafe { println!("ERROR: {:?}", e); sys::nil() }, } }
81. let result = docopt::Docopt::new(rust_usage) .and_then(|d| d.help(false).argv(vec_args.into_iter()).parse()); match result { Ok(args) => { let args = Box::new(args); let klass = unsafe { sys::mrb_class_get_under(mrb, sys::mrb_module_get(mrb, cstr!("Docopt")), cstr!("Options")) }; unsafe { sys::mrb_obj_value(mrb_obj.data_object_alloc::<docopt::ArgvMap>(kl ass, args, &docopt_option_type)) } }, Err(e) => unsafe { println!("ERROR: {:?}", e); sys::nil() }, } }
82. class CommandLine USAGE = <<USAGE Heroku Docker Ruby Util. Usage: ruby-util detect-ruby <gemfile-path> ruby-util detect-gem <profiled-path> <gem> ruby-util install-ruby <gemfile-path> <install-path> <profiled-path> Options: -h --help Show this message USAGE def run(argv) options = Docopt.parse(USAGE, argv) if options["detect-ruby"] puts CLI::DetectRuby.new(File.read(options["<gemfile-path>"])).ruby_ver sion
83. mruby-docopt in the wild --- a/mrbgem.rake +++ b/mrbgem.rake @@ -12,6 +12,7 @@ MRuby::Gem::Specification.new('heroku-docker-ruby-util') do |spec| spec.add_dependency 'mruby-stringio', github: 'ksss/mruby-stringio' spec.add_dependency 'mruby-process', github: 'hone/mruby-process', branch: 'header' spec.add_dependency 'mruby-io', github: 'hone/mruby-io', branch: 'popen_status' + spec.add_dependency 'mruby-docopt', github: 'hone/mruby-docopt', branch: 'rust' # test deps spec.add_dependency 'mruby-time', core: 'mruby-time'
84. What's Next?
85. mferuby • Cover more mruby C API with Rust binding • Create higher level abstractions • Using mferuby shouldn't require "unsafe"
86. mrbgems around low level rust crates
87. tokio.rs
88. File I/O
89. Integrate with Helix
90. Ruby hasn't had a good Windows history
91. mruby doesn't have to repeat this
92. mruby+rust can create a Windows friendly ecosystem
93. Let's make ruby better for everyone
94. Thank You @hone02 https://github.com/hone/mferuby https://github.com/hone/mruby-docopt