Road to Go Gem #rubykaigi

https://rubykaigi.org/2025/presentations/sue445.html#day3

Transcript

1. Road to Go gem RubyKaigi 2025 / #rubykaigi #rubykaigiC pixiv Inc. sue445 2025.04.18
2. 2 Hello!
3. 3 My name is Go The Go gopher was designed by Renee French. (http://reneefrench.blogspot.com/) The design is licensed under the Creative Commons 4.0 Attributions license. Read this article for more details: https://blog.golang.org/gopher
4. 4 About sue445 • Go Sueyoshi (a.k.a. sue445, sue-san) • https://x.com/sue445 • https://github.com/sue445 • Gopher since 1982 • Shibuya.rb • Tokyu.rb sue445
5. 5 Day job at pixiv.Inc • Infrastructure Unit • AWS and Google Cloud ◦ Organization Owner ◦ Solution Architect • Certifications possessed ◦ AWS Certified Solutions Architect – Professional (SAP-C01) ◦ Google Cloud Professional Cloud Architect sue445
6. 6 Day job at pixiv.Inc • on-premises Kubernetes • Maintainer of In-house tools (GitLab, Sentry etc) • etc etc etc!!!!!!!!!! sue445
7. 7 CM. About pixiv Inc. https://www.pixiv.co.jp/service/ 10years+ 10years+ 10years+ NEW!
8. https://conference.pixiv.co.jp/2025/rubymusicmixin 8 CM. RubyMusicMixin 2025
9. 9 CM. RubyMusicMixin 2025 RubyMusicMixin 2025 We are here
10. 10 Riverside (川) #rubyriver
11. • All about native extension with Go • Tips for creating native extensions in a new language 11 Goal in this talk
12. • Let's recall RubyKaigi 2015 • Getting started with Go gem • About go-gem-wrapper • Road to go-gem-wrapper • Use “ruby.h” in go building • Add --ext=go to bundle gem command • Pros/Cons of Go gem • Real use case 12 Agenda
13. • Using https://github.com/ruby-go-gem/go-gem-wrapper makes it very easy to make native extension with Go • We can write Ruby gem in Go in areas where Ruby is not good at (Go is very good at) • goroutine is very good! 13 Conclusion
14. https://rubykaigi.org/2015/presentations/mmasaki/ 14 Let’s recall RubyKaigi 2015
15. • cgo (go build -buildmode=c-shared) was introduced at Go 1.5 • cgo creates shared object file (*.so) from Go • Use this technology to create a native Ruby extension with Go ◦ a.k.a. Go gem 15 tl;dr;
16. 1. The code from 10 years ago (Go 1.5) doesn't work now (Go 1.24) ◦ a later mention 2. To make a gem in Go, you need to copy and paste a lot of patches. ◦ e.g. ▪ ext/gem_name/extconf.rb ▪ ext/gem_name/my_gem.go ▪ ext/gem_name/go.mod ▪ bindings ▪ etc… 16 Past issues
17. https://github.com/mmasaki/memberlist/blob/master/ext/memberlist/wrapper.go 17 e.g. 10 years ago Go gem
18. • Technically possible. But it takes a lot of work. 18 Past issues
19. • Library of patches that needed to be done manually • There was no standard package manager for Go at 10 years ago • But today, we have Go module 19 github.com/ruby-go-gem/go-gem-wrapper
20. 1. bundle gem <gem_name> –ext=c 2. Run “patch_for_go_gem.rb” 3. go get -u github.com/ruby-go-gem/go-gem-wrapper@latest 4. Write gem using Go! 20 Getting started with Go gem
21. $ bundle gem example --ext=c Creating gem 'example'... create example/Gemfile create example/lib/example.rb create example/lib/example/version.rb (snip) create example/ext/example/extconf.rb create example/ext/example/example.h create example/ext/example/example.c Gem 'gem_name' was successfully created. For more information on making a RubyGem visit https://bundler.io/guides/creating_gem.html bundle gem <gem_name> –ext=c 21
22. • https://github.com/ruby-go-gem/go-gem-wrapper/tree/main/_tools/patch_ for_go_gem • Patch to make a gem into a Go gem right after bundle gem • This is a temporary tool until bundle gem --ext=go is added to the bundler ◦ c.f. https://github.com/rubygems/rubygems/pull/8183 (a later mention) 22 Run “patch_for_go_gem.rb”
23. $ ruby patch_for_go_gem.rb --file example/example.gemspec [INFO] /path/to/example/ext/example/example.go is created [INFO] /path/to/example/ext/example/go.mod is created [INFO] /path/to/example/ext/example/example.c is updated [INFO] /path/to/example/ext/example/extconf.rb is updated [INFO] example/example.gemspec is updated Run “patch_for_go_gem.rb” 23
24. package main /* #include "example.h" */ import "C" import ( "github.com/ruby-go-gem/go-gem-wrapper/ruby" ) //export Init_example func Init_example() { rb_mExample := ruby.RbDefineModule("Example") } func main() { } ext/<gem_name>/＜gem_name>.go 24
25. #include "my_gem.h" #include "_cgo_export.h" // Added ext/<gem_name>/<gem_name>.c 25
26. require "mkmf" require "go_gem/mkmf" # Added append_cflags("-fvisibility=hidden") # create_makefile("example/example") create_go_makefile("example/example") # Added ext/<gem_name>/extconf.rb 26
27. Gem::Specification.new do |spec| spec.add_dependency "go_gem" # Added end <gem_name>.gemspec 27
28. module Example def self.sum(a, b) a + b end end e.g. Impl Example#sum 28
29. //export rb_example_sum func rb_example_sum(_ C.VALUE, a C.VALUE, b C.VALUE) C.VALUE { aLong := ruby.NUM2LONG(ruby.VALUE(a)) bLong := ruby.NUM2LONG(ruby.VALUE(b)) sum := aLong + bLong return C.VALUE(ruby.LONG2NUM(sum)) } ext/<gem_name>/＜gem_name>.go 29
30. package main /* #include "example.h" // Append this VALUE rb_example_sum(VALUE self, VALUE a, VALUE b); */ import "C" ext/<gem_name>/＜gem_name>.go 30
31. //export Init_example func Init_example() { rb_mExample := ruby.RbDefineModule("Example") // Append this ruby.RbDefineSingletonMethod(rb_mExample, "sum", C.rb_example_sum, 2) } ext/<gem_name>/＜gem_name>.go 31
32. • We can write a gem in Go just as you write a gem in C! 32 Easy, right?
33. • Go module: github.com/ruby-go-gem/go-gem-wrapper • Ruby gem: https://rubygems.org/gems/go_gem • Requiremets ◦ Go 1.23+ ◦ Ruby 3.3+ ◦ glibc (a later mention) 33 About go-gem-wrapper
34. github.com/ruby-go-gem/go-gem-wrapper is Go module for Go gem 34 About github.com/ruby-go-gem/go-gem-wrapper // ext/example/go.mod module github.com/sue445/example go 1.24 require ( github.com/ruby-go-gem/go-gem-wrapper v0.7.1 )
35. https://github.com/ruby-go-gem/go-gem-wrapper/blob/v0.7.1/ruby/function_ruby_3_3_generated.go#L2687-L2697 35 e.g. RbDefineSingletonMethod func RbDefineSingletonMethod(obj VALUE, mid string, arg3 unsafe.Pointer, arity int) { char, clean := string2Char(mid) defer clean() C.rb_define_singleton_method(C.VALUE(obj), char, toCFunctionPointer(arg3), C.int(arity)) }
36. 36 There are 1,100 functions in “ruby.h” $ ls ruby/*_generated.go | xargs wc -l 59 ruby/enum_ruby_3_3_generated.go 57 ruby/enum_ruby_3_4_generated.go 9264 ruby/function_ruby_3_3_generated.go 9198 ruby/function_ruby_3_4_generated.go 105 ruby/type_ruby_3_3_generated.go 103 ruby/type_ruby_3_4_generated.go 18786 total https://github.com/ruby-go-gem/go-gem-wrapper/tree/v0.7.1/ruby
37. • Helpers for compiling Go extensions for ruby ◦ https://rubygems.org/gems/go_gem ◦ https://github.com/ruby-go-gem/go-gem-wrapper/tree/v0.7.1/_gem • Similar to https://rubygems.org/gems/rb_sys in Rust gem 37 About go_gem
38. require "mkmf" require "go_gem/mkmf" create_go_makefile("example/example") # Pass debug flags to `go build` create_go_makefile("example/example", go_build_args: "-gcflags='all=-N -l'") create_go_makefile Add a task and hacks to build Go against the “Makefile” generated by create_makefile. (similar to create_rust_makeflile) 38
39. 39 create_go_makefile def create_go_makefile(target, srcprefix: nil, go_build_args: nil) # (snip) File.open("Makefile", "a") do |f| f.write <<~MAKEFILE.gsub(/^ {8}/, "\t") $(DLLIB): Makefile $(srcdir)/*.go cd $(srcdir); \ CGO_CFLAGS='$(INCFLAGS)' CGO_LDFLAGS='#{ldflags}' GOFLAGS='#{goflags}' \ go build -p 4 -buildmode=c-shared -o #{current_dir}/$(DLLIB) #{go_build_args} MAKEFILE end end https://github.com/ruby-go-gem/go-gem-wrapper/blob/v0.7.1/_gem/lib/go_gem/mkmf.rb
40. # Rakefile require "go_gem/rake_task" GoGem::RakeTask.new("example") GoGem::RakeTask Provides rake tasks for go test with CRuby. (a later mention) 40
41. • I had to go through many hardships to create go-gem-wrapper • I will now introduce them 41 Road to go-gem-wrapper
42. 1. cgo compatibility 2. Generate Go’s bindings from “ruby.h” ◦ Type mismatch between outside and inside of module ◦ C pointer ◦ deprecated functions ◦ Removed functions ◦ Supports https://pkg.go.dev/ 3. Go with CRuby 42 Road to go-gem-wrapper
43. • This technique was introduced in RubyKaigi 2015 • No problem at Go 1.5, but SEGV at Go 1.24 43 1. cgo compatibility https://www.slideshare.net/slideshow/ruby-meets-go/56074507
44. • Current Go's string isn't a null-terminated string, so using *(*[]byte)(unsafe.Pointer(&str)) will return a non-null-terminated byte array and can't be *C.char • Go 1.7+ are spec'd this way 44 Why?
45. 45 e.g. Print Go string via C package main /* #include <stdio.h> */ import "C" func Print(str string) { cstr := GOSTRING_PTR(str) C.fputs(cstr, (*C.FILE)(C.stdout)) } func main() { Print("ABCD") }
46. 46 OMG! $ go run main.go ABCDtrueallgallprootitabsbrkidledeadsync is LEAFbase of ) = <==GOGC] = pc=+Inf-Inf: p=cas1cas2cas3cas4cas5cas6 at m= sp= sp: lr: fp= gp= mp=) m=boolint8uintchanfuncopenstatfile in sha1sha2timeJuneJuly/etcfalsedefersweeptestRtestWexecWhchanexecRschedsudogtimergscanm heaptracepanicsleeparm64 cnt=gcing MB, got= ... max=scav ptr ] = (init ms, fault and tab= top=[...], fp:int16int32int64uint8arrayslicewriteclosegetwdlstatGreekpmullcrc32cpuidfcntlMarchAp rilLocalerrno stringsysmontimersefenceselect, not GOROOTobject next= jobs= goid sweep B -> % util alloc free span= prev= list=, i = code= addr=], sp= m->p= p->m=SCHED curg= ctxt: min= max= bad ts(...)
47. 47 Solution func string2Char(str string) (*C.char, func()) { cstr := C.CString(str) clean := func() { C.free(unsafe.Pointer(cstr)) } return cstr, clean } https://github.com/ruby-go-gem/go-gem-wrapper/blob/v0.7.1/ruby/wrapper.go#L26-L48
48. 48 Solution func RbDefineSingletonMethod(obj VALUE, mid string, arg3 unsafe.Pointer, arity int) { char, clean := string2Char(mid) defer clean() C.rb_define_singleton_method(C.VALUE(obj), char, toCFunctionPointer(arg3), C.int(arity)) } https://github.com/ruby-go-gem/go-gem-wrapper/blob/v0.7.1/ruby/function_ruby_3_3_ge nerated.go#L2687-L2697
49. • Go, similar to Ruby, is a very backward-compatible language. • However, cgo has characteristics that depend on Go's internal implementation and can be broken by Go version upgrades • c.f. “Why Go's old code almost never stops working“ ◦ https://zenn.dev/catatsuy/articles/fda1e42acad421 (ja) 49 Tips
50. 50 2. Generate Go’s bindings from “ruby.h”
51. • In this talk, binding is a Go function that corresponds to a CRuby function as follows 51 2. Generate Go’s bindings from “ruby.h” func RbDefineSingletonMethod(obj VALUE, mid string, arg3 unsafe.Pointer, arity int) { char, clean := string2Char(mid) defer clean() C.rb_define_singleton_method(C.VALUE(obj), char, toCFunctionPointer(arg3), C.int(arity)) }
52. • Just on https://docs.ruby-lang.org/ja/latest/function/index.html , there are less than 200 • There are about 2,000+ C functions defined in ruby.h • About 1,100 functions can be included and actually used • At first (about 30 functions), I wrote them by hand, but I decided to generate them automatically from ruby.h because it's just too hard. 52 2. Generate Go’s bindings from “ruby.h”
53. 1. Use existing OSS 2. Create it myself 53 Choices
54. Automatic C-Go Bindings Generator for Go Programming Language 54 https://github.com/xlab/c-for-go
55. Fixed an error when passing “ruby.h” 55 https://github.com/xlab/c-for-go/pull/171
56. • Bindings generated by c-for-go was abandoned due to lack of portability ◦ e.g. Bindings generated on MacOS isn’t usable on Ubuntu 56 Result
57. • It may be due to the fact that there are many #ifdef branches in “ruby.h” for each OS and build environment. • e.g. 57 Result https://github.com/ruby/ruby/blob/v3_4_2/internal/process.h#L14-L20 #ifdef HAVE_SYS_TYPES_H # include <sys/types.h> /* for mode_t */ #endif #ifdef _WIN32 # include "ruby/win32.h" /* for mode_t */ #endif
58. • I can change the source code for each environment we build by creating a file similar to “ruby_linux_amd64.go” or “ruby_darwin_arm64.go” (Build constraints) • But this increases the amount of code, and CI also needs to prepare all build constraints • I gave up on it because it would be too hard maintain ◦ https://github.com/ruby-go-gem/go-gem-wrapper/pull/117#issuecomm ent-2336583098 58 Result
59. • However, I couldn't figure out how to create it… 59 Create it myself
60. 60 Hint: @joker1007 ’s Rust gem talk https://speakerdeck.com/joker1007/rustdezuo-rutree-sitterpasanorubybaindeingu?slide=16
61. • In a nutshell, I implemented what Rust gem does in Go gem • I decided to generate Go source code automatically from ruby.h 61 tl;dr;
62. • https://github.com/rust-lang/rust-bindgen ◦ Auto-generate C/C++ code from Rust FFI bindings • https://github.com/oxidize-rb/rb-sys ◦ Auto-generate Rust code from C API using bindgen ◦ https://rubygems.org/gems/rb_sys : Helpers for compiling Rust extensions for ruby • https://github.com/matsadler/magnus ◦ High level Ruby bindings for Rust ◦ depends on rb-sys 62 Architecture of Rust gem
63. • rust-bindgen (Generate bindings) + rb-sys (Low layer API) = go-gem-wrapper • go-gem-wrapper doesn’t include high layer API similar to magnus (maybe…) • I created go-gem-wrapper by myself 63 What is go-gem-wrapper
64. 64 Generate Go code from “ruby.h”
65. 65 Generate Go code from “ruby.h” ruby.h PORO Go Parser Generator
66. • Parser for “ruby.h” • Wrapper for https://github.com/universal-ctags/ctags 66 https://github.com/ruby-go-gem/ruby_header_parser
67. • https://github.com/ruby-go-gem/go-gem-wrapper/tree/v0.7.1/_tools/ruby _h_to_go • Generate Go code from “ruby.h” using ruby_header_parser 67 ruby_h_to_go
68. • Issue 1. Type mismatch between outside and inside of module • Issue 2. C pointer • Issue 3. deprecated functions • Issue 4. Removed functions • Issue 5. Supports https://pkg.go.dev/ 68 Many many issues…
69. 69 Issue 1. Type mismatch between outside and inside of module
70. package main // snip import ( "github.com/ruby-go-gem/go-gem-wrapper" ) //export rb_dummy_sum func rb_dummy_sum(_ C.VALUE, a C.VALUE, b C.VALUE) C.VALUE { aLong := ruby.RbNum2Long(a) bLong := ruby.RbNum2Long(b) sum := aLong + bLong return ruby.RbLong2NumInline(sum) } PoC (1/2) 70
71. // in github.com/ruby-go-gem/go-gem-wrapper module package ruby /* #include "ruby.h" */ import "C" func RbNum2Long(n C.VALUE) C.long { return C.rb_num2long(n) } PoC (2/2) 71
72. ./dummy.go:16:35: cannot use a (variable of type _Ctype_ulong) as ruby._Ctype_ulong value in argument to ruby.RbNum2Long ./dummy.go:17:35: cannot use b (variable of type _Ctype_ulong) as ruby._Ctype_ulong value in argument to ruby.RbNum2Long ./dummy.go:21:39: cannot use sum (variable of type _Ctype_ulong) as ruby.Long value in argument to ruby.RbLong2NumInline make: *** [dummy.bundle] Error 1 Build error!!!!! 72
73. https://pkg.go.dev/cmd/cgo says > Cgo translates C types into equivalent unexported Go types. Because the translations are unexported, a Go package should not expose C types in its exported API: a C type used in one Go package is different from the same C type used in another. ref. https://github.com/golang/go/issues/13467 73 Answer. cgo specification
74. package ruby /* #include "ruby.h" */ import "C" type Long C.long type VALUE C.VALUE func RbNum2long(n VALUE) Long { return Long(C.rb_num2long(C.VALUE(n))) } Workarround 74
75. 75 Issue 2. C pointer
76. • C lang's pointers are very difficult for programming beginners • C lang's pointers are very difficult for C lang's parser and Go generator too! 76 Issue 2. C pointer
77. • Please answer Go type that you would expect given C type • e.g. C type: int -> Go type: int 77 Quiz. Convert C type to Go type
78. 78 Q1. C type char* -> Go type ???
79. • A. Case by case ◦ string (in many cases) ◦ char* (when RSTRING_PTR and RSTRING_END returns) 79 Q1. C type char* -> Go type ??? https://github.com/ruby/ruby/blob/v3_4_2/include/ruby/internal/core/rstring.h#L408-L416 /** * Queries the contents pointer of the string. * * @param[in] str String in question. * @return Pointer to its contents. * @pre `str` must be an instance of ::RString. */ static inline char * RSTRING_PTR(VALUE str)
80. 80 Q2. C type VALUE* -> Go type ???
81. • A. Case by case ◦ Array ◦ Pointer for input/output ◦ Pointer for output only ◦ Pointer for input only 81 Q2. C type VALUE* -> Go type ???
82. 82 e.g. Use VALUE* as Array // RbFuncallv calls `rb_funcallv` in C // // Original definition is following // // VALUE rb_funcallv(VALUE recv, ID mid, int argc, const VALUE *argv) func RbFuncallv(recv VALUE, mid ID, argc int, argv []VALUE) VALUE { return VALUE(C.rb_funcallv(C.VALUE(recv), C.ID(mid), C.int(argc), toCArray[VALUE, C.VALUE](argv))) } https://github.com/ruby-go-gem/go-gem-wrapper/blob/v0.7.3/ruby/function_ruby_3_4_generated.go#L3885-L3892
83. 83 e.g. Use VALUE* as in/out or out only pointer // RbCvarFind calls `rb_cvar_find` in C // // Original definition is following // // VALUE rb_cvar_find(VALUE klass, ID name, VALUE *front) func RbCvarFind(klass VALUE, name ID, front *VALUE) VALUE { var cFront C.VALUE ret := VALUE(C.rb_cvar_find(C.VALUE(klass), C.ID(name), &cFront)) *front = VALUE(cFront) return ret } https://github.com/ruby-go-gem/go-gem-wrapper/blob/v0.7.3/ruby/function_ruby_3_4_generated.go#L2278-L2288
84. 84 e.g. Use VALUE* as input only pointer // RbDefineVariable calls `rb_define_variable` in C // // Original definition is following // // void rb_define_variable(const char *name, VALUE *var) func RbDefineVariable(name string, v *VALUE) { char, clean := string2Char(name) defer clean() C.rb_define_variable(char, (*C.VALUE)(v)) } https://github.com/ruby-go-gem/go-gem-wrapper/blob/v0.7.3/ruby/function_ruby_3_4_generated.go#L2688-L2698
85. 85 Q3. C type xxx** -> Go type ???
86. • A. Case by case ◦ Multiple pointer ◦ Array of pointer ◦ Array of String 86 Q3. C type xxx** -> Go type ???
87. • Normal pointer • Input only pointer • Special one for multiple pointer • Array • Array of pointer • Array of string • Function pointer 87 Pointer list in “ruby.h”
88. • C type to Go type not uniquely defined • Case by case!!!!! 88 Problem. Case by case
89. • All were handled individually • About 37 / 1,100 functions 89 Solution
90. 90 ruby_header_parser/config/default.yml.erb function: pointer_hint: RSTRING_END: self: raw RSTRING_PTR: self: raw rb_data_object_make: 4: sref rb_data_typed_object_make: 3: sref https://github.com/ruby-go-gem/ruby_header_parser/blob/main/config/default.yml.erb#L83-L95
91. • Configuration file specification ◦ https://github.com/ruby-go-gem/ruby_header_parser/blob/v0.4.2/CON FIG.md • Hint ◦ https://github.com/xlab/c-for-go/wiki/Translator-config-section 91 ruby_header_parser/config/default.yml.erb
92. • Some functions in “ruby.h” are deprecated • Deprecated function bindings should not be generated 92 Issue 3. deprecated functions $ bundle exec rake build_all /Users/sue445/.rbenv/versions/3.4.1/include/ruby-3.4.0/ruby/internal/core/rdata.h:314 :1: note: 'rb_data_object_get_warning' has been explicitly marked deprecated here cgo-gcc-prolog:4427:11: warning: 'rb_data_object_wrap_warning' is deprecated: by TypedData [-Wdeprecated-declarations]
93. 93 Solution. ruby_header_parser/config/default.yml.erb function: exclude_name: # deprecated functions - !ruby/regexp /^rb_check_safe_str$/i - !ruby/regexp /^rb_clear_constant_cache$/i - !ruby/regexp /^rb_clone_setup$/i - !ruby/regexp /^rb_complex_polar$/i - !ruby/regexp /^rb_data_object_alloc$/i - !ruby/regexp /^rb_data_object_get_warning$/i - !ruby/regexp /^rb_data_object_wrap_warning$/i - !ruby/regexp /^rb_data_typed_object_alloc$/i https://github.com/ruby-go-gem/ruby_header_parser/blob/v0.4.2/config/default.yml.erb#L7
94. 94 Solution. ruby_header_parser/config/default.yml.erb function: exclude_name: # internal macros - !ruby/regexp /^rb_define_global_function_(m?[0-9]+|notimpl)$/i - !ruby/regexp /^rb_define_method_(m?[0-9]+|notimpl)$/i - !ruby/regexp /^rb_define_method_id_(m?[0-9]+|notimpl)$/i - !ruby/regexp /^rb_define_module_function_(m?[0-9]+|notimpl)$/i - !ruby/regexp /^rb_define_private_method_(m?[0-9]+|notimpl)$/i - !ruby/regexp /^rb_define_protected_method_(m?[0-9]+|notimpl)$/i - !ruby/regexp /^rb_define_singleton_method_(m?[0-9]+|notimpl)$/i https://github.com/ruby-go-gem/ruby_header_parser/blob/v0.4.2/config/default.yml.erb#L33
95. • Some functions were removed in Ruby 3.4 ◦ e.g. rb_data_object_make ◦ https://bugs.ruby-lang.org/issues/20265 ◦ https://bugs.ruby-lang.org/issues/18290 • Calling a non-existent C function from within Go code will result in an error at build time 95 Issue 4. Removed functions
96. • Even if these are removed in Ruby 3.4+, Ruby 3.3 bindings need to be able to use it as these are • It was necessary to use different Go bindings depending on the version of Ruby at the time of build 96 Issue 4. Removed functions
97. 97 Solution. Split binding files $ ls -1 ruby/*_generated.go ruby/enum_ruby_3_3_generated.go ruby/enum_ruby_3_4_generated.go ruby/function_ruby_3_3_generated.go ruby/function_ruby_3_4_generated.go ruby/type_ruby_3_3_generated.go ruby/type_ruby_3_4_generated.go https://github.com/ruby-go-gem/go-gem-wrapper/tree/v0.7.1/ruby
98. • Ability to change files to be built according to conditions ◦ e.g. “ruby_linux_amd64.go”, “ruby_darwin_arm64.go” • https://pkg.go.dev/go/build#hdr-Build_Constraints • https://pkg.go.dev/cmd/go#hdr-Build_constraints 98 Build Constraints (a.k.a. Build tag)
99. 99 Custom Build Constraints // THE AUTOGENERATED LICENSE. ALL THE RIGHTS ARE RESERVED BY ROBOTS. // WARNING: This file has automatically been generated // Code generated by ruby_h_to_go. DO NOT EDIT. //go:build ruby_3_4 package ruby https://github.com/ruby-go-gem/go-gem-wrapper/blob/v0.7.3/ruby/function_ruby_3_4_generated.go#L1-L8 custom build tag
100. 100 GOFLAGS=’-tags=ruby_x_x’ go build def create_go_makefile(target, srcprefix: nil, go_build_args: nil) (snip) goflags = "-tags=#{GoGem::Util.ruby_minor_version_build_tag}" File.open("Makefile", "a") do |f| f.write <<~MAKEFILE.gsub(/^ {8}/, "\t") $(DLLIB): Makefile $(srcdir)/*.go cd $(srcdir); \ CGO_CFLAGS='$(INCFLAGS)' CGO_LDFLAGS='#{ldflags}' GOFLAGS='#{goflags}' \ go build -p 4 -buildmode=c-shared -o #{current_dir}/$(DLLIB) #{go_build_args} MAKEFILE end end https://github.com/ruby-go-gem/go-gem-wrapper/blob/v0.7.3/_gem/lib/go_gem/mkmf.rb#L38-L47
101. • https://pkg.go.dev/ is the official reference site for Go standard libraries and modules ◦ e.g. https://pkg.go.dev/github.com/ruby-go-gem/go-gem-wrapper ◦ Similar to https://docs.ruby-lang.org/ + https://rubydoc.info/ 101 Issue 5. Supports https://pkg.go.dev/
102. • Before Custom Build Constraints was introduced, reference was displayed correctly on the https://pkg.go.dev/ • But after, broken… 102 Issue 5. Supports https://pkg.go.dev/
103. Good 103 All go bindings were missing Bad
104. • https://pkg.go.dev/ doesn't support Custom Build Constraints • I don't want 1,100 bindings not showing up on https://pkg.go.dev/ • I thought about building this myself and hosting this on GitHub Pages, but I thought this would be confusing to have Go module references published outside of https://pkg.go.dev/ 104 Why?
105. • I want to fallback to ruby_3_3 if there is no Custom Build Constraints • However, Go's current Build Constraints doesn’t allow such a condition to be specified. (maybe) 105 Ideal and Reality
106. 106 Solution: ruby_h_to_go/config.yml default_tag: ruby_3_3 available_tags: - ruby_3_3 - ruby_3_4 https://github.com/ruby-go-gem/go-gem-wrapper/blob/v0.7.3/_tools/ruby_h_to_go/config.yml
107. 107 ruby_h_to_go/lib/ruby_h_to_go/go_util.rb ruby_build_tag = GoGem::Util.ruby_minor_version_build_tag header << if ruby_build_tag == RubyHToGo.config.default_tag other_tags = RubyHToGo.config.available_tags - [RubyHToGo.config.default_tag] condition = other_tags.map { |tag| "!#{tag}" }.join(" && ") <<~GO // FIXME: https://pkg.go.dev/ doesn't support custom build tag. // Therefore, if no build tag is passed, treat it as the default tag //go:build #{ruby_build_tag} || (#{condition}) GO else <<~GO //go:build #{ruby_build_tag} GO end https://github.com/ruby-go-gem/go-gem-wrapper/blob/v0.7.3/_tools/ruby_h_to_go/lib/ruby_h_to_go/go_util.rb#L27-L45
108. 108 function_ruby_3_3_generated.go // THE AUTOGENERATED LICENSE. ALL THE RIGHTS ARE RESERVED BY ROBOTS. // WARNING: This file has automatically been generated // Code generated by ruby_h_to_go. DO NOT EDIT. // FIXME: https://pkg.go.dev/ doesn't support custom build tag. // Therefore, if no build tag is passed, treat it as the default tag //go:build ruby_3_3 || !ruby_3_4 package ruby https://github.com/ruby-go-gem/go-gem-wrapper/blob/v0.7.3/ruby/function_ruby_3_3_generated.go
109. 109 After ruby 3.5 is released (maybe) // THE AUTOGENERATED LICENSE. ALL THE RIGHTS ARE RESERVED BY ROBOTS. // WARNING: This file has automatically been generated // Code generated by ruby_h_to_go. DO NOT EDIT. // FIXME: https://pkg.go.dev/ doesn't support custom build tag. // Therefore, if no build tag is passed, treat it as the default tag //go:build ruby_3_3 || (!ruby_3_4 && !ruby_3_5) package ruby https://github.com/ruby-go-gem/go-gem-wrapper/blob/v0.7.3/ruby/function_ruby_3_3_generated.go
110. 110 Tips. Propagate Ruby Reference to Go
111. 111 RubyKaigi 2015 https://www.slideshare.net/slideshow/ruby-meets-go/56074507#32
112. 112 RubyKaigi 2015 https://www.slideshare.net/slideshow/ruby-meets-go/56074507#33
113. • go-gem-wrapper provides a mechanism for managing Ruby reference counters in Go 113 Solution
114. 114 Example (1/2) //export rb_example_go_struct_set func rb_example_go_struct_set(self C.VALUE, x C.VALUE, y C.VALUE) { data := (*GoStruct)(ruby.GetGoStruct(ruby.VALUE(self))) data.x = ruby.NUM2INT(ruby.VALUE(x)) data.y = ruby.NUM2INT(ruby.VALUE(y)) } //export rb_example_go_struct_get func rb_example_go_struct_get(self C.VALUE) C.VALUE { data := (*GoStruct)(ruby.GetGoStruct(ruby.VALUE(self))) ret := []ruby.VALUE{ ruby.INT2NUM(data.x), ruby.INT2NUM(data.y), } return C.VALUE(ruby.Slice2rbAry(ret)) } https://pkg.go.dev/github.com/ruby-go-gem/go-gem-wrapper/ruby#NewGoStruct
115. 115 Example (2/2) //export go_struct_alloc func go_struct_alloc(klass C.VALUE) C.VALUE { data := GoStruct{} return C.VALUE(ruby.NewGoStruct(ruby.VALUE(klass), unsafe.Pointer(&data))) } //export Init_example func Init_example() { rb_mExample := ruby.RbDefineModule("Example") // Create Example::GoStruct class rb_cGoStruct := ruby.RbDefineClassUnder(rb_mExample, "GoStruct", ruby.VALUE(C.rb_cObject)) ruby.RbDefineAllocFunc(rb_cGoStruct, C.go_struct_alloc) ruby.RbDefineMethod(rb_cGoStruct, "set", C.rb_example_go_struct_set, 2) ruby.RbDefineMethod(rb_cGoStruct, "get", C.rb_example_go_struct_get, 0) } https://pkg.go.dev/github.com/ruby-go-gem/go-gem-wrapper/ruby#NewGoStruct
116. Use go build in Ruby native extension building 116 Up to here CRuby Go
117. Use “ruby.h” in go building 117 From here CRuby Go
118. 118 Use “ruby.h” in go building
119. 1. go test 2. golangci-lint 119 Use case
120. • go test is standard testing command for Go • Functions created in native extension (not just Go) should be tested from Ruby ◦ Because this is called from Ruby • Functions created in pure-Go should be tested from Go ◦ Because this is called from Go 120 Use case 1. go test
121. 郷に入りては郷に従え Go ni irite wa Go ni shitagae 121 When in Rome, do as the Romans do
122. • In order to execute go test, go build must be able to execute • “ruby.h” is needed to execute go build in Go gem • Various environment variables are required to build with “ruby.h” 122 Problem: go test requires building with “ruby.h”
123. 123 Minimum required environment variables for go build ENV["CGO_CFLAGS"] = "#{RbConfig::CONFIG["CFLAGS"]}" \ " -I#{RbConfig::CONFIG["rubyarchhdrdir"]}" \ " -I#{RbConfig::CONFIG["rubyhdrdir"]}" ENV["CGO_LDFLAGS"] = "-L#{RbConfig::CONFIG["libdir"]}" \ " -l#{RbConfig::CONFIG["RUBY_SO_NAME"]}" ENV["LD_LIBRARY_PATH"] = "#{RbConfig::CONFIG["libdir"]}"
124. 124 go test full command GOFLAGS=-tags=ruby_3_4 CGO_CFLAGS=-fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fdeclspec -O3 -fno-fast-math -ggdb3 -Wall -Wextra -Wextra-tokens -Wdeprecated-declarations -Wdivision-by-zero -Wdiv-by-zero -Wimplicit-function-declaration -Wimplicit-int -Wpointer-arith -Wshorten-64-to-32 -Wwrite-strings -Wold-style-definition -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wunused-variable -Wmisleading-indentation -Wundef -fno-common -pipe -I/Users/sue445/.rbenv/versions/3.4.1/include/ruby-3.4.0/arm64-darwin24 -I/Users/sue445/.rbenv/versions/3.4.1/include/ruby-3.4.0 CGO_LDFLAGS=-L/Users/sue445/.rbenv/versions/3.4.1/lib -lruby.3.4 -undefined dynamic_lookup LD_LIBRARY_PATH=/Users/sue445/.rbenv/versions/3.4.1/lib go test -mod=readonly -count=1 ./...
125. # Rakefile require "go_gem/rake_task" GoGem::RakeTask.new("example") Solution GoGem::RakeTask Provides rake tasks for go test with CRuby 125
126. $ bundle exec rake -T | grep "go:" rake go:build_envs[env_name] # Print build envs for `go build` rake go:build_tag # Print build tag rake go:fmt # Run go fmt rake go:test # Run go test rake go:testrace # Run go test -race rake go tasks 126
127. • With Go, in addition to the official linter, there are various 3rd party tools for each linter • In Ruby, it's similar to the way gem is created for each RuboCop’s Cop 127 Go’s linter situation
128. • golangci-lint is linters runner for Go • https://github.com/golangci/golangci-lint • This is 3rd party tool but widely used in the Go community 128 Use case 2. golangci-lint
129. # Rakefile require "go_gem/rake_task" go_task = GoGem::RakeTask.new("gem_name") namespace :go do desc "Run golangci-lint" task :lint do go_task.within_target_dir do sh "which golangci-lint" do |ok, _| raise "golangci-lint isn't installed. See. https://golangci-lint.run/welcome/install/" unless ok end build_tag = GoGem::Util.ruby_minor_version_build_tag sh GoGem::RakeTask.build_env_vars, "golangci-lint run --build-tags #{build_tag} --modules-download-mode=readonly" end end end e.g. Use golangci-lint in Go gem 129 https://github.com/ruby-go-gem/go-gem-wrapper/tree/v0.7.3/_gem#example-add-additional-tasks
130. • “patch_for_go_gem” is needed to make Go gem • I want to use bundler gem similar to --ext=c and --ext=rust 130 Add --ext=go to bundle gem command
131. 131 https://github.com/orgs/rubygems/discussions/8128
132. • I waited for a while and got no response • So I had hsbt triage it on ruby-jp slack 132 Feature Proposal
133. 133 LGTM
134. 134 https://github.com/rubygems/rubygems/pull/8183
135. 135 Pros/Cons of Go gem
136. • We can use Go modules in Ruby 136 Pros 1
137. • We can write in Go in areas where Ruby is not good at (and Go is very good at) • Go's features not found in other languages ◦ goroutine: lightweight thread ▪ https://go.dev/tour/concurrency/1 ◦ channel: Messaging between multiple goroutine ▪ https://go.dev/tour/concurrency/2 137 Pros 2
138. • Benchmark with tak function ◦ https://www.ruby-lang.org/en/news/2020/12/25/ruby-3-0-0-released/ • Multi thread and multi process are not eligible ◦ Because the results depend on the number of CPU cores on the machine being benchmarked ◦ This is not fair to compare these to goroutine 138 Ractor vs Fiber vs goroutine
139. 139 goroutine is 30x faster than Ractor $ ruby tarai.rb go version go1.24.1 darwin/arm64 ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +PRISM [arm64-darwin24] Comparison: Go: goroutine: 1.6 i/s Go: sequential: 0.5 i/s - 3.65x slower Ruby: Ractor: 0.1 i/s - 31.58x slower Ruby: sequential: 0.0 i/s - 92.85x slower Ruby: Fiber: 0.0 i/s - 93.08x slower https://github.com/ruby-go-gem/go-gem-wrapper/tree/v0.7.3/_benchmark
140. 140 @osyoyu says https://speakerdeck.com/osyoyu/zigdeckuo-zhang-wozuo-ru-2024-edition?slide=34
141. 141 Native extension with Zig https://speakerdeck.com/osyoyu/zigdeckuo-zhang-wozuo-ru-2024-edition
142. • go-gem-wrapper is a low-layer wrapper, so not much different than making Native extension with C ◦ Similar to writing in Go where you would write in C ◦ Need to be aware of both CRuby and Go 142 Cons 1
143. • Go's variable-length arguments couldn't be passed directly to C ◦ CRuby provides both variable length argument and non-variable length argument versions of its functions. ◦ If we only have functions with variable-length arguments, it is a bit of a problem ▪ e.g. rb_raise 143 Cons 2
144. 144 Workaround for rb_raise https://github.com/ruby-go-gem/go-gem-wrapper/blob/v0.7.3/ruby/ruby_internal_error.go /* #include "ruby.h" void __rb_raise(VALUE exception, const char *str) { rb_raise(exception, "%s", str); } */ import "C" func RbRaise(exc VALUE, format string, a ...interface{}) { str := fmt.Sprintf(format, a...) char, clean := string2Char(str) defer clean() C.__rb_raise(C.VALUE(exc), char) }
145. • https://github.com/orgs/ruby-go-gem/discussions/236 145 I'm looking for good ideas and patches!
146. • Not intended for use with ruby-head, since binding is required for each Ruby version 146 Cons 3
147. • SEGV when calling CRuby functions in goroutine • c.f. https://github.com/ruby-go-gem/go-gem-wrapper/issues/286 147 Cons 4
148. This doesn’t work… (SEGV!!!!) 148 e.g. Call yield inside goroutine //export rb_example_call_proc_inside_goroutine func rb_example_call_yield_inside_goroutine(self C.VALUE) { if ruby.RbBlockGivenP() == 0 { ruby.RbRaise(C.rb_eArgError, "Block not given") } go func() { ruby.RbYield(C.Qnil) }() }
149. • glibc is required ◦ Not supported for use with musl (e.g. alpine) ◦ c.f. https://github.com/ruby-go-gem/go-gem-wrapper/issues/253 ◦ Windows (maybe) 149 Cons 5
150. 150 Real use case
151. • Perform HTTP requests in parallel with goroutine • Unlike thread and process, we don't have to specify concurrency ◦ e.g. max threads, max processes 151 https://github.com/sue445/funnel_http
152. 152 Usage require "funnel_http" client = FunnelHttp::Client.new requests = [ { method: :get, url: "https://example.com/api/user/1", }, ] responses = client.perform(requests) # => [ # { status_code: 200, body: "Response of /api/user/1", # header: { "Content-Type" => ["text/plain;charset=utf-8"]} } # ]
153. NOTE: open-uri and net/http doesn’t work in Ractor… 153 benchmark (Fiber vs goroutine) $ bundle exec ruby benchmark.rb go version go1.24.1 darwin/arm64 ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [arm64-darwin24] Comparison: FunnelHttp::Client#perform: 22.7 i/s sequential: 7.0 i/s - 3.27x slower Parallel with Fiber: 3.7 i/s - 6.14x slower https://github.com/sue445/funnel_http/tree/v0.3.2/benchmark
154. • Using https://github.com/ruby-go-gem/go-gem-wrapper makes native extension with Go very easy to make • We can write Ruby gem in Go in areas where Ruby is not good at (Go is very good at) • goroutine is very good! 154 Conclusion
155. goroutine is very good! 155 Conclusion
156. goroutine is very good! 156 Conclusion