Native Extensions. How do they work?

Ever seen those nasty configure/make backtraces on gem install? Most of your critical product dependencies are native extensions, yet very few Rubyists understand how they work. With the Ruby VM ecosystem constantly growing, discrepancies between development and production environments and wrapped libraries not always evolving with stable interfaces, things can and will go wrong. I am the author of several bindings that runs seamless on different Ruby implementations, cursed a lot to get there and in this talk would like to touch on:

Basic structure and how extensions are loaded at runtime
The MRI C api VS foreign function interface (FFI)
Plugging into the Ruby objects ecosystem and Garbage Collection
Playing well with Threads
When things go wrong

Transcript

1. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Native Extensions: how do they work ? Lourens Naudé RailsWayCon 2012 Berlin - http://railswaycon.com/2012 Wednesday, June 6, 12
2. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Operations @ WildfireApp.com Bio Wednesday, June 6, 12
3. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Agenda Important, yet volatile When things go wrong FFI VS C api GC Threading Wednesday, June 6, 12
4. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ERROR: Failed to build gem native extension. Seen this ? Wednesday, June 6, 12
5. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 [BUG] Segmentation fault this ? Wednesday, June 6, 12
6. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 dyld: Symbol not found: _mysql2_set_field_string_encoding Referenced from: ... Expected in: flat namespace or this ? Wednesday, June 6, 12
7. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Your critical path and I/O depends on it. Why this is important ? Wednesday, June 6, 12
8. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Often misunderstood ... Wednesday, June 6, 12
9. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Bindings ... not only for performance Wednesday, June 6, 12
10. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Integration requirements Lower subsystems Commercial static linking C lib code reuse Instrumentation Wednesday, June 6, 12
11. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Much more likely to break than ANY Ruby code. Volatile components Wednesday, June 6, 12
12. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Breakage - operating system First run on OS OS agnostic library Extension, NOT Upgrades Wednesday, June 6, 12
13. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Breakage - interpreter Interpreter version bump Deprecations GC changes Threading changes Wednesday, June 6, 12
14. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Breakage - implementation MRI only ? Encodings Non-standard macros Wednesday, June 6, 12
15. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Breakage - library ABI Incompatible system lib apt-get, homebrew etc. Extension should assert versions Wednesday, June 6, 12
16. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Breakage - operations lib removal (dynamic linking) Dependencies bumped Incompatible compiler versions New AMI Wednesday, June 6, 12
17. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Breakage - author sloppiness No GC integration No threading support Memory allocs will fail Ignore error returns Wednesday, June 6, 12
18. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Breakage - malpractice Read the manual ? Can it be called out to ? Ruby and C disconnect Respect resources Wednesday, June 6, 12
19. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 System or tarball dependency ? Distribution Wednesday, June 6, 12
20. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Distribution - system Dev + ops overhead Package manager mkmf is stable Hard to enforce versions Bug reports: error surface Wednesday, June 6, 12
21. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Distribution - tarball No dev (or ops) overhead !! OS agnostic installs !! Sandbox - beta versions Wednesday, June 6, 12
22. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 require “mysql2” Loading Wednesday, June 6, 12
23. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 What happens dlopen() or similar mysql2.dylib Init_mysql2 symbol Modules, objects and global state Method prototypes Wednesday, June 6, 12
24. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 VALUE mMysql2, cMysql2Error; /* Ruby Extension initializer */ void Init_mysql2() { mMysql2 = rb_define_module("Mysql2"); cMysql2Error = rb_const_get(mMysql2, rb_intern("Error")); init_mysql2_client(); init_mysql2_result(); } eg. mysql2 Wednesday, June 6, 12
25. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Reporting problems When things go wrong Wednesday, June 6, 12
26. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 OS and compiler OS version Compiler version Platform: 32 or 64 bit Wednesday, June 6, 12
27. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 library library and version Package manager compile ? Installed for correct platform ? Wednesday, June 6, 12
28. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 methodmissing:lib lourens$ otool -L rbczmq_ext.bundle rbczmq_ext.bundle: /Users/lourens/.rvm/rubies/ruby-1.8.7-p357/lib/libruby.dylib (compatibility version 1.8.0, current version 1.8.7) /Users/lourens/projects/rbczmq/ext/rbczmq/dst/lib/libzmq. 1.dylib (compatibility version 2.0.0, current version 2.1.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 52.0.0) Linked versions Wednesday, June 6, 12
29. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Ruby install Implementation and version How was it compiled ? Native C API ? FFI gem ? Wednesday, June 6, 12
30. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 1.8.7 :001 > require 'rbconfig' 1.8.7 :002 > pp RbConfig::CONFIG {"build"=>"i686-apple-darwin11.0.1", "LDSHARED"=>"cc -arch x86_64 -dynamiclib -undefined suppress -flat_namespace", "configure_args"=> " '--prefix=/Users/lourens/.rvm/rubies/ruby-1.8.7-p352' '--enable- shared' '--disable-install-doc' 'CC=/usr/bin/gcc-4.2' 'CFLAGS=-arch x86_64 -g -Os -pipe -no-cpp-precomp' 'LDFLAGS=-arch x86_64 - bind_at_load'", "CPP"=>"/usr/bin/gcc-4.2 -E" Ruby config Wednesday, June 6, 12
31. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Questions to ask Program under stress ? Non-deterministic ? (GC && threads) Conflicting native gem requires ? Wednesday, June 6, 12
32. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Runtime context Context Wednesday, June 6, 12
33. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Runtime - gdb Available on most systems Tracing execution C backtrace gdb -c <CMD> gdb -p <PID> Wednesday, June 6, 12
34. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 (gdb) bt 3 #0 0x00007fff8f51dda8 in memmove$VARIANT$sse42 () #1 0x0000000104b63345 in rb_thread_save_context (th=0x104dd50e0) at eval.c:10627 #2 0x0000000104b6e071 in rb_thread_schedule () at eval.c:11307 (More stack frames follow...) Example backtrace Wednesday, June 6, 12
35. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Runtime - strace System calls && signals Observing app <=> OS System backtrace strace <CMD> strace -p <PID> Wednesday, June 6, 12
36. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ( Apple fanboys ) sudo dtruss <CMD> sudo dtruss -p <PID> Wednesday, June 6, 12
37. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 select(14, [5 7 13], [], [], {0, 90519}) = 0 (Timeout) gettimeofday({1328183002, 561382}, NULL) = 0 munmap(0x7f576eaee000, 41766912) = 0 brk(0x16f685000) = 0x16f685000 brk(0x16a6db000) = 0x16a6db000 ^CProcess 5360 detached Example system trace Wednesday, June 6, 12
38. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Runtime - lsof Files, pipes, sockets && devices File ownership Assert correct libs are loaded Enumerate file descriptors lsof -p <PID> Wednesday, June 6, 12
39. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ruby 22103 user 0r CHR 1,3 0t0 735 /dev/null ruby 22103 user 1u CHR 1,3 0t0 735 /dev/null ruby 22103 user 2u CHR 1,3 0t0 735 /dev/null ruby 22103 user 3r FIFO 0,8 0t0 324848777 pipe ruby 22103 user 4w FIFO 0,8 0t0 324848740 pipe ruby 22103 user 5u IPv4 284775119 0t0 TCP *:17673 (LISTEN) Process descriptors Wednesday, June 6, 12
40. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Managing Risk Checklist Wednesday, June 6, 12
41. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Devil’s advocate Do you really need an extension ? CLI library interface ? Wrapped library best practices ? Resource limits ? Ruby implementation limits ? Wednesday, June 6, 12
42. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Best practices Do some research Shell out for once off requests Know tools for when things go wrong Detail bug reports Native dependencies in gems gem pristine -all Wednesday, June 6, 12
43. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Foreign Function Interface FFI Wednesday, June 6, 12
44. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Code written in one language can call code in another. libffi Wednesday, June 6, 12
45. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 gem install ffi Ruby integration Wednesday, June 6, 12
46. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Disclaimer Never written a FFI extension Know the C API very well Many folks in the same boat Wednesday, June 6, 12
47. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Evolving Ruby ecosystem Why it matters Wednesday, June 6, 12
48. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 JRuby C API since 1.6 Not perfect, but compiles most exts Deprecated Great FFI support Wednesday, June 6, 12
49. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Rubinius Very stable C API Compiles most MRI extensions Great FFI support Wednesday, June 6, 12
50. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 MRI Standard C API Good FFI support, flaky on 1.8 Wednesday, June 6, 12
51. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Caveats Performance penalty Negligible out of critical path Dependencies - libffi + ffi gem More moving parts Wednesday, June 6, 12
52. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Garbage Collector Integration Wednesday, June 6, 12
53. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 struct RData { struct RBasic basic; void (*dmark) _((void*)); void (*dfree) _((void*)); void *data; }; GC callbacks Wednesday, June 6, 12
54. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 GC mark callback Invoked during mark phase Let GC know of other objects Wednesday, June 6, 12
55. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 GC free callback Invoked when no refs found Frees self and other resources Sockets, engines etc. Wednesday, June 6, 12
56. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 typedef struct { size_t delay; size_t times; Bool cancelled; VALUE callback; } zmq_timer_wrapper; GC - struct example Wednesday, June 6, 12
57. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 VALUE timer; zmq_timer_wrapper *tr = NULL; timer = Data_Make_Struct(rb_cZmqTimer, zmq_timer_wrapper, rb_czmq_mark_timer, rb_czmq_free_timer_gc, tr); GC - struct integration Wednesday, June 6, 12
58. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 static void rb_czmq_mark_timer(void *ptr) { zmq_timer_wrapper *timer = (zmq_timer_wrapper *)ptr; rb_gc_mark(timer->callback); } GC - mark callback example Wednesday, June 6, 12
59. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 static void rb_czmq_free_timer_gc(void *ptr) { zmq_timer_wrapper *timer = (zmq_timer_wrapper *)ptr; if (timer) xfree(timer); } GC - free callback example Wednesday, June 6, 12
60. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Threads Integration Wednesday, June 6, 12
61. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 int rb_thread_select _((int, fd_set *, fd_set *, fd_set *, struct timeval *)); rb_thread_select Wednesday, June 6, 12
62. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 rb_thread_select User space replacement for select() Allows other threads to run I/O bound threads Blocks process otherwise mysql VS mysql2 Wednesday, June 6, 12
63. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 GIL API for releasing the GIL MRI 1.9 and Rubinuis Calling C funcs + system calls Unstable API Wednesday, June 6, 12
64. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 static VALUE rb_czmq_nogvl_zctx_destroy(void *ptr) { zmq_ctx_wrapper *ctx = ptr; zctx_destroy(&ctx->ctx); return Qnil; } rb_thread_blocking_region(rb_czmq_nogvl_zctx_ destroy, ctx, RUBY_UBF_IO, 0); rb_thread_select() Wednesday, June 6, 12
65. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Passing thoughts Vibrant ruby ecosystem Extensions: brittle integration point Tokaido ( formerly Rails.app ) Shipping binary builds http://yehudakatz.com/2012/06/05/ tokaido-status-update- implementation-details Wednesday, June 6, 12
66. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Questions ? O fim Wednesday, June 6, 12
67. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 @methodmissing github.com/methodmissing Thanks! Wednesday, June 6, 12