Introduction to CRuby Source Code MWRC

Understanding of CRuby source code has profound effects on every Ruby developer. In my talk, I will show you how to build Ruby from source. I will explain how to install and configure your new Ruby build on Mac and Linux. I will walk you through CRuby source code and introduce you to a few of the most important CRuby files. I will show you how to hack CRuby and modify some of the fundament Ruby classes in C. I will demonstrate how to write complete Ruby classes in C. Finally, I will show you that CRuby code can run 100 times faster than Ruby code. I hope that this talk will inspire you to learn more about CRuby and hack it on your own.

Transcript

1. Introduction to CRuby Source Code Andy Pliszka ! ! @AntiTyping AntiTyping.com github.com/AntiTyping
2. Agenda • Motivation • CRuby Overview • Hacking CRuby @AntiTyping
3. Prolog
   Motivation
4. Understand Ruby • Object Model • Metaprogramming • Garbage Collection @AntiTyping
5. Speed Up Ruby • Rewrite parts of your code in C • 10x - 50x times faster @AntiTyping
6. C Extensions • Write Ruby extensions in C • Gems @AntiTyping
7. Ruby + C • Ruby productivity and ecosystem • C efficiency, speed, and algorithms @AntiTyping
8. Level 1
   Build Ruby from Scratch
9. Get the source ! $ mkdir ~/mwrc-ruby $ chmod go-w ~/mwrc-ruby $ cd ~/mwrc-ruby $ git clone git@github.com:ruby/ruby.git $ cd ruby $ git checkout v2_0_0_247 fix for make check @AntiTyping
10. Configure (Mac) $ brew install openssl $ autoconf $ ./configure --prefix=$HOME/myruby --with- opt-dir=/usr/local/Cellar/openssl/1.0.1f optflags="-O0" debugflags="-g" --disable- install-doc @AntiTyping
11. ./configure @AntiTyping
12. Configure (Linux) $ sudo apt install libssl-dev $ autoconf $ ./configure --prefix=$HOME/myruby optflags="-O0" debugflags="-g3 -ggdb" -- disable-install-doc @AntiTyping
13. Compile and Build $ make
14. Check build $ make check
15. Install $ make install
16. Setup PATHs $ export PATH=$HOME/myruby/bin:$PATH $ export GEM_HOME=$HOME/myruby/lib/ ruby/gems/2.0.0 $ export GEM_PATH=$HOME/myruby/lib/ruby/ gems/2.0.0 @AntiTyping
17. Trust but Verify $ which irb /Users/apliszka/myruby/bin/irb $ irb irb(main):001:0> raise "hi" RuntimeError: hi from (irb):1 from /Users/apliszka/myruby/bin/irb:12:in `<main>' $ which ruby /Users/apliszka/myruby/bin/ruby @AntiTyping
18. gem env $ gem env ! RubyGems Environment: - RUBYGEMS VERSION: 2.0.3 - RUBY VERSION: 2.0.0 (2013-06-27 patchlevel 247) [x86_64-darwin12.5.0] - INSTALLATION DIRECTORY: /Users/apliszka/myruby/lib/ruby/gems/2.0.0 - RUBY EXECUTABLE: /Users/apliszka/myruby/bin/ruby - EXECUTABLE DIRECTORY: /Users/apliszka/myruby/bin - RUBYGEMS PLATFORMS: - ruby - x86_64-darwin-12 - GEM PATHS: - /Users/apliszka/myruby/lib/ruby/gems/2.0.0 - GEM CONFIGURATION: - :update_sources => true - :verbose => true - :backtrace => false - :bulk_threshold => 1000 - REMOTE SOURCES: - https://rubygems.org/ @AntiTyping
19. Gems $ gem list ! *** LOCAL GEMS *** ! bigdecimal (1.2.0) io-console (0.4.2) json (1.7.7) minitest (4.3.2) psych (2.0.0) rake (0.9.6) rdoc (4.0.0) test-unit (2.0.0.0) $ ls ~/myruby/lib/ruby/gems/2.0.0/gems/! rake-0.9.6! rdoc-4.0.0! test-unit-2.0.0.0 @AntiTyping
20. Install Rails $ ls ~/myruby/lib/ruby/gems/2.0.0/gems/! ! actionmailer-4.0.0! actionpack-4.0.0! activemodel-4.0.0! activerecord-4.0.0! activerecord-deprecated_finders-1.0.3! activesupport-4.0.0! arel-4.0.1! atomic-1.1.14! builder-3.1.4! ... $ gem install rails --no-doc @AntiTyping
21. Rails app $ rails new HelloRuby! create! create README.rdoc! create Rakefile! ...! ! $ cd HelloRuby! ! $ rails s! ! => Booting WEBrick! => Rails 4.0.0 application starting in development on http:// 0.0.0.0:3000! => Run `rails server -h` for more startup options! => Ctrl-C to shutdown server! [2013-10-29 15:54:21] INFO WEBrick 1.3.1! ...
22. RubyMine Setup
23. Level 1 Complete • Build our own version of Ruby • Installed it • Rails app is using our Ruby (~/myruby) • It took ~30min 10
24. Level 2
    Debugger
25. Debuggers • lldb • gdb • Xcode lldb • Vim • Emacs • cgdb @AntiTyping
26. lldb $ lldb ~/myruby/bin/ruby upstring.rb! (lldb) breakpoint set -l 4779 -f string.c (lldb) run puts "Hello".upcase String#upcase @AntiTyping
27. lldb output
28. XCode lldb
29. Level 2 Complete • Debug our CRuby code using lldb 17
30. Level 3
    CRuby Source Code
31. Folder structure @AntiTyping
32. Fixnum @AntiTyping
33. String @AntiTyping
34. Method definition rb_define_method(rb_cArray, "length", rb_ary_length, 0); static VALUE rb_ary_length(VALUE self) { long len = RARRAY_LEN(self); return LONG2NUM(len); } #1 #2 @AntiTyping
35. Class definition void Init_CLongArray(void) { ! # 1. Create a class cCLongArray = rb_define_class("CLongArray", rb_cObject); ! # 2. Allocate memory rb_define_alloc_func(cCLongArray, array_alloc); ! # 3. Define constructor rb_define_method(cCLongArray, "initialize", array_initialize, 1); ! # 4. Define constructor rb_define_method(cCLongArray, "qsort", array_quick_sort, 0); rb_define_method(cCLongArray, "[]", array_aref, 1); rb_define_method(cCLongArray, "[]=", array_aset, 2); } @AntiTyping
36. Two Worlds • C! • Working directly with memory • malloc/free • Ruby! • Working with heap and objects • GC @AntiTyping
37. Type conversion Ruby Fixnum -> C long long c_num = NUM2LONG(ruby_num); C long -> Ruby Fixnum VALUE ruby_num = LONG2NUM(c_num); @AntiTyping
38. Level 3 Complete • CRuby folder structure • How to define a method • How to define a new class • How to convert data C <-> Ruby 20
39. Level 4
    Hacking CRuby
40. Fixnum#fib long fibonacci(long n) { long u = 0; long v = 1; long i, t; for(i = 2; i <= n; i++) { t = u + v; u = v; v = t; } return v; } class Fixnum def fib u = 0 v = 1 t = 1 2.upto(self) do t = u + v u = v v = t end t end end Ruby C @AntiTyping
41. Fixnum#fib long fibonacci(long n) { long u = 0; long v = 1; long i, t; for(i = 2; i <= n; i++) { t = u + v; u = v; v = t; } return v; } static VALUE fix_cfib(VALUE self) { long u = 0; long v = 1; long i, t; for(i = 2; i <= NUM2LONG(self); i++ { t = u + v; u = v; v = t; } return LONG2NUM(v); } C CRuby rb_define_method(rb_cFixnum, "cfib", fix_cfib, 0); numeric.c @AntiTyping
42. #fib performance it "1M benchmark" do puts "fib(80)" puts Benchmark.measure() { 1000000.times { 80.fib } } end ! it "C 1M benchmark" do puts "cfib(80)" puts Benchmark.measure() { 1000000.times { 80.cfib } } end fib(80) 26.440000 0.050000 26.490000 ( 26.704735) cfib(80) 0.870000 0.000000 0.870000 ( 0.879011) CRuby is ~30x faster than Ruby (Macbook Pro) @AntiTyping
43. Fixnum#fib Demo
44. Fixnum#prime? static VALUE fix_cprime(VALUE self) { long number = NUM2LONG(self); long i; for (i = 2; i < number; i++) { if (number % i == 0 && i != number) return Qfalse; } return Qtrue; } class Fixnum def prime? 2.upto(self) do |i| if self % i == 0 && i != self return false; end end true end end Ruby CRuby rb_define_method(rb_cFixnum, "cprime?", fix_cprime, 0) numeric.c @AntiTyping
45. #cprime? performance prime? 30.510000 0.060000 30.570000 ( 31.414153) cprime? 1.590000 0.000000 1.590000 ( 1.774056) CRuby is ~17x faster than Ruby (Macbook Pro) it "prime? benchmark" do puts "prime?" puts Benchmark.measure { 94418953.prime? } #Markov end ! it "cprime? benchmark" do puts "cprime?" puts Benchmark.measure { 94418953.cprime? } #Markov end @AntiTyping
46. CLongArray class void Init_CLongArray(void) { cCLongArray = rb_define_class("CLongArray", rb_cObject); ! rb_define_alloc_func(cCLongArray, array_alloc); ! rb_define_method(cCLongArray, "initialize", array_initialize, 1); rb_define_method(cCLongArray, "qsort", array_quick_sort, 0); rb_define_method(cCLongArray, "[]", array_aref, 1); rb_define_method(cCLongArray, "[]=", array_aset, 2); } @AntiTyping
47. CLongArray alloc static VALUE array_alloc(VALUE klass) { VALUE self; array_t *array; self = Data_Make_Struct(klass, array_t, arr_mark, arr_free, array); return self; } typedef struct { long *array; long size; } array_t; @AntiTyping
48. CLongArray init static VALUE array_initialize(VALUE self, long size) { array_t *array; Data_Get_Struct(self, array_t, array); array->array = malloc(size * sizeof(long)); array->size = size; return self; } typedef struct { long *array; long size; } array_t; CLongArray.new(10) @AntiTyping
49. CLongArray [] static VALUE array_aref(VALUE self, VALUE index) { array_t *array; Data_Get_Struct(self, array_t, array); long idx = NUM2LONG(index); return LONG2NUM(array->array[idx]); } ! puts c_long_array[1] rb_define_method(cCLongArray, "[]", array_aref, 1); @AntiTyping
50. CLongArray []= static VALUE array_aset(VALUE self, VALUE index, VALUE val) { array_t *array; Data_Get_Struct(self, array_t, array); long idx = NUM2LONG(index); array->array[idx] = NUM2LONG(val); return val; } ! c_long_array[1] = 99 rb_define_method(cCLongArray, "[]=", array_aset, 2); @AntiTyping
51. CLongArray qsort static VALUE array_quick_sort(VALUE self) { array_t *array; Data_Get_Struct(self, array_t, array); quick_sort(array->array, array->size); return self; } c_long_array.qsort rb_define_method(cCLongArray, "qsort", array_quick_sort, 0); @AntiTyping
52. Plain C QuickSort void quick_sort (long *a, long n) { if (n < 2) return; long p = a[n / 2]; long *l = a; long *r = a + n - 1; while (l <= r) { if (*l < p) { l++; continue; } if (*r > p) { r--; continue; } long t = *l; *l++ = *r; *r-- = t; } quick_sort(a, r - a + 1); quick_sort(l, a + n - l); } @AntiTyping
53. QuickSort Performance CRuby is ~10x faster than Ruby (Macbook Air) it “C 1_000_000 numbers benchmark" do puts "CLongArray#qsort" puts Benchmark.measure() { @c_long_array.qsort } end ! it "1_000_000 numbers benchmark" do puts "Array#sort!" puts Benchmark.measure() { @ruby_array.sort! } end CLongArray#qsort 0.270000 0.000000 0.270000 ( 0.173161) Array#sort! 1.910000 0.010000 1.920000 ( 1.786345) @AntiTyping
54. Level 4 Complete • Fixnum#cfib 30x faster • Fixnum#cprime? 17x faster • CLongArray@qsort 10x faster 31
55. Level 5
    Graphs
56. Graph Representation Adjacency-List Representation typedef struct _node_t { long node; struct _node_t *next; } node_t; typedef struct _graph_t { long side; node_t **graph; } graph_t; @AntiTyping
57. CGraph class void Init_cgraph(void) { cGraph = rb_define_class("CGraph", rb_cObject); ! rb_define_alloc_func(cGraph, cgraph_alloc); ! rb_define_method(cGraph, "initialize", cgraph_initialize, 1); rb_define_method(cGraph, "bfs", cgraph_bfs, 1); rb_define_method(cGraph, "dfs", cgraph_dfs, 1); rb_define_method(cGraph, "prim", cgraph_prim, 1); } @AntiTyping
58. CGraph alloc static VALUE cgraph_alloc(VALUE klass) { VALUE self; graph_t *graph; self = Data_Make_Struct(klass, graph_t, cgraph_mark, cgraph_free, graph); ! return self; } @AntiTyping
59. CGraph init static VALUE cgraph_initialize(VALUE self, VALUE side) { graph_t *graph; Data_Get_Struct(self, graph_t, graph); graph->side = NUM2LONG(side); graph->graph = make_grid_graph(graph->side); ! return self; } @AntiTyping
60. CGraph BFS static VALUE cgraph_bfs(VALUE self, VALUE source) { graph_t *graph; Data_Get_Struct(self, graph_t, graph); long side = graph->side; long *path; ! path = bfs(side, graph->graph, NUM2LONG(source)); ! VALUE trace = rb_ary_new(); ! rb_gc_register_address(&trace); int i; for (i = 0; i < side*side; i++) { rb_ary_push(trace, LONG2NUM(path[i])); }; rb_gc_unregister_address(&trace); ! xfree(path); return trace; } @AntiTyping
61. Plain C BFS long *bfs(long side, node_t **graph, long source) { long *path = calloc(side*side, sizeof(long)); long path_tail = 0; long *q = calloc(side*side, sizeof(long)); queue_t queue = {0, 0, q}; long *visited = calloc(side*side, sizeof(long)); ! enqueue(&queue, source); visited[source] = 1; while(any(&queue)) { long node = dequeue(&queue); path[path_tail++] = node; node_t *p = graph[node]; while (p) { long i = p->node; if (visited[i] == 0) { enqueue(&queue, i); visited[i] = 1; } p = p->next; } } return path; } @AntiTyping
62. Grid Graph Creation C -> 535.926 milliseconds Ruby -> 27,901.748 milliseconds CRuby is ~52x faster than Ruby (Macbook Air) 2048*2048 grid = 4,194,304 nodes @AntiTyping
63. Grid Graph BFS C -> 228.536 milliseconds Ruby -> 4,426.137 milliseconds CRuby is ~19x faster than Ruby (Macbook pro) 1024* 1024 grid = 1,048,576 nodes @AntiTyping
64. Level 5 Complete • BFS in C • CRuby ~19x faster than Ruby 47
65. Is it safe
 in production? @AntiTyping
66. Extract code into a gem @AntiTyping
67. Epilog • Ruby internals • Ruby metaprogramming • Ruby object model • Use C to speed up your Ruby algorithms • Rewrite in C are usually very easy • Original C algorithm code can be used without modifications @AntiTyping
68. Pivotal is hiring NYC, LA, SF, … everywhere :)
69. Questions? ! @AntiTyping