Bindings in Ruby - behind the magic of blocks

> Why block can see local variables defined before him? Why can it change them? What kind of sorcery is this?

This is some kind of question that I will try to answer in my talk. We will see examples of block and hidden secret hero behind the magic - `binding` object.

In our magical journey we will look into some Ruby code and some MRI/Rubinius/jRuby internals to help us better understand the magic. We will later use the new knowledge in practice to:

* understand erb
* understand binding.pry
* write our own openstruct-like implementation that stores data internally using local variables inside binding. In benchmark it's faster than openstruct, so yay us! ;)

Great magicians never reveal their secrets - but thankfully Ruby is open source, so we can lift the veil and see the real truth behind it. And believe me, it's quite beautiful!

Transcript

1. Bindings in Ruby behind the music magic of blocks
2. About me Piotr Szmielew http://github.com/esse http://piotr.szmielew.pl @essepl on twitter In Rails since 2009
3. a = 2 10.times do a += 1 end puts a
4. a = 2 10.times do a += 1 end puts a # => 12
5. Obvious?
6. Now harder:
7. class BlockTest def self.add_method local = 2 define_method : do local end end end b = BlockTest.new BlockTest.add_method b.
8. Will it work?
9. class BlockTest def self.add_method local = 2 define_method : do local end end end b = BlockTest.new BlockTest.add_method b. # => 2
14. Code (lambda) environment Then - what is a closure?
15. p = proc do a = 2 10.times do a += 1 end end puts RubyVM::InstructionSequence.disasm(p)
17. == disasm: <RubyVM::InstructionSequence:block in irb_binding@(irb)>===== 0004 putobject 2 0006 setlocal a, 3 0011 putobject 10 0013 send <callinfo!mid:times, argc:0, block:block (2 levels) in irb_binding> 0017 leave == disasm: <RubyVM::InstructionSequence:block (2 levels) in irb_binding@(irb)> 0004 getlocal a, 4 0007 putobject_OP_INT2FIX_O_1_C_ 0008 opt_plus <callinfo!mid:+, argc:1, ARGS_SIMPLE> 0010 dup 0011 setlocal a, 4
18. == disasm: <RubyVM::InstructionSequence:block in irb_binding@(irb)>===== 0004 putobject 2 0006 setlocal a, 3 0011 putobject 10 0013 send <callinfo!mid:times, argc:0, block:block (2 levels) in irb_binding> 0017 leave == disasm: <RubyVM::InstructionSequence:block (2 levels) in irb_binding@(irb)> 0004 getlocal a, 4 0007 putobject_OP_INT2FIX_O_1_C_ 0008 opt_plus <callinfo!mid:+, argc:1, ARGS_SIMPLE> 0010 dup 0011 setlocal a, 4
21. Code (lambda) environment What is a closure?
22. YARV Virtual machine with stack and heap
23. YARV Current execution state – contained in stackframe Therefore local variables are on stack
24. YARV Current execution state – contained in stackframe Therefore local variables are on stack
25. Using lambda / proc changes block in normal variable (first class citizen)
26. BUT
29. 2.2.4 :048 > binding => #<Binding:0x007ff5f9930fe8> 2.2.4 :049 > binding.class => Binding 2.2.4 :050 > binding.class.superclass => Object
30. Binding Object binding Instance
31. Binding is created simultaneously with lambda
32. How exactly is binding created?
33. MRI // proc.c static VALUE rb_f_binding(VALUE self) { return rb_binding_new(); } (responsible for Kernel.binding method)
34. MRI // proc.c VALUE rb_binding_new(void) { rb_thread_t *th = GET_THREAD(); return rb_vm_make_binding(th, th->cfp); }
35. Next examples are from vm.c
36. VALUE rb_vm_make_binding(rb_thread_t *th, const rb_control_frame_t *src_cfp) { (...) rb_binding_t *bind; (...) envval = vm_make_env_object(th, cfp); (...) vm_bind_update_env(bind, envval); }
37. static VALUE vm_make_env_object(rb_thread_t *th, rb_control_frame_t *cfp) { VALUE envval = vm_make_env_each(th, cfp); (...) } MRI
38. vm_make_env_each(rb_thread_t *const th, rb_control_frame_t *const cfp) { (...)
39. /* * # local variables on a stack frame (N == local_size) * [lvar1, lvar2, ..., lvarN, SPECVAL] * ^ * ep[0] * * # moved local variables * [lvar1, lvar2, ..., lvarN, SPECVAL, Envval, BlockProcval (if needed)] * ^ ^ * env->env[0] ep[0] */
40. env_size = local_size + 1 /* envval */ + (blockprocval ? 1 : 0) /* blockprocval */; env_body = ALLOC_N(VALUE, env_size); MEMCPY(env_body, ep - (local_size - 1 /* specval */), VALUE, local_size); (...) env_ep = &env_body[local_size - 1 /* specval */]; (...) env = vm_env_new(env_ep, env_body, env_size, env_iseq); (...) return (VALUE)env; }
41. Rubinius def binding return Binding.setup( Rubinius::VariableScope.of_sender, Rubinius::CompiledCode.of_sender, Rubinius::LexicalScope.of_sender, Rubinius::VariableScope.of_sender.self, Rubinius::Location.of_closest_ruby_method ) end
42. # core / binding.rb def self.setup(variables, code, lexical_scope, recv=nil, location=nil) bind = allocate() bind.receiver = self_context(recv, variables) bind.variables = variables bind.compiled_code = code bind.lexical_scope = lexical_scope bind.location = location return bind end
43. # core / variable_scope.rb module Rubinius class VariableScope (...) def self.of_sender Rubinius.primitive :variable_scope_of_sender (...) end (...) end end Rubinius
44. Rubinius // machine/class/variable_scope.cpp VariableScope* VariableScope::of_sender(STATE) { if(CallFrame* frame = state->vm()->get_ruby_frame(1)) { return frame->promote_scope(state); } return nil<VariableScope>(); }
45. Rubinius // machine/call_frame.hpp VariableScope* promote_scope(STATE) { if(VariableScope* vs = scope->on_heap()) return vs; return promote_scope_full(state); }
46. Rubinius // machine/call_frame.cpp VariableScope* CallFrame::promote_scope_full(STATE) { return scope->create_heap_alias(state, this, !has_closed_scope_p()); }
47. // machine/stack_variable.cpp VariableScope* StackVariables::create_heap_alias(STATE, CallFrame* call_frame, bool full) { if(on_heap_) return on_heap_; (...)
48. VariableScope* scope = state->memory()->new_object<VariableScope>(state, G(variable_scope)); scope->self(state, self_); scope->block(state, block_); scope->module(state, module_); scope->method(state, call_frame->compiled_code); scope->heap_locals(state, nil<Tuple>()); scope->last_match(state, last_match_); scope->fiber(state, state->vm()->thread()- >current_fiber());
49. (...) if(!full) { scope->isolated(1); scope->heap_locals(state, Tuple::create(state, mcode->number_of_locals)); for(int i = 0; i < scope- >number_of_locals(); i++) { scope->set_local(state, i, locals_[i]); } }
50. scope->locals(locals_); scope->dynamic_locals(state, nil<LookupTable>()); on_heap_ = scope; return scope; }
51. Calling binding method results in copying variables from stackframe to heap
52. Local variables became persistent!
53. How about jRuby?
54. Binding vs RubyBinding
55. Binding /** * Internal live representation of a block ({...} or do ... end). */ public class Binding { ...
56. RubyBinding @JRubyClass(name="Binding") public class RubyBinding extends RubyObject { private Binding binding; ... }
57. jRuby // RubyKernel.java @JRubyMethod(name = "binding", ...) public static RubyBinding binding19(ThreadContext context, IRubyObject recv, Block block) { return RubyBinding.newBinding(context.runtime, context.currentBinding()); }
58. // RubyBinding.java public static RubyBinding newBinding(Ruby runtime, Binding binding) { return new RubyBinding(runtime, runtime.getBinding(), binding); } public RubyBinding(Ruby runtime, RubyClass rubyClass, Binding binding) { super(runtime, rubyClass); this.binding = binding; }
59. //RubyGlobal.java public static void createGlobals(ThreadContext context, Ruby runtime) { GlobalVariables globals = runtime.getGlobalVariables(); runtime.defineGlobalConstant("TOPLEVEL_BINDING", runtime.newBinding()); (...) }
60. //Block.java public Block(BlockBody body, Binding binding) { assert binding != null; this.body = body; this.binding = binding; }
61. //RubyEnumerable.java public static IRubyObject each(ThreadContext context, IRubyObject self, BlockBody body) { Block block = new Block(body, context.currentBinding(self, Visibility.PUBLIC)); (...) }
62. Let's go back to MRI world…
63. def get_binding a = 2 b = 3 binding end get_binding.local_variables # => [:a, :b] get_binding.local_variable_get(:a) # => 2
64. But be careful!
65. def test_lambda a = 10 l = lambda { a } a = "OH HAI" l end test_lambda.call #=> "OH HAI"
66. YARV internally copies all variables from stackframe to heap, then use them as new stackframe
67. a = 0 add = lambda { a += 1 } sub = lambda { a -= 1 } add.call sub.call add.call sub.call add.call a # => ?
68. a = 0 add = lambda { a += 1 } sub = lambda { a -= 1 } add.call sub.call add.call sub.call add.call a # => 1
69. All lamdbas are actually operating on same copy of stackframe
70. So – can we change binding we're actually working in?
71. Almost
72. def test_metody set = lambda { new_var = 10 } set.call new_var end test_metody # => NameError: undefined local variable or method `new_var' for main:Object
73. We can't do this because in original stackframe this variable wasn't present. Therefore it isn't present in current lexical scope. It is created in other lexical scope – scope of lambda.
74. And what if we call it from inside an object?
75. class BindingTest def initialize @a = 1 end def get_binding binding end end b = BindingTest.new.get_binding b.receiver # => #<BindingTest:0x007ff5fa285c10 @a=1>
76. What else can we do with binding?
77. def get_binding a = 2 b = 3 binding end eval('a + b', get_binding) # => 5
78. We can also change binding
79. def get_binding a = 2 b = 3 binding end b = get_binding eval('c = a + b', b) b.local_variable_get(:c) # => 5
80. And since binding is an object and it gives access to all local variables…
81. # lib/pry/core_extensions.rb class Object # Start a Pry REPL on self. (...) def pry(object=nil, hash={}) if object.nil? || Hash === object Pry.start(self, object || {}) else Pry.start(object, hash) end end (...) end
82. Local variables are also quite convenient in templates
83. require 'erb' template =<<EOF <html> <%= variable %> </html> EOF variable = 10 erb = ERB.new(template) erb.result NameError: undefined local variable or method 'variable' for main:Object (...) b = binding erb.result(b)
84. <html> 10 </html>
85. # erb.rb:856 def result(b=new_toplevel) if @safe_level proc { $SAFE = @safe_level eval(@src, b, (@filename || '(erb)'), @lineno) }.call else eval(@src, b, (@filename || '(erb)'), @lineno) end end Source:
87. class Cat def to_s a = 1 b = 2 puts "CAT" end end
88. bindings = [] trace = TracePoint.new(:return) do |tp| puts tp.path puts tp.lineno bindings << tp.binding end trace.enable c = Cat.new c.to_s trace.disable c = nil
89. p bindings.first.local_variables # [:a, :b] p bindings.first.local_variable_get(:a) # 1 p bindings.first.local_variable_get(:b) # 2 p bindings.first.receiver # #<Cat:0x007fdd53854868>
90. require 'pry' class Cat def traced_m a = 1 b = 2 puts "CAT" end end
91. trace = TracePoint.new(:return) do |tp| tp.binding.pry if tp.method_id == :traced_m end trace.enable c = Cat.new c.traced_m
93. We're going to write microcontainer for variables – OpenStruct style (because we can)
94. Common for all examples class Binder def initialize(b) @binding = b end end
95. V1
96. def method_missing(name, *args) if @binding.local_variables.include?(name) @binding.local_variable_get(name) elsif name.to_s[-1] == '=' @binding.local_variable_set(name.to_s.del ete('=').to_sym, args[0]) else super end end
97. How we're going to benchmark it?
98. class GetBinding def self.get_binding a = 10 str = 'a1b2' binding end end
99. b = GetBinding.get_binding hash = { a: 10, str: 'a1b2' } puts 'creation' Benchmark.ips do |x| x.report { namespace::Binder.new(b) } x.report { OpenStruct.new(hash) } x.compare! end
100. binder = namespace::Binder.new(b) struct = OpenStruct.new(hash) puts 'get' Benchmark.ips do |x| x.report { binder.a; binder.str } x.report { struct.a; struct.str } x.compare! end
101. puts 'set' Benchmark.ips do |x| x.report { binder.a = 2; binder.string_2 = 'abc' } x.report { struct.a = 2; struct.string_2 = 'abc' } x.compare! end puts 'set different' Benchmark.ips do |x| x.report { binder.send("var_#{n}=".to_sym, 1) } x.report { struct.send("var_#{n}=".to_sym, 1) } x.compare! end
102. Version V1 creation V1 binder: 2849210.5 i/s OpenStruct: 719580.0 i/s - 3.96x slower get OpenStruct: 3243263.4 i/s V1 binder: 502957.6 i/s - 6.45x slower set OpenStruct: 1390626.0 i/s V1 binder: 172057.9 i/s - 8.08x slower set different OpenStruct: 1023580.8 i/s V1 binder: 265165.3 i/s - 3.86x slower
103. V2
104. def initialize(b) @binding = b @binding.local_variables.each do |var| l = -> { @binding.local_variable_get(var) } define_singleton_method(var, l) end end
105. Version V2 creation OpenStruct: 691431.3 i/s V2 binder: 135494.8 i/s - 5.10x slower get OpenStruct: 3242890.9 i/s V2 binder: 2768257.5 i/s - same-ish: difference falls within error set OpenStruct: 1708887.9 i/s V2 binder: 204508.0 i/s - 8.36x slower set different OpenStruct: 1173427.4 i/s V2 binder: 298250.5 i/s - 3.93x slower
106. V3
107. def initialize(b) @binding = b @binding.local_variables.each do |name| get = -> { @binding.local_variable_get(name) } set = ->(new_var) { @binding.local_variable_set(name, new_var) } define_singleton_method(name, get) define_singleton_method("#{name}=".to_sym, set) end end
108. Version V3 creation OpenStruct: 724209.0 i/s V3 binder: 96906.5 i/s - 7.47x slower get OpenStruct: 3298616.7 i/s V3 binder: 2970314.4 i/s - same-ish: difference falls within error set OpenStruct: 1609587.6 i/s V3 binder: 398771.9 i/s - 4.04x slower set different OpenStruct: 985401.4 i/s V3 binder: 273593.3 i/s - 3.60x slower
109. V4
110. def method_missing(name, *args) if @binding.local_variables.include?(name) @binding.local_variable_get(name) elsif name.to_s[-1] == '=' pure_name = name.to_s.delete('=').to_sym @binding.local_variable_set(pure_name, args[0]) unless methods.include?(pure_name) set = ->(new_var) { @binding.local_variable_set(pure_name, new_var) } get = -> { @binding.local_variable_get(pure_name) } define_singleton_method(name, set) define_singleton_method(pure_name, get) end else super end end
111. Version V4 creation OpenStruct: 733813.6 i/s V4 binder: 82442.4 i/s - 8.90x slower get OpenStruct: 3615367.7 i/s V4 binder: 2928936.0 i/s - same-ish difference falls within error set V4 binder: 2637039.0 i/s OpenStruct: 1779210.5 i/s - 1.48x slower set different V4 binder: 1295138.7 i/s OpenStruct: 1172012.4 i/s - same-ish difference falls within error
112. https://github.com/esse/openstruct_in_binding
113. https://github.com/esse/bb_openstruct gem 'bb_openstruct'
115. creation Openstruct: 719494.0 i/s BBOpenstruct: 54072.0 i/s - 13.31x slower get Openstruct: 3548217.5 i/s BBOpenstruct: 3170859.8 i/s - same-ish: difference falls within error set BBOpenstruct: 2470835.7 i/s Openstruct: 1730367.3 i/s - 1.43x slower set different Openstruct: 1163716.8 i/s BBOpenstruct: 1145259.5 i/s - same-ish: difference falls within error
116. … and (almost) full openstruct compatibility!
117. Fun fact
118. Proc and lambda are actually same struct in C (only difference is one flag)
119. Questions?
120. Thanks! Proudly illustrated by Emilia Mucha emilamp@gmail.com