Bindings in Ruby - behind the magic of blocks

> Why block can see local variables defined before him? Why can it change them? What kind of sorcery is this?

This is some kind of question that I will try to answer in my talk. We will see examples of block and hidden secret hero behind the magic - `binding` object.

In our magical journey we will look into some Ruby code and some MRI/Rubinius/jRuby internals to help us better understand the magic. We will later use the new knowledge in practice to:

* understand erb
* understand binding.pry
* write our own openstruct-like implementation that stores data internally using local variables inside binding. In benchmark it's faster than openstruct, so yay us! ;)

Great magicians never reveal their secrets - but thankfully Ruby is open source, so we can lift the veil and see the real truth behind it. And believe me, it's quite beautiful!

Transcript

1. Bindings in Ruby
   behind the music magic of blocks
   

   View Slide

2. About me
   Piotr Szmielew
   http://github.com/esse
   http://piotr.szmielew.pl
   @essepl on twitter
   In Rails since 2009
   

   View Slide

3. a = 2
   10.times do
   a += 1
   end
   puts a
   

   View Slide

4. a = 2
   10.times do
   a += 1
   end
   puts a # => 12
   

   View Slide

5. Obvious?
   

   View Slide

6. Now harder:
   

   View Slide

7. class BlockTest
   def self.add_method
   local = 2
   define_method : do
   local
   end
   end
   end
   b = BlockTest.new
   BlockTest.add_method
   b.
   

   View Slide

8. Will it work?
   

   View Slide

9. class BlockTest
   def self.add_method
   local = 2
   define_method : do
   local
   end
   end
   end
   b = BlockTest.new
   BlockTest.add_method
   b. # => 2
   

   View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. Code (lambda)
    environment
    Then - what is a closure?
    

    View Slide

15. p = proc do
    a = 2
    10.times do
    a += 1
    end
    end
    puts RubyVM::InstructionSequence.disasm(p)
    

    View Slide

16. View Slide

17. == disasm: =====
    0004 putobject 2
    0006 setlocal a, 3
    0011 putobject 10
    0013 send irb_binding>
    0017 leave
    == disasm:
    0004 getlocal a, 4
    0007 putobject_OP_INT2FIX_O_1_C_
    0008 opt_plus
    0010 dup
    0011 setlocal a, 4
    

    View Slide

18. == disasm: =====
    0004 putobject 2
    0006 setlocal a, 3
    0011 putobject 10
    0013 send irb_binding>
    0017 leave
    == disasm:
    0004 getlocal a, 4
    0007 putobject_OP_INT2FIX_O_1_C_
    0008 opt_plus
    0010 dup
    0011 setlocal a, 4
    

    View Slide

19. View Slide

20. View Slide

21. Code (lambda)
    environment
    What is a closure?
    

    View Slide

22. YARV
    Virtual machine with stack and heap
    

    View Slide

23. YARV
    Current execution state – contained in stackframe
    Therefore local variables are on stack
    

    View Slide

24. YARV
    Current execution state – contained in stackframe
    Therefore local variables are on stack
    

    View Slide

25. Using lambda / proc
    changes block in normal variable
    (first class citizen)
    

    View Slide

26. BUT
    

    View Slide

27. View Slide

28. View Slide

29. 2.2.4 :048 > binding
    => #
    2.2.4 :049 > binding.class
    => Binding
    2.2.4 :050 > binding.class.superclass
    => Object
    

    View Slide

30. Binding
    Object
    binding
    Instance
    

    View Slide

31. Binding is created
    simultaneously with
    lambda
    

    View Slide

32. How exactly is binding
    created?
    

    View Slide

33. MRI
    // proc.c
    static VALUE
    rb_f_binding(VALUE self)
    {
    return rb_binding_new();
    }
    (responsible for Kernel.binding method)
    

    View Slide

34. MRI
    // proc.c
    VALUE
    rb_binding_new(void)
    {
    rb_thread_t *th = GET_THREAD();
    return rb_vm_make_binding(th, th->cfp);
    }
    

    View Slide

35. Next examples are from vm.c
    

    View Slide

36. VALUE
    rb_vm_make_binding(rb_thread_t *th, const
    rb_control_frame_t *src_cfp)
    {
    (...)
    rb_binding_t *bind;
    (...)
    envval = vm_make_env_object(th, cfp);
    (...)
    vm_bind_update_env(bind, envval);
    }
    

    View Slide

37. static VALUE
    vm_make_env_object(rb_thread_t *th,
    rb_control_frame_t *cfp)
    {
    VALUE envval = vm_make_env_each(th, cfp);
    (...)
    }
    MRI
    

    View Slide

38. vm_make_env_each(rb_thread_t *const th,
    rb_control_frame_t *const cfp)
    {
    (...)
    

    View Slide

39. /*
    * # local variables on a stack frame (N ==
    local_size)
    * [lvar1, lvar2, ..., lvarN, SPECVAL]
    * ^
    * ep[0]
    *
    * # moved local variables
    * [lvar1, lvar2, ..., lvarN, SPECVAL, Envval,
    BlockProcval (if needed)]
    * ^ ^
    * env->env[0] ep[0]
    */
    

    View Slide

40. env_size = local_size +
    1 /* envval */ +
    (blockprocval ? 1 : 0) /* blockprocval */;
    env_body = ALLOC_N(VALUE, env_size);
    MEMCPY(env_body, ep - (local_size - 1 /* specval */),
    VALUE, local_size);
    (...)
    env_ep = &env_body[local_size - 1 /* specval */];
    (...)
    env = vm_env_new(env_ep, env_body, env_size,
    env_iseq);
    (...)
    return (VALUE)env;
    }
    

    View Slide

41. Rubinius
    def binding
    return Binding.setup(
    Rubinius::VariableScope.of_sender,
    Rubinius::CompiledCode.of_sender,
    Rubinius::LexicalScope.of_sender,
    Rubinius::VariableScope.of_sender.self,
    Rubinius::Location.of_closest_ruby_method
    )
    end
    

    View Slide

42. # core / binding.rb
    def self.setup(variables, code, lexical_scope,
    recv=nil, location=nil)
    bind = allocate()
    bind.receiver = self_context(recv, variables)
    bind.variables = variables
    bind.compiled_code = code
    bind.lexical_scope = lexical_scope
    bind.location = location
    return bind
    end
    

    View Slide

43. # core / variable_scope.rb
    module Rubinius
    class VariableScope
    (...)
    def self.of_sender
    Rubinius.primitive :variable_scope_of_sender
    (...)
    end
    (...)
    end
    end
    Rubinius
    

    View Slide

44. Rubinius
    // machine/class/variable_scope.cpp
    VariableScope* VariableScope::of_sender(STATE) {
    if(CallFrame* frame = state->vm()->get_ruby_frame(1)) {
    return frame->promote_scope(state);
    }
    return nil();
    }
    

    View Slide

45. Rubinius
    // machine/call_frame.hpp
    VariableScope* promote_scope(STATE) {
    if(VariableScope* vs = scope->on_heap())
    return vs;
    return promote_scope_full(state);
    }
    

    View Slide

46. Rubinius
    // machine/call_frame.cpp
    VariableScope*
    CallFrame::promote_scope_full(STATE) {
    return scope->create_heap_alias(state, this,
    !has_closed_scope_p());
    }
    

    View Slide

47. // machine/stack_variable.cpp
    VariableScope*
    StackVariables::create_heap_alias(STATE,
    CallFrame* call_frame,
    bool full)
    {
    if(on_heap_) return on_heap_;
    (...)
    

    View Slide

48. VariableScope* scope =
    state->memory()->new_object(state,
    G(variable_scope));
    scope->self(state, self_);
    scope->block(state, block_);
    scope->module(state, module_);
    scope->method(state, call_frame->compiled_code);
    scope->heap_locals(state, nil());
    scope->last_match(state, last_match_);
    scope->fiber(state, state->vm()->thread()-
    >current_fiber());
    

    View Slide

49. (...)
    if(!full) {
    scope->isolated(1);
    scope->heap_locals(state,
    Tuple::create(state, mcode->number_of_locals));
    for(int i = 0; i < scope-
    >number_of_locals(); i++) {
    scope->set_local(state, i, locals_[i]);
    }
    }
    

    View Slide

50. scope->locals(locals_);
    scope->dynamic_locals(state, nil());
    on_heap_ = scope;
    return scope;
    }
    

    View Slide

51. Calling binding method
    results in copying
    variables from
    stackframe to heap
    

    View Slide

52. Local variables became
    persistent!
    

    View Slide

53. How about jRuby?
    

    View Slide

54. Binding vs RubyBinding
    

    View Slide

55. Binding
    /**
    * Internal live representation of a block
    ({...} or do ... end).
    */
    public class Binding {
    ...
    

    View Slide

56. RubyBinding
    @JRubyClass(name="Binding")
    public class RubyBinding extends
    RubyObject {
    private Binding binding;
    ...
    }
    

    View Slide

57. jRuby
    // RubyKernel.java
    @JRubyMethod(name = "binding", ...)
    public static RubyBinding binding19(ThreadContext context,
    IRubyObject recv, Block block) {
    return RubyBinding.newBinding(context.runtime,
    context.currentBinding());
    }
    

    View Slide

58. // RubyBinding.java
    public static RubyBinding newBinding(Ruby runtime,
    Binding binding) {
    return new RubyBinding(runtime,
    runtime.getBinding(), binding);
    }
    public RubyBinding(Ruby runtime, RubyClass rubyClass,
    Binding binding) {
    super(runtime, rubyClass);
    this.binding = binding;
    }
    

    View Slide

59. //RubyGlobal.java
    public static void createGlobals(ThreadContext
    context, Ruby runtime) {
    GlobalVariables globals =
    runtime.getGlobalVariables();
    runtime.defineGlobalConstant("TOPLEVEL_BINDING",
    runtime.newBinding());
    (...)
    }
    

    View Slide

60. //Block.java
    public Block(BlockBody body, Binding binding) {
    assert binding != null;
    this.body = body;
    this.binding = binding;
    }
    

    View Slide

61. //RubyEnumerable.java
    public static IRubyObject each(ThreadContext
    context, IRubyObject self, BlockBody body) {
    Block block = new Block(body,
    context.currentBinding(self,
    Visibility.PUBLIC));
    (...)
    }
    

    View Slide

62. Let's go back to MRI
    world…
    

    View Slide

63. def get_binding
    a = 2
    b = 3
    binding
    end
    get_binding.local_variables # => [:a, :b]
    get_binding.local_variable_get(:a) # => 2
    

    View Slide

64. But be careful!
    

    View Slide

65. def test_lambda
    a = 10
    l = lambda { a }
    a = "OH HAI"
    l
    end
    test_lambda.call #=> "OH HAI"
    

    View Slide

66. YARV internally
    copies all variables
    from stackframe to
    heap, then use
    them as new
    stackframe
    

    View Slide

67. a = 0
    add = lambda { a += 1 }
    sub = lambda { a -= 1 }
    add.call
    sub.call
    add.call
    sub.call
    add.call
    a # => ?
    

    View Slide

68. a = 0
    add = lambda { a += 1 }
    sub = lambda { a -= 1 }
    add.call
    sub.call
    add.call
    sub.call
    add.call
    a # => 1
    

    View Slide

69. All lamdbas are actually
    operating on same copy
    of stackframe
    

    View Slide

70. So – can we change
    binding we're actually
    working in?
    

    View Slide

71. Almost
    

    View Slide

72. def test_metody
    set = lambda { new_var = 10 }
    set.call
    new_var
    end
    test_metody
    # => NameError: undefined local variable or
    method `new_var' for main:Object
    

    View Slide

73. We can't do this because in
    original stackframe this variable
    wasn't present. Therefore it isn't
    present in current lexical scope.
    It is created in other lexical
    scope – scope of lambda.
    

    View Slide

74. And what if we call it from
    inside an object?
    

    View Slide

75. class BindingTest
    def initialize
    @a = 1
    end
    def get_binding
    binding
    end
    end
    b = BindingTest.new.get_binding
    b.receiver
    # => #
    

    View Slide

76. What else can we do with
    binding?
    

    View Slide

77. def get_binding
    a = 2
    b = 3
    binding
    end
    eval('a + b', get_binding) # => 5
    

    View Slide

78. We can also change binding
    

    View Slide

79. def get_binding
    a = 2
    b = 3
    binding
    end
    b = get_binding
    eval('c = a + b', b)
    b.local_variable_get(:c) # => 5
    

    View Slide

80. And since binding is an object and it gives
    access to all local variables…
    

    View Slide

81. # lib/pry/core_extensions.rb
    class Object
    # Start a Pry REPL on self.
    (...)
    def pry(object=nil, hash={})
    if object.nil? || Hash === object
    Pry.start(self, object || {})
    else
    Pry.start(object, hash)
    end
    end
    (...)
    end
    

    View Slide

82. Local variables are also quite convenient
    in templates
    

    View Slide

83. require 'erb'
    template =<
    <%= variable %>
    
    EOF
    variable = 10
    erb = ERB.new(template)
    erb.result
    NameError: undefined local variable or method 'variable'
    for main:Object
    (...)
    b = binding
    erb.result(b)
    

    View Slide

84. View Slide

85. # erb.rb:856
    def result(b=new_toplevel)
    if @safe_level
    proc {
    $SAFE = @safe_level
    eval(@src, b, (@filename || '(erb)'), @lineno)
    }.call
    else
    eval(@src, b, (@filename || '(erb)'), @lineno)
    end
    end
    Source:
    

    View Slide

86. View Slide

87. class Cat
    def to_s
    a = 1
    b = 2
    puts "CAT"
    end
    end
    

    View Slide

88. bindings = []
    trace = TracePoint.new(:return) do |tp|
    puts tp.path
    puts tp.lineno
    bindings << tp.binding
    end
    trace.enable
    c = Cat.new
    c.to_s
    trace.disable
    c = nil
    

    View Slide

89. p bindings.first.local_variables
    # [:a, :b]
    p bindings.first.local_variable_get(:a) # 1
    p bindings.first.local_variable_get(:b) # 2
    p bindings.first.receiver
    # #
    

    View Slide

90. require 'pry'
    class Cat
    def traced_m
    a = 1
    b = 2
    puts "CAT"
    end
    end
    

    View Slide

91. trace = TracePoint.new(:return) do |tp|
    tp.binding.pry if tp.method_id == :traced_m
    end
    trace.enable
    c = Cat.new
    c.traced_m
    

    View Slide

92. View Slide

93. We're going to write
    microcontainer for
    variables –
    OpenStruct style
    (because we can)
    

    View Slide

94. Common for all examples
    class Binder
    def initialize(b)
    @binding = b
    end
    end
    

    View Slide

95. V1
    

    View Slide

96. def method_missing(name, *args)
    if @binding.local_variables.include?(name)
    @binding.local_variable_get(name)
    elsif name.to_s[-1] == '='
    @binding.local_variable_set(name.to_s.del
    ete('=').to_sym, args[0])
    else
    super
    end
    end
    

    View Slide

97. How we're going to benchmark it?
    

    View Slide

98. class GetBinding
    def self.get_binding
    a = 10
    str = 'a1b2'
    binding
    end
    end
    

    View Slide

99. b = GetBinding.get_binding
    hash = { a: 10, str: 'a1b2' }
    puts 'creation'
    Benchmark.ips do |x|
    x.report { namespace::Binder.new(b) }
    x.report { OpenStruct.new(hash) }
    x.compare!
    end
    

    View Slide

100. binder = namespace::Binder.new(b)
     struct = OpenStruct.new(hash)
     puts 'get'
     Benchmark.ips do |x|
     x.report { binder.a; binder.str }
     x.report { struct.a; struct.str }
     x.compare!
     end
     

     View Slide

101. puts 'set'
     Benchmark.ips do |x|
     x.report { binder.a = 2; binder.string_2 = 'abc' }
     x.report { struct.a = 2; struct.string_2 = 'abc' }
     x.compare!
     end
     puts 'set different'
     Benchmark.ips do |x|
     x.report { binder.send("var_#{n}=".to_sym, 1) }
     x.report { struct.send("var_#{n}=".to_sym, 1) }
     x.compare!
     end
     

     View Slide

102. Version V1
     creation
     V1 binder: 2849210.5 i/s
     OpenStruct: 719580.0 i/s - 3.96x slower
     get
     OpenStruct: 3243263.4 i/s
     V1 binder: 502957.6 i/s - 6.45x slower
     set
     OpenStruct: 1390626.0 i/s
     V1 binder: 172057.9 i/s - 8.08x slower
     set different
     OpenStruct: 1023580.8 i/s
     V1 binder: 265165.3 i/s - 3.86x slower
     

     View Slide

103. V2
     

     View Slide

104. def initialize(b)
     @binding = b
     @binding.local_variables.each do |var|
     l = -> { @binding.local_variable_get(var) }
     define_singleton_method(var, l)
     end
     end
     

     View Slide

105. Version V2
     creation
     OpenStruct: 691431.3 i/s
     V2 binder: 135494.8 i/s - 5.10x slower
     get
     OpenStruct: 3242890.9 i/s
     V2 binder: 2768257.5 i/s - same-ish: difference falls within error
     set
     OpenStruct: 1708887.9 i/s
     V2 binder: 204508.0 i/s - 8.36x slower
     set different
     OpenStruct: 1173427.4 i/s
     V2 binder: 298250.5 i/s - 3.93x slower
     

     View Slide

106. V3
     

     View Slide

107. def initialize(b)
     @binding = b
     @binding.local_variables.each do |name|
     get = -> { @binding.local_variable_get(name) }
     set = ->(new_var) { @binding.local_variable_set(name, new_var) }
     define_singleton_method(name, get)
     define_singleton_method("#{name}=".to_sym, set)
     end
     end
     

     View Slide

108. Version V3
     creation
     OpenStruct: 724209.0 i/s
     V3 binder: 96906.5 i/s - 7.47x slower
     get
     OpenStruct: 3298616.7 i/s
     V3 binder: 2970314.4 i/s - same-ish: difference falls within error
     set
     OpenStruct: 1609587.6 i/s
     V3 binder: 398771.9 i/s - 4.04x slower
     set different
     OpenStruct: 985401.4 i/s
     V3 binder: 273593.3 i/s - 3.60x slower
     

     View Slide

109. V4
     

     View Slide

110. def method_missing(name, *args)
     if @binding.local_variables.include?(name)
     @binding.local_variable_get(name)
     elsif name.to_s[-1] == '='
     pure_name = name.to_s.delete('=').to_sym
     @binding.local_variable_set(pure_name, args[0])
     unless methods.include?(pure_name)
     set = ->(new_var) { @binding.local_variable_set(pure_name,
     new_var) }
     get = -> { @binding.local_variable_get(pure_name) }
     define_singleton_method(name, set)
     define_singleton_method(pure_name, get)
     end
     else
     super
     end
     end
     

     View Slide

111. Version V4
     creation
     OpenStruct: 733813.6 i/s
     V4 binder: 82442.4 i/s - 8.90x slower
     get
     OpenStruct: 3615367.7 i/s
     V4 binder: 2928936.0 i/s - same-ish
     difference falls within error
     set
     V4 binder: 2637039.0 i/s
     OpenStruct: 1779210.5 i/s - 1.48x slower
     set different
     V4 binder: 1295138.7 i/s
     OpenStruct: 1172012.4 i/s - same-ish
     difference falls within error
     

     View Slide

112. https://github.com/esse/openstruct_in_binding
     

     View Slide

113. https://github.com/esse/bb_openstruct
     gem 'bb_openstruct'
     

     View Slide

114. View Slide

115. creation
     Openstruct: 719494.0 i/s
     BBOpenstruct: 54072.0 i/s - 13.31x slower
     get
     Openstruct: 3548217.5 i/s
     BBOpenstruct: 3170859.8 i/s - same-ish: difference falls within error
     set
     BBOpenstruct: 2470835.7 i/s
     Openstruct: 1730367.3 i/s - 1.43x slower
     set different
     Openstruct: 1163716.8 i/s
     BBOpenstruct: 1145259.5 i/s - same-ish: difference falls within error
     

     View Slide

116. … and (almost) full openstruct compatibility!
     

     View Slide

117. Fun fact
     

     View Slide

118. Proc and lambda are actually same struct in C
     (only difference is one flag)
     

     View Slide

119. Questions?
     

     View Slide

120. Thanks!
     Proudly illustrated by
     Emilia Mucha
     emilamp@gmail.com
     

     View Slide