ErRuby - Ruby on Erlang

Talk at Rubykaigi 2016.

Transcript

1. ErRuby - Ruby on Erlang/OTP +PIO
   -JO
   !KPIOMJOWD
   
2. John Lin • @johnlinvc • From Taiwan • Senior Engineer At iCHEF
3. I ❤ programming languages
5. • It’s Ruby-Like Language running on Erlang VM(BEAM). • It’s good at concurrency, just like Erlang.
6. So I wonder • Do we have a Real® Ruby on the Erlang VM? Just Like JRuby. • It’ll have all the nice thing of Ruby and great concurrency features of Erlang. • Best of both world.
7. Was there a Ruby on Erlang VM?
8. No
10. ErRuby • A Ruby interpreter written in Erlang, running on the BEAM VM. • https://github.com/johnlinvc/erruby • Goals are: • Implement major features of Ruby on Erlang VM. • experimenting builtin concurrent features. • exploring issues we’ll encounter when we introduce first class concurrency to Ruby.
11. Erlang features • Immutable data • Actor Model • Selective message receive • Lightweight process
12. Immutable data • Everything is a constant. • Things like “erlang”.capitalize! is impossible in Erlang • Everything will be the same after you assign value to it.
13. Immutable variable • Each Variable can only get assigned once. • Erlang will raise error if an variable get assigned more than once. • The value of variable is defined at one unique place. Which make debugging much easier.
14. Actor model • A concurrent computation model based on Actor and message passing. • Actors shares nothing with each others, only communicate with each other by sending and receiving messages.
15. Actor • Actor is the minimum computation unit. • Each of them has it’s own brain and storage. • Actors can only communicate with each others by sending and receiving messages. • Each actor have one or more addresses to receive messages.
16. Email system a@foo.com b@baz.com m@maz.com
17. When receiving message • Actor can send messages to some Actors. • Actor can create some more Actors. • Actor can change it’s internal state.
18. Actor can send messages ruby-core@ruby-lang.org RubyKaigi is
 Awesome
19. Actor can create actors a@foo.com b@foo.com c@foo.com d@foo.com
20. Actor can change its state ruby-core-join@ruby-lang.org Join List
21. Selective receive • Actor can choose what they want to process when waiting to receive. • Like me waiting for the CFP result mail from RubyKaigi, I ignore every other mail.
22. Selective receive ruby-core@ruby-lang.org junkmail junk@non.member
23. Light weight process • In Erlang each Actor is an independent Process. • Actors are the fundamental of Erlang. Just like Object in Ruby. • So spawning new processes need to be effective. • Erlang process are actually M to N green threads that shares nothing with each others.
24. Erlang is good at concurrency • Immutable data prevents data racing problems • Actor model provides great horizontal scaling ability • Lightweight Processes supports the whole concurrent model.
25. Implementing ErRuby
 in Erlang
26. What should a Ruby interpreter do? • Parse source code • Manage local variables • Create Ruby objects • Manage instance variable • Define method • Invoke method
27. What should a Ruby interpreter do? • Parse source code • Manage local variables • Create Ruby objects • Manage instance variable • Define method • Invoke method
28. Parser • Ruby is a complex language. It’s very hard to parse it correctly. • Build upon the Parser Gem. • made some minor tweaks for new concurrent features.
29. What should a Ruby interpreter do? • Parse source code • Manage local variables • Create Ruby objects • Manage instance variable • Define method • Invoke method
30. define local var greeting = "hello world" greeting
31. Abstract syntax tree begin lvasgn :greeting str “hello world” lvar :greeting
32. Conventional stack begin lvasgn :greeting str “hello world” lvar :greeting stack_ptr
33. Conventional stack begin lvasgn :greeting str “hello world” lvar :greeting stack_ptr greeting
34. Creating mutable realm in Erlang • Erlang is an immutable language • We couldn’t have a mutable variable for the stack • How can we make an mutable world in an immutable universe?
35. Continuation passing • The whole ruby running stack is represented by a single environment dictionary. • The dictionary is passed as an argument when evaluating an AST Node. • Evaluating an AST Node will return a new dictionary, the new dictionary will be an argument of next evaluation.
36. Continuation passing AST1 AST2 AST3 Env1 = Eval(AST1, Env0)
37. Continuation passing AST1 AST2 AST3 Env1 = Eval(AST1, Env0) Env2 = Eval(AST2, Env1)
38. Continuation passing AST1 AST2 AST3 Env1 = Eval(AST1, Env0) Env2 = Eval(AST2, Env1) Env3 = Eval(AST3, Env2)
39. Contents of environment dictionary • Dictionary of local variable’s name to reference • Reference to the self object of current scope • Last expression’s return value • Content of last frame.
40. ErRuby stack begin lvasgn :greeting str “hello world” lvar :greeting eval_ast(begin,
 {
 “lvars” => {},
 “self” => <ref>,
 “ret_val” => nil
 }
 )
41. begin lvasgn :greeting str “hello world” lvar :greeting eval_ast(lvasgn,
 {
 “lvars” => {},
 “self” => <ref>,
 “ret_val” => nil
 }
 ) ErRuby stack
42. begin lvasgn :greeting str “hello world” lvar :greeting eval_ast(lvar,
 {
 “lvars” => {
 “greeting” =>
 “hello world”
 },
 “self” => <ref>,
 “ret_val” => nil
 }
 ) ErRuby stack
43. begin lvasgn :greeting str “hello world” lvar :greeting ErRuby stack {
 “lvars” => {
 “greeting” =>
 “hello world”
 },
 “self” => <ref>,
 “ret_val” => 
 “hello world”
 }
44. What should a Ruby interpreter do? • Parse source code • Manage local variables • Create Ruby objects • Manage instance variable • Define method • Invoke method
45. Ruby object • Also influenced by Smalltalk • In Ruby, method call actually calls send under the hood. • The major difference from Actor is that it must get response before proceeding.
46. Making Ruby objects with Erlang process • Every Ruby Object is a distinct Erlang actor. • All object operations are done by message passing instead of direct memory access. • Update object state after receive message
47. Object process Waiting for message Update State {Sender, Type, Message} Handle message Call Stack Object Handle object related AST Waiting for response {Message, Result} Handle next entry
48. Waiting for message Update State {Sender, sync, Message} Handle message Call Stack Object Handle object related AST Waiting for response {Message, Result} Handle next entry Object process
49. Waiting for message Update State {Sender, async, Message} Handle message Call Stack Object Handle object related AST Waiting for response Handle next entry Object process
50. What’s inside object state? • Instance variables’s name and reference. • Constants’s name and reference • Reference to it’s class • Reference to self
51. What should a Ruby interpreter do? • Parse source code • Manage local variables • Create Ruby objects • Manage instance variable • Define method • Invoke method
52. Setting/Getting instance variable @conf = "rubykaigi" @conf
53. @conf = "rubykaigi" Waiting for message Update State Handle message Call Stack Object Handle object related AST Waiting for response Handle next entry {ivar => {}}
54. Waiting for message Update State {CallStackID, async, {def_ivar, “conf”, “rubykaigi”}} Handle message Call Stack Object Handle object related AST Waiting for response Handle next entry @conf = "rubykaigi" {ivar => {}}
55. Waiting for message Update State {CallStackID, async, {def_ivar, “conf”, “rubykaigi”}} Handle message Call Stack Object Handle object related AST Waiting for response Handle next entry {ivars => {“conf”=>
 “rubykaigi”}} @conf = "rubykaigi"
56. @conf Waiting for message Update State Handle message Call Stack Object Handle object related AST Waiting for response Handle next entry {ivars => {“conf”=>
 “rubykaigi”}}
57. Waiting for message Update State {CallStackID, sync, {find_ivar, “conf”}} Handle message Call Stack Object Handle object related AST Waiting for response Handle next entry {ivars => {“conf”=>
 “rubykaigi”}} @conf
58. Waiting for message Update State {CallStackID, sync, {find_ivar, “conf”}} Handle message Call Stack Object Handle object related AST Waiting for response Handle next entry {{find_ivar, “conf”}, “rubykaigi”} {ivars => {“conf”=>
 “rubykaigi”}} @conf
59. What should a Ruby interpreter do? • Parse source code • Manage local variables • Create Ruby objects • Manage instance variable • Define method • Invoke method
60. Define Method class Conf def attend(rubyist) "hello #{rubyist}" end end
61. Define Method Waiting for message Update State Handle message Call Stack Conf Handle object related AST Waiting for response Handle next entry {methods => {}}
62. Define Method Waiting for message Update State Handle message Call Stack Conf Handle object related AST Waiting for response Handle next entry {CallStackID, async, {def_method, “attend”,
 [“rubyist”], BodyAST
 }} {methods => {}}
63. Define Method Waiting for message Update State Handle message Call Stack Conf Handle object related AST Waiting for response Handle next entry {CallStackID, async, {def_method, “attend”,
 [“rubyist”], BodyAST
 }} {methods => {“attend” => {[“rubyist”], BodyAST, 1}}}
64. What should a Ruby interpreter do? • Parse source code • Manage local variables • Create Ruby objects • Manage instance variable • Define method • Invoke method
65. Execute Method rubykaigi_2016 = Conf.new rubykaigi_2016.attend("john lin")
66. Waiting for message Update State Handle message Call Stack rubykaigi_2016 Handle send Waiting for response Handle next entry Find method definition {class => Conf}
67. Waiting for message Update State {CallStackID, sync, {find_instance_ method, “attend”}} Handle message Call Stack rubykaigi_2016 Handle send Waiting for response Handle next entry Find method definition {class => Conf}
68. Waiting for message Update State {CallStackID, sync, {find_instance_ method, “attend”}} Handle message rubykaigi_2016 {class => Conf} Waiting for message Update State Handle message Conf {methods => {attend => Definition}} Find method definition
69. Waiting for message Update State {CallStackID, sync, {find_instance_ method, “attend”}} Handle message rubykaigi_2016 {class => Conf} Waiting for message Update State {ObjectID, sync, {find_method, “attend”}} Handle message Conf {methods => {attend => Definition}} Find method definition
70. Waiting for message Update State {CallStackID, sync, {find_instance_ method, “attend”}} Handle message rubykaigi_2016 {class => Conf} Waiting for message Update State {ObjectID, sync, {find_method, “attend”}} Handle message Conf {methods => {attend => Definition}} {{find_method, “attend”}, Definition} Find method definition
71. Waiting for message Update State {CallStackID, sync, {find_instance_ method, “attend”}} Handle message Call Stack rubykaigi_2016 Handle send Waiting for response Eval method {class => Conf} {ObjectID, sync, {find_method, “attend”}} {{find_method, “attend”}, Definition} Find method definition {{find_instance _method, “attend”}, Definition}
72. Eval argument Push a new frame Bind argument Eval Body Pop frame {lvars => {}, self => Toplevel} rubykaigi_2016.attend("john lin")
73. Eval argument Push a new frame Bind argument Eval Body Pop frame {lvars => {}, self => Toplevel} rubykaigi_2016.attend("john lin")
74. rubykaigi_2016.attend("john lin") Eval argument Push a new frame Bind argument Eval Body Pop frame {lvars => {}, self => rubykaigi_2016, prev_frame => {lvars => {},self => Toplevel} }
75. rubykaigi_2016.attend("john lin") Eval argument Push a new frame Bind argument Eval Body Pop frame {lvars => {rubyist => “john lin”}, self => rubykaigi_2016, prev_frame => 
 {lvars => {},self => Toplevel} }
76. rubykaigi_2016.attend("john lin") Eval argument Push a new frame Bind argument Eval Body Pop frame {lvars => {rubyist => “john lin”}, self => rubykaigi_2016,
 ret_val => 
 “hello john lin” prev_frame => 
 {lvars => {},self => Toplevel} }
77. rubykaigi_2016.attend("john lin") Eval argument Push a new frame Bind argument Eval Body Pop frame {lvars => {}, self => Toplevel, ret_val => 
 “hello john lin”}
78. What should a Ruby interpreter do? • Parse source code • Manage local variables • Create Ruby objects • Manage instance variable • Define method • Invoke method
79. Current features • local/instance variables • method definition and calling • class and inheritance • block & yield • Boolean, Integer, String, Array with limited methods.
80. Experimental features
 ⚗
81. First class concurrency • One of the main goals of ErRuby. • Which should make programmers happy, not the machine. • Principle of least astonishment
82. foo|.bar • No impact on existing code. • Inspired by actor model & gorouting • The symbol itself feels pretty concurrent. • | (or) is a good friend of & (and)
83. Example def upload_file(payload) #upload file to a remote server end def send_heartbeat #send heartbeat to client every minute end response = self|.upload_file(kyoto_photos) self|.send_heartbeat #do other stuffs puts(response.code)
84. Features • Asynchronous method calling. • Returns a Future object, which wraps the return value of the method call. • Only blocks when we actually need the result.
85. Under the hood • Creates a new process for the method invocation. • Returns a Future object immediately . • Sends back the return value after it’s done. • Blocks and waits for the result when getting data from the Future object.
86. Future • A proxy object represent the result of a computation event. • Can be passed around just like normal objects. • Blocks and wait for the result when accessing the value.
87. Blocks if the result isn’t ready future = foo|.bar access actual
 value of future Running bar concurrently blocks
88. Doesn’t block if the result is ready future = foo|.bar access actual
 value of future Running bar concurrently
89. When do we need the value? • The later the better. • If it blocks right away, we’re not having any concurrency at all. • Some operations don’t need the actual value.
90. foo = future • We don’t actually need the value when assign it to a variable. • A variable is just a reference of the object. We can just reference the future instead of the object
91. future.to_s • We need the actual value to find out it’s class • we should blocks when calling method on Future
92. puts “hello” if future • We’ll need the actual value of the Future when use it in language constructs • We’ll block in this case
93. Passing Future as argument • There’s 2 different type of methods to consider • User defined method • Language builtin method
94. kaigi.attend(future) • Inside user method, we can do one of the following things • assign a it to a variable • call method on it • pass it as argument of system method. • there’s no need to block in this case.
95. puts(future) • If it’s a language method, we’ll block. • We need the actual value to do actual works.
96. What foo|.bar doesn’t solve • Data sharing problem • Racing conditions
97. Future Challenges • Find a way to handle data sharing problem between processes. • Garbage collection • Exceptions
98. Takeaways • ErRuby is an experimental Ruby interpreter on Erlang. • It’s possible to implement Ruby in a Functional language • foo|.bar and call by future fits into Ruby.
99. Thank You For Listening
100. Q&A • Erruby is at https://github.com/johnlinvc/erruby • My Twitter handle is @johnlinvc