ErRuby - Ruby on Erlang

by Johnlin

Slide 1

Slide 1 text

ErRuby - Ruby on Erlang/OTP +PIO-JO!KPIOMJOWD

Slide 2

Slide 2 text

John Lin • @johnlinvc • From Taiwan • Senior Engineer At iCHEF

Slide 3

Slide 3 text

I ❤ programming languages

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

• It’s Ruby-Like Language running on Erlang VM(BEAM). • It’s good at concurrency, just like Erlang.

Slide 6

Slide 6 text

So I wonder • Do we have a Real® Ruby on the Erlang VM? Just Like JRuby. • It’ll have all the nice thing of Ruby and great concurrency features of Erlang. • Best of both world.

Slide 7

Slide 7 text

Was there a Ruby on Erlang VM?

Slide 8

Slide 8 text

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

ErRuby • A Ruby interpreter written in Erlang, running on the BEAM VM. • https://github.com/johnlinvc/erruby • Goals are: • Implement major features of Ruby on Erlang VM. • experimenting builtin concurrent features. • exploring issues we’ll encounter when we introduce ﬁrst class concurrency to Ruby.

Slide 11

Slide 11 text

Erlang features • Immutable data • Actor Model • Selective message receive • Lightweight process

Slide 12

Slide 12 text

Immutable data • Everything is a constant. • Things like “erlang”.capitalize! is impossible in Erlang • Everything will be the same after you assign value to it.

Slide 13

Slide 13 text

Immutable variable • Each Variable can only get assigned once. • Erlang will raise error if an variable get assigned more than once. • The value of variable is deﬁned at one unique place. Which make debugging much easier.

Slide 14

Slide 14 text

Actor model • A concurrent computation model based on Actor and message passing. • Actors shares nothing with each others, only communicate with each other by sending and receiving messages.

Slide 15

Slide 15 text

Actor • Actor is the minimum computation unit. • Each of them has it’s own brain and storage. • Actors can only communicate with each others by sending and receiving messages. • Each actor have one or more addresses to receive messages.

Slide 16

Slide 16 text

Email system [email protected] [email protected] [email protected]

Slide 17

Slide 17 text

When receiving message • Actor can send messages to some Actors. • Actor can create some more Actors. • Actor can change it’s internal state.

Slide 18

Slide 18 text

Actor can send messages [email protected] RubyKaigi is  Awesome

Slide 19

Slide 19 text

Actor can create actors [email protected] [email protected] [email protected] [email protected]

Slide 20

Slide 20 text

Actor can change its state [email protected] Join List

Slide 21

Slide 21 text

Selective receive • Actor can choose what they want to process when waiting to receive. • Like me waiting for the CFP result mail from RubyKaigi, I ignore every other mail.

Slide 22

Slide 22 text

Selective receive [email protected] junkmail [email protected]

Slide 23

Slide 23 text

Light weight process • In Erlang each Actor is an independent Process. • Actors are the fundamental of Erlang. Just like Object in Ruby. • So spawning new processes need to be effective. • Erlang process are actually M to N green threads that shares nothing with each others.

Slide 24

Slide 24 text

Erlang is good at concurrency • Immutable data prevents data racing problems • Actor model provides great horizontal scaling ability • Lightweight Processes supports the whole concurrent model.

Slide 25

Slide 25 text

Implementing ErRuby  in Erlang

Slide 26

Slide 26 text

What should a Ruby interpreter do? • Parse source code • Manage local variables • Create Ruby objects • Manage instance variable • Deﬁne method • Invoke method

Slide 27

Slide 27 text

What should a Ruby interpreter do? • Parse source code • Manage local variables • Create Ruby objects • Manage instance variable • Deﬁne method • Invoke method

Slide 28

Slide 28 text

Parser • Ruby is a complex language. It’s very hard to parse it correctly. • Build upon the Parser Gem. • made some minor tweaks for new concurrent features.

Slide 29

Slide 29 text

What should a Ruby interpreter do? • Parse source code • Manage local variables • Create Ruby objects • Manage instance variable • Deﬁne method • Invoke method

Slide 30

Slide 30 text

deﬁne local var greeting = "hello world" greeting

Slide 31

Slide 31 text

Abstract syntax tree begin lvasgn :greeting str “hello world” lvar :greeting

Slide 32

Slide 32 text

Conventional stack begin lvasgn :greeting str “hello world” lvar :greeting stack_ptr

Slide 33

Slide 33 text

Conventional stack begin lvasgn :greeting str “hello world” lvar :greeting stack_ptr greeting

Slide 34

Slide 34 text

Creating mutable realm in Erlang • Erlang is an immutable language • We couldn’t have a mutable variable for the stack • How can we make an mutable world in an immutable universe?

Slide 35

Slide 35 text

Continuation passing • The whole ruby running stack is represented by a single environment dictionary. • The dictionary is passed as an argument when evaluating an AST Node. • Evaluating an AST Node will return a new dictionary, the new dictionary will be an argument of next evaluation.

Slide 36

Slide 36 text

Continuation passing AST1 AST2 AST3 Env1 = Eval(AST1, Env0)

Slide 37

Slide 37 text

Continuation passing AST1 AST2 AST3 Env1 = Eval(AST1, Env0) Env2 = Eval(AST2, Env1)

Slide 38

Slide 38 text

Continuation passing AST1 AST2 AST3 Env1 = Eval(AST1, Env0) Env2 = Eval(AST2, Env1) Env3 = Eval(AST3, Env2)

Slide 39

Slide 39 text

Contents of environment dictionary • Dictionary of local variable’s name to reference • Reference to the self object of current scope • Last expression’s return value • Content of last frame.

Slide 40

Slide 40 text

ErRuby stack begin lvasgn :greeting str “hello world” lvar :greeting eval_ast(begin,  {  “lvars” => {},  “self” => ,  “ret_val” => nil  }  )

Slide 41

Slide 41 text

begin lvasgn :greeting str “hello world” lvar :greeting eval_ast(lvasgn,  {  “lvars” => {},  “self” => ,  “ret_val” => nil  }  ) ErRuby stack

Slide 42

Slide 42 text

begin lvasgn :greeting str “hello world” lvar :greeting eval_ast(lvar,  {  “lvars” => {  “greeting” =>  “hello world”  },  “self” => ,  “ret_val” => nil  }  ) ErRuby stack

Slide 43

Slide 43 text

begin lvasgn :greeting str “hello world” lvar :greeting ErRuby stack {  “lvars” => {  “greeting” =>  “hello world”  },  “self” => ,  “ret_val” =>   “hello world”  }

Slide 44

Slide 44 text

What should a Ruby interpreter do? • Parse source code • Manage local variables • Create Ruby objects • Manage instance variable • Deﬁne method • Invoke method

Slide 45

Slide 45 text

Ruby object • Also inﬂuenced by Smalltalk • In Ruby, method call actually calls send under the hood. • The major difference from Actor is that it must get response before proceeding.

Slide 46

Slide 46 text

Making Ruby objects with Erlang process • Every Ruby Object is a distinct Erlang actor. • All object operations are done by message passing instead of direct memory access. • Update object state after receive message

Slide 47

Slide 47 text

Object process Waiting for message Update State {Sender, Type, Message} Handle message Call Stack Object Handle object related AST Waiting for response {Message, Result} Handle next entry

Slide 48

Slide 48 text

Waiting for message Update State {Sender, sync, Message} Handle message Call Stack Object Handle object related AST Waiting for response {Message, Result} Handle next entry Object process

Slide 49

Slide 49 text

Waiting for message Update State {Sender, async, Message} Handle message Call Stack Object Handle object related AST Waiting for response Handle next entry Object process

Slide 50

Slide 50 text

What’s inside object state? • Instance variables’s name and reference. • Constants’s name and reference • Reference to it’s class • Reference to self

Slide 51

Slide 51 text

What should a Ruby interpreter do? • Parse source code • Manage local variables • Create Ruby objects • Manage instance variable • Deﬁne method • Invoke method

Slide 52

Slide 52 text

Setting/Getting instance variable @conf = "rubykaigi" @conf

Slide 53

Slide 53 text

@conf = "rubykaigi" Waiting for message Update State Handle message Call Stack Object Handle object related AST Waiting for response Handle next entry {ivar => {}}

Slide 54

Slide 54 text

Waiting for message Update State {CallStackID, async, {def_ivar, “conf”, “rubykaigi”}} Handle message Call Stack Object Handle object related AST Waiting for response Handle next entry @conf = "rubykaigi" {ivar => {}}

Slide 55

Slide 55 text

Slide 56

Slide 56 text

@conf Waiting for message Update State Handle message Call Stack Object Handle object related AST Waiting for response Handle next entry {ivars => {“conf”=>  “rubykaigi”}}

Slide 57

Slide 57 text

Waiting for message Update State {CallStackID, sync, {find_ivar, “conf”}} Handle message Call Stack Object Handle object related AST Waiting for response Handle next entry {ivars => {“conf”=>  “rubykaigi”}} @conf

Slide 58

Slide 58 text

Waiting for message Update State {CallStackID, sync, {find_ivar, “conf”}} Handle message Call Stack Object Handle object related AST Waiting for response Handle next entry {{find_ivar, “conf”}, “rubykaigi”} {ivars => {“conf”=>  “rubykaigi”}} @conf

Slide 59

Slide 59 text

What should a Ruby interpreter do? • Parse source code • Manage local variables • Create Ruby objects • Manage instance variable • Deﬁne method • Invoke method

Slide 60

Slide 60 text

Deﬁne Method class Conf def attend(rubyist) "hello #{rubyist}" end end

Slide 61

Slide 61 text

Deﬁne Method Waiting for message Update State Handle message Call Stack Conf Handle object related AST Waiting for response Handle next entry {methods => {}}

Slide 62

Slide 62 text

Deﬁne Method Waiting for message Update State Handle message Call Stack Conf Handle object related AST Waiting for response Handle next entry {CallStackID, async, {def_method, “attend”,  [“rubyist”], BodyAST  }} {methods => {}}

Slide 63

Slide 63 text

Slide 64

Slide 64 text

What should a Ruby interpreter do? • Parse source code • Manage local variables • Create Ruby objects • Manage instance variable • Deﬁne method • Invoke method

Slide 65

Slide 65 text

Execute Method rubykaigi_2016 = Conf.new rubykaigi_2016.attend("john lin")

Slide 66

Slide 66 text

Waiting for message Update State Handle message Call Stack rubykaigi_2016 Handle send Waiting for response Handle next entry Find method deﬁnition {class => Conf}

Slide 67

Slide 67 text

Waiting for message Update State {CallStackID, sync, {find_instance_ method, “attend”}} Handle message Call Stack rubykaigi_2016 Handle send Waiting for response Handle next entry Find method deﬁnition {class => Conf}

Slide 68

Slide 68 text

Waiting for message Update State {CallStackID, sync, {find_instance_ method, “attend”}} Handle message rubykaigi_2016 {class => Conf} Waiting for message Update State Handle message Conf {methods => {attend => Definition}} Find method deﬁnition

Slide 69

Slide 69 text

Waiting for message Update State {CallStackID, sync, {find_instance_ method, “attend”}} Handle message rubykaigi_2016 {class => Conf} Waiting for message Update State {ObjectID, sync, {find_method, “attend”}} Handle message Conf {methods => {attend => Definition}} Find method deﬁnition

Slide 70

Slide 70 text

Slide 71

Slide 71 text

Waiting for message Update State {CallStackID, sync, {find_instance_ method, “attend”}} Handle message Call Stack rubykaigi_2016 Handle send Waiting for response Eval method {class => Conf} {ObjectID, sync, {find_method, “attend”}} {{find_method, “attend”}, Definition} Find method deﬁnition {{find_instance _method, “attend”}, Definition}

Slide 72

Slide 72 text

Eval argument Push a new frame Bind argument Eval Body Pop frame {lvars => {}, self => Toplevel} rubykaigi_2016.attend("john lin")

Slide 73

Slide 73 text

Eval argument Push a new frame Bind argument Eval Body Pop frame {lvars => {}, self => Toplevel} rubykaigi_2016.attend("john lin")

Slide 74

Slide 74 text

rubykaigi_2016.attend("john lin") Eval argument Push a new frame Bind argument Eval Body Pop frame {lvars => {}, self => rubykaigi_2016, prev_frame => {lvars => {},self => Toplevel} }

Slide 75

Slide 75 text

rubykaigi_2016.attend("john lin") Eval argument Push a new frame Bind argument Eval Body Pop frame {lvars => {rubyist => “john lin”}, self => rubykaigi_2016, prev_frame =>   {lvars => {},self => Toplevel} }

Slide 76

Slide 76 text

rubykaigi_2016.attend("john lin") Eval argument Push a new frame Bind argument Eval Body Pop frame {lvars => {rubyist => “john lin”}, self => rubykaigi_2016,  ret_val =>   “hello john lin” prev_frame =>   {lvars => {},self => Toplevel} }

Slide 77

Slide 77 text

rubykaigi_2016.attend("john lin") Eval argument Push a new frame Bind argument Eval Body Pop frame {lvars => {}, self => Toplevel, ret_val =>   “hello john lin”}

Slide 78

Slide 78 text

What should a Ruby interpreter do? • Parse source code • Manage local variables • Create Ruby objects • Manage instance variable • Deﬁne method • Invoke method

Slide 79

Slide 79 text

Current features • local/instance variables • method deﬁnition and calling • class and inheritance • block & yield • Boolean, Integer, String, Array with limited methods.

Slide 80

Slide 80 text

Experimental features  ⚗

Slide 81

Slide 81 text

First class concurrency • One of the main goals of ErRuby. • Which should make programmers happy, not the machine. • Principle of least astonishment

Slide 82

Slide 82 text

foo|.bar • No impact on existing code. • Inspired by actor model & gorouting • The symbol itself feels pretty concurrent. • | (or) is a good friend of & (and)

Slide 83

Slide 83 text

Example def upload_file(payload) #upload file to a remote server end def send_heartbeat #send heartbeat to client every minute end response = self|.upload_file(kyoto_photos) self|.send_heartbeat #do other stuffs puts(response.code)

Slide 84

Slide 84 text

Features • Asynchronous method calling. • Returns a Future object, which wraps the return value of the method call. • Only blocks when we actually need the result.

Slide 85

Slide 85 text

Under the hood • Creates a new process for the method invocation. • Returns a Future object immediately . • Sends back the return value after it’s done. • Blocks and waits for the result when getting data from the Future object.

Slide 86

Slide 86 text

Future • A proxy object represent the result of a computation event. • Can be passed around just like normal objects. • Blocks and wait for the result when accessing the value.

Slide 87

Slide 87 text

Blocks if the result isn’t ready future = foo|.bar access actual  value of future Running bar concurrently blocks

Slide 88

Slide 88 text

Doesn’t block if the result is ready future = foo|.bar access actual  value of future Running bar concurrently

Slide 89

Slide 89 text

When do we need the value? • The later the better. • If it blocks right away, we’re not having any concurrency at all. • Some operations don’t need the actual value.

Slide 90

Slide 90 text

foo = future • We don’t actually need the value when assign it to a variable. • A variable is just a reference of the object. We can just reference the future instead of the object

Slide 91

Slide 91 text

future.to_s • We need the actual value to ﬁnd out it’s class • we should blocks when calling method on Future

Slide 92

Slide 92 text

puts “hello” if future • We’ll need the actual value of the Future when use it in language constructs • We’ll block in this case

Slide 93

Slide 93 text

Passing Future as argument • There’s 2 different type of methods to consider • User deﬁned method • Language builtin method

Slide 94

Slide 94 text

kaigi.attend(future) • Inside user method, we can do one of the following things • assign a it to a variable • call method on it • pass it as argument of system method. • there’s no need to block in this case.

Slide 95

Slide 95 text

puts(future) • If it’s a language method, we’ll block. • We need the actual value to do actual works.

Slide 96

Slide 96 text

What foo|.bar doesn’t solve • Data sharing problem • Racing conditions

Slide 97

Slide 97 text

Future Challenges • Find a way to handle data sharing problem between processes. • Garbage collection • Exceptions

Slide 98

Slide 98 text

Takeaways • ErRuby is an experimental Ruby interpreter on Erlang. • It’s possible to implement Ruby in a Functional language • foo|.bar and call by future ﬁts into Ruby.

Slide 99

Slide 99 text

Thank You For Listening

Slide 100

Slide 100 text

Q&A • Erruby is at https://github.com/johnlinvc/erruby • My Twitter handle is @johnlinvc