Introducing the Jet Programming Language i2y (Yasushi Itoh)

About Me • Software Engineer working at Fuji Xerox Co., Ltd. in Yokohama • i2y on github (https://github.com/i2y) • @i2y_ at twitter (https://twitter.com/i2y_) • Fan of several programming languages • Ruby, Python, Lisp and more… • Creator of some programming languages Creator of Jet Language (https://github.com/i2y/jet) • Jet is dynamic typed concurrent object oriented language runs on top of Erlang VM.

The content of this presentation is irrelevant to Fuji Xerox Co., Ltd.

Acknowledgments

In this presentation, I will explain the specification and implementation etc. of the Jet language.

Background

Distributed processing and multicore utilization are generally • In recent years, the performance improvement of a single CPU core has been stagnant. • Therefore, it is common to deal with CPU load by combining scale-up and scale-out. • Scale-up Parallel processing using multi-core CPU • Scale-out Distributed processing to multiple CPUs (machines)

Problems of utilizing multi-core in CRuby (MRI) • CRuby (MRI) has Global Interpreter Lock (GIL). • To utilize multi-core … • Multi os-processes • large overhead of process creation and interprocess communication • Releasing GIL properly with native library implementation • large number of man-hours for handling in existing libraries

Solution for GIL problem Three solutions (from the standpoint of creator of Ruby like dynamic typed language) 3VCZMJLF MBOHVBHF JNQMFNFOUBUJPO XJUIPVU(*- ɾ(PCZ ɾ&MJYJS ɾ3FJB 3VCZMBOHVBHF JNQMFNFOUBUJPO XJUIPVU(*- ɾ+3VCZ ɾ*SPO3VCZ ɾ&S3VCZ .PEJpDBUJPOPG D3VCZ ɾ3FNPWBMPG(*- ɾ"EEJOHNFDIBOJTNGPS QBSBMMFMQSPDFTTJOH ྫ͑ ͹(VJMEʣ

Solution for GIL problem Three solutions (from the standpoint of creator of Ruby like dynamic typed language) 3VCZMJLF MBOHVBHF JNQMFNFOUBUJPO XJUIPVU(*- ɾ(PCZ ɾ&MJYJS ɾ3FJB 3VCZMBOHVBHF JNQMFNFOUBUJPO XJUIPVU(*- ɾ+3VCZ ɾ*SPO3VCZ ɾ&S3VCZ .PEJpDBUJPOPG D3VCZ ɾ3FNPWBMPG(*- ɾ"EEJOHNFDIBOJTNGPS QBSBMMFMQSPDFTTJOH ྫ͑ ͹(VJMEʣ

Solution for GIL problem Three solutions (from the standpoint of creator of Ruby like dynamic typed language) 3VCZMJLF MBOHVBHF JNQMFNFOUBUJPO XJUIPVU(*- ɾ(PCZ ɾ&MJYJS ɾ3FJB 3VCZMBOHVBHF JNQMFNFOUBUJPO XJUIPVU(*- ɾ+3VCZ ɾ*SPO3VCZ ɾ&S3VCZ .PEJpDBUJPOPG D3VCZ ɾ3FNPWBMPG(*- ɾ"EEJOHNFDIBOJTNGPS QBSBMMFMQSPDFTTJOH ྫ͑ ͹(VJMEʣ The languages work on Erlang VM (BEAM). Erlang VM (BEAM) supports concurrency. => The strategy of getting on the shoulder of the giant

Erlang and BEAM • Erlang is a dynamically typed functional language released by Ericsson as open source in 1998. • Erlang gets concurrent programming and distributed programming easy. • Lightweight processes by Actor Model on Erlang VM (BEAM) • Single assignment and pattern matching • All data structures are immutable data.

• Each actor runs on concurrent • Each actor can send messages to another actors • Each actor can receive messages from other actors • Each actor does not share any data Actor Model Advantages • Increasing the number of actors makes it easy to scale-up using multi- core. • The processing of closing inside an actor is thread-safe (actor-safe?ʣ. • Scale-out can be done easily if messaging between actors is location transparent.

Actor Model on Erlang VM (BEAM) • Erlang's process is lightweight and like an actor on actor model. • Preemptive Scheduler (on multicore) • Each process queues messages from other processes. • Each process has a Process Dictionary. The process dictionary is a mutable Hash. • BEAM's GC is independent in each process. • If a process crashes, it does not affect other processes.

Erlang language and its problem • Erlang language is a language derived from Prolog. • For above reason, the Erlang language has many specifications that are difficult for major language users. • For example, • Variables are capitalized beginnings. • It is necessary to separate sentences (expressions) appropriately using commas, semicolons, and periods.

Solution for GIL problem is also for Erlang Language problem Three solutions (from the standpoint of implementor of Ruby like dynamic typed language processing system) 3VCZMJLF MBOHVBHF JNQMFNFOUBUJPO XJUIPVU(*- ɾ(PCZ ɾ&MJYJS ɾ3FJB 3VCZMBOHVBHF JNQMFNFOUBUJPO XJUIPVU(*- ɾ+3VCZ ɾ*SPO3VCZ ɾ&S3VCZ .PEJpDBUJPOPG D3VCZ ɾ3FNPWBMPG(*- ɾ"EEJOHNFDIBOJTNGPS QBSBMMFMQSPDFTTJOH ྫ͑ ͹(VJMEʣ The languages work on Erlang VM (BEAM). Erlang VM (BEAM) supports concurrent programming. => The strategy of getting on the shoulder of the giant

Classification of languages run on BEAM Object Oriented Programming Functional Programming Compiler Generating BEAM Bytecodes Interpreter AST Interpreter or VM on BEAM

Classification of languages run on BEAM 3FJB &MJYJS &S3VCZ Object Oriented Programming Functional Programming Compiler Generating BEAM Bytecodes Interpreter AST Interpreter or VM on BEAM

3FJB Classification of languages run on BEAM &MJYJS &S3VCZ Object Oriented Programming Functional Programming Compiler Generating BEAM Bytecodes Interpreter AST Interpreter or VM on BEAM

3FJB Classification of languages run on BEAM &MJYJS &S3VCZ Object Oriented Programming Functional Programming Compiler Generating BEAM Bytecodes Interpreter AST Interpreter or VM on BEAM Good ✔ Bad ✖ Neutral Neutral

3FJB Classification of languages run on BEAM &MJYJS &S3VCZ Object Oriented Programming Functional Programming Compiler Generating BEAM Bytecodes Interpreter AST Interpreter or VM on BEAM Good ✔ Bad ✖ Neutral Neutral A class instance is a process. → Is not it inefficient?

3FJB Classification of languages run on BEAM &MJYJS &S3VCZ Object Oriented Programming Functional Programming Compiler Generating BEAM Bytecodes Interpreter AST Interpreter or VM on BEAM Good ✔ Bad ✖ Neutral Neutral Development of Reia has been discontinued.

My Solution Jet = Erlang VM + Syntax like Ruby + OOP So I created a language that looks like Ruby, supporting object oriented programming and concurrency.

Classification of languages run on BEAM &MJYJS &S3VCZ Object Oriented Programming Functional Programming Compiler Generating BEAM Bytecodes Interpreter AST Interpreter or VM on BEAM Bad ✖ Neutral Neutral +FU Perfect? ✔

Specification

Design Policy 1. What can be expressed in Erlang language can be expressed in Jet language. 2. Jet inherits my favorite Ruby’s features. 3. Jet is not Ruby. I want to make Jet simpler than ruby. Jet doesn’t inherits some features from Ruby. 4. Jet has heigh level interface for concurrency.

Design Policy 1. What can be expressed in Erlang language can be expressed in Jet language. 2. Jet inherits my favorite Ruby’s features. 3. Jet is not Ruby. I want to make Jet simpler than ruby. Jet doesn’t inherits some features from Ruby. 4. Jet has heigh level interface for concurrency.

Design Policy and Spec Summary 1. What can be expressed in Erlang language can be expressed in Jet language. • Immutable data • Pattern matching, Function overloading by arity • Lightweight processes shared nothing • Module is just a erlang’s module. • Operator overloading is not supported. • Erlang code can be called from Jet with zero-overhead

1. What can be expressed in Erlang language can be expressed in Jet language. • Immutable data • Pattern matching, Function overloading by arity • Lightweight processes shared nothing • Module is just a erlang’s module. • Operator overloading is not supported. • Erlang code can be called from Jet with zero-overhead Design Policy and Spec Summary

1. What can be expressed in Erlang language can be expressed in Jet language. • Immutable data • Pattern matching, Function overloading by arity • Lightweight processes shared nothing • Module is just a erlang’s module. • Operator overloading is not supported. • Erlang code can be called from Jet with zero-overhead Design Policy and Spec Summary # Map (Ruby’s Hash-like) m = {foo: “bar”} m2 = m.put(:foo, “΄͛΄͛”) m2.get(:foo) # => “΄͛΄͛” m.get(:foo) # => “bar” # List l = [1, 2] l2 = l.add(3) l2 # => [1, 2, 3] l # => [1, 2]

1. What can be expressed in Erlang language can be expressed in Jet language. • Immutable data • Pattern matching, Function overloading by arity • Lightweight processes shared nothing • Module is just a erlang’s module. • Operator overloading is not supported. • Erlang code can be called from Jet with zero-overhead Design Policy and Spec Summary

1. What can be expressed in Erlang language can be expressed in Jet language. • Immutable data • Pattern matching, Function overloading by arity • Lightweight processes shared nothing • Module is just a erlang’s module. • Operator overloading is not supported. • Erlang code can be called from Jet with zero-overhead # Pattern matching [a, b, c] = [1, 2, 3] [a, b, c] # => [1, 2, 3] match [1, 2, 3] case [x, y, z] x + y + z end # => 6 Design Policy and Spec Summary

1. What can be expressed in Erlang language can be expressed in Jet language. • Immutable data • Pattern matching, Function overloading by arity • Lightweight processes shared nothing • Module is just a erlang’s module. • Operator overloading is not supported. • Erlang code can be called from Jet with zero-overhead Design Policy and Spec Summary

1. What can be expressed in Erlang language can be expressed in Jet language. • Immutable data • Pattern matching, Function overloading by arity • Lightweight processes shared nothing • Module is just a erlang’s module. • Operator overloading is not supported. • Erlang code can be called from Jet with zero-overhead Design Policy and Spec Summary # Function(Method) Overloading by arity module Adder def add(a, b) a + b end def add(a, b, c) a + b + c end end

1. What can be expressed in Erlang language can be expressed in Jet language. • Immutable data • Pattern matching, Function overloading by arity • Lightweight processes shared nothing • Module is just a erlang’s module. • Operator overloading is not supported. • Erlang code can be called from Jet with zero-overhead Design Policy and Spec Summary

1. What can be expressed in Erlang language can be expressed in Jet language. • Immutable data • Pattern matching, Function overloading by arity • Lightweight processes shared nothing • Module is just a erlang’s module. • Operator overloading is not supported. • Erlang code can be called from Jet with zero-overhead # Lightweight process def echo(message) receive case name (“Hello ” ++ name).puts end end # Creating a lightweight process p = erlang::spawn(&echo/1, [“Hello”]) # You can omit “erlang::”. p ! “Ruby” # Sending a message # -> Hello Ruby Design Policy and Spec Summary

1. What can be expressed in Erlang language can be expressed in Jet language. • Immutable data • Pattern matching, Function overloading by arity • Lightweight processes shared nothing • Module is just a erlang’s module. • Operator overloading is not supported. • Erlang code can be called from Jet with zero-overhead Design Policy and Spec Summary

1. What can be expressed in Erlang language can be expressed in Jet language. • Immutable data • Pattern matching, Function overloading by arity • Lightweight processes shared nothing • Module is just a erlang’s module. • Operator overloading is not supported. • Erlang code can be called from Jet with zero-overhead # Module is just a erlang’ module… # Single jet file contains single module. Nothing can’t be written outside of a module. module Adder def add(a, b) a + b end def add(a, b, c) a + b + c end end # Nothing can’t be written outside of a module. Design Policy and Spec Summary

1. What can be expressed in Erlang language can be expressed in Jet language. • Immutable data • Pattern matching, Function overloading by arity • Lightweight processes shared nothing • Module is just a erlang’s module. • Operator overloading is not supported. • Erlang code can be called from Jet with zero-overhead. Design Policy and Spec Summary

Design Policy 1. What can be expressed in Erlang language can be expressed in Jet language. 2. Jet inherits my favorite Ruby’s features. 3. Jet is not Ruby. I want to make Jet simpler than ruby. Jet doesn’t inherits some features from Ruby. 4. Jet has heigh level interface for concurrency.

Design Policy and Spec Summary 2. Jet inherits my favorite Ruby’s features. • Syntax looks like Ruby • Ruby like object system (Class Definition, Mixin etc.)

Design Policy and Spec Summary 2. Jet inherits my favorite Ruby’s features. • Syntax looks like Ruby • Ruby like object system (Class Definition, Mixin etc.)

Design Policy and Spec Summary 2. Jet inherits my favorite Ruby’s features. • Syntax looks like Ruby • Ruby like object system (Class Definition, Mixin etc.) As you can see on the previous slide, Jet's syntax is Ruby-like syntax.

Design Policy and Spec Summary 2. Jet inherits my favorite Ruby’s features. • Syntax looks like Ruby • Ruby like object system (Class Definition, Mixin etc.)

Design Policy and Spec Summary 2. Jet inherits my favorite Ruby’s features. • Syntax looks like Ruby • Ruby like object system (Class Definition, Mixin etc.) # Class definition module Examples class MyList
 include Enumerable
 
 def initialize(list)
 {list: list}
 end 
 def each(func)
 lists::foreach(func, @list)
 end end end ml = Examples::MyList.new([1, 2, 3]) ml.each do |item| item + 1 end

Design Policy 1. What can be expressed in Erlang language can be expressed in Jet language. 2. Jet inherits my favorite Ruby’s features. 3. Jet is not Ruby. I want to make Jet simpler than ruby. Jet doesn’t inherits some features from Ruby. 4. Jet has heigh level interface for concurrency.

Design Policy and Spec Summary 3. Jet is not Ruby. I want to make Jet simpler than ruby. Jet doesn’t inherits some features from Ruby • You can omit parentheses enclose arguments when calling a method in Ruby. • But, you can omit the parentheses when only calling a method with zero-arguments in Jet. • Ruby supports for class inheritance. • But, currently, Jet don’t supports for class inheritance.Mixin and class inheritance are for specification and implementation inheritance. Either one may be enough.

Design Policy and Spec Summary • Block is just a closure (anonymous function). • Method calling with block is supported by trailing closure like Swift.

Design Policy and Spec Summary • Block is just a closure (anonymous function). • Method calling with block is supported by trailing closure like Swift. # Class definition module Examples class MyList
 include Enumerable
 
 def initialize(list)
 {list: list}
 end 
 def each(func)
 lists::foreach(func, @list)
 end end end ml = Examples::MyList.new([1, 2, 3]) ml.each do |item| item + 1 end

Design Policy 1. What can be expressed in Erlang language can be expressed in Jet language. 2. Jet inherits my favorite Ruby’s features. 3. Jet is not Ruby. I want to make Jet simpler than ruby. Jet doesn’t inherits some features from Ruby. 4. Jet has heigh level interface for concurrency.

4. Jet has heigh level interface for concurrency. • An instance of Actor Metaclass can create objects like a Celluloid 's Concurrent Object. Actor Metaclass

4. Jet has heigh level interface for concurrency. • An instance of Actor Metaclass can create objects like a Celluloid 's Concurrent Object. Actor Metaclass # Class definition module Demo class DemoActor meta Actor def initialize() put(:x, 3) end def x() @x end end end

4. Jet has heigh level interface for concurrency. • An instance of Actor Metaclass can create objects like a Celluloid 's Concurrent Object. Actor Metaclass # Class definition module Demo class DemoActor meta Actor def initialize() put(:x, 3) end def x() @x end end end obj = Demo::DemoActor.spawn # DemoActor hasn’t “new” method. obj.x # => 3 (synchronous method call) obj2 = obj.async future = obj2.x # Asynchronous method call future.await # => 3 (wait for return value) obj3 = obj.cast obj3.x # => :ok (not wait for return value (one direction messaging))

Specification summary • Module definition (Erlang’s module) • Class definition with mixin (not supports inheritance) • Immutable data • Immutable object (immutable state and method) • Method overloading by arity • Block definition syntax, Method calling with block (by trailing closure like Swift) • Immutable data, Pattern matching • Lightweight processes shared nothing by Actor Metaclass • Erlang code can be called from Jet with zero-overhead

Implementation

Flow of compiling a module Parsing Generating Erlang's AST Erlang’s AST BEAM’s byte code Generating bytecode Erlang’s erl_syntax module Erlang’s compile module def hoge() “hoge” end Lexical analysis def hoge “hoge” ( ) end Method Name Args Body hoge “hoge” empty Jet’s AST Source code Sequence of tokens Scanner produced by leex Parser produced by yecc

Class definition and inner representation of an instance • Er Method definitions in a class definition is defined as a function in an Erlang module. Foo.jet module Foo class Adder def initialize(value) {value: value} end def add(value) @value + value end def add(value1, value2) @value + value1 + value2 end end end Foo.erl -module(Foo). -compile(export_all). Adder() -> % returns Adder class. _Adder_initialize(Self, Value) -> #{value => Value}. _Adder_add(Self, Value) -> {__object__, State, Methods} = Self maps:get(value, State) + Value; _Adder_add(Self, Value1, Value2) -> {__object__, State, Methods} = Self maps:get(value, State) + Value1 + Value2.

Inner representation of a class instance Tag State Method Table A class instance (object) is represented as a triple tuple (fixed length array). Immutable Hash (key = {name of a method, arity}, value=reference of a function) Immutable Hash (key=name of an instance variable, value=value) Tag indicating that it is an instance (object) It may change to record in erlang because Dialyzer may be able to check the type of the element.

1. Jet’s runtime gets instance method table from class instance. A. If it is cached in the process dictionary, Jet’s runtime gets it from the process dictionary. B. Otherwise, Jet’s runtime copies the references of the function defined as the instance method to the method table (Hash), then cache it in the process dictionary. 2. Jet’s runtime creates an instance with the method table and empty status. 3. Jet’s runtime creates an instance with the method table and initial status that is gotten by calling the instance's initialize method. Creation and Initialization of an instance adder = Foo::Adder.new(1) class Adder def initialize(value) {value: value} end def add(value) @value + value end end :__object__ {value: 1} {initialize: ⚫, add: ⚫} :__object__ {} {initialize: ⚫, add: ⚫} {initialize: ⚫, add: ⚫}

Calling methodsʢsynchronous) • The method call basically gets a function reference from the method table and then passes the instance as the first argument, and finally calls it.

Calling methodsʢasynchronous) • An instance of Actor Metaclass can create objects like a Celluloid 's Concurrent Object. • OTP gen_server module is used for lightweight process generation • A method call is converted to a messaging to the lightweight process by ghost method (method_missing). • An asynchronous method call is converted async_call of erlang’s rpc module by ghost method. • A synchronous method call to self is converted to a function call (that is, a normal method call), not a message transmission by ghost method. • When realizing synchronous method invocation to self by sending messages, waiting for call result acquisition will last forever.

Performance

Method calls Function call Synchronous method call One direction method call (Actor) Synchronous method call (Actor) Asynchronous method call (Actor) 0 1000 2000 3000 4000 3,632 2160 1376 168 20 Measuring method • Average of 250 times execution time • The function or method body to be measured only returns :ok. in nanoseconds Evaluation • The difference between function calls and method calls and method calls to method calls and method calls are large. Simple Countermeasure? • Use of the C extension library (NIF) as much as possible • Compiling Jet's Runtime Module with HiPE => As a result of trying this, there was no change :’-(

Misc.

How to connect to Ruby Program • ErlPort • http://erlport.org/ • grpc • https://github.com/Bluehouse-Technology/grpc • and more …

Future Plan • REPL (Read Eval Print Loop) • Supporting Dialyzer • Dialyzer is a static analysis tool that detects type errors. • Dialyzer supports Erlang source code or AST (ASF). • Jet compiler generates Erlang AST. • Therefore I think that it is easy for Jet to support Dialyzer. • Improving the perfection and documentation

Conclusion

Conclusion In this presentation, I introduced the Jet Programming Language’s specification and implementation. I am very happy if there are people who cooperate in development of Jet. You can reflect your preferences in the language specification by participating in its development.

Thank you for listening!

About copyrights and trademarks of others • Copyrights • The background image of the second page was acquired from Wikipedia (Wikimedia). • It’s license is “CC BY-SA 4.0”. (http://creativecommons.org/licenses/by-sa/4.0) • The author is https://commons.wikimedia.org/wiki/User:Kakidai. • The title of the image is “ΈͳͱΈΒ͍21ͷ໷ܠʢ2014೥12݄ʣ" (https://ja.wikipedia.org/wiki/ %E3%83%95%E3%82%A1%E3%82%A4%E3%83%AB:The_night_view_of_Minato_Mirai_21.jpg) • Trademarks • ERLANG is a trademark or registered trademark of Telefonaktiebolaget LM Ericsson. • Other company names and product names mentioned are trademarks or registered trademarks of each company. • About the Keynote template used in this presentation • I customized https://github.com/shoya140/zebra and used it.