Self-hosting Whitespace

by pocke

Slide 1

Slide 1 text

Self-hosting Whitespace 24th, May. 2019 Roppongi.rb#10 https://roppongirb.connpass.com/event/129603/

Slide 2

Slide 2 text

⚠ IT NEEDS RUBY TRUNK (2.7) ⚠

Slide 3

Slide 3 text

Agenda I implemented a self-hosted Whitespace interpreter. I’ll talk about the interpreter.

Slide 4

Slide 4 text

Agenda ● Prerequisite knowledge ○ What is self-hosting? ○ What is Whitespace? ● Self-hosted interpreter ○ How implement the interpreter ○ Implementation Tips

Slide 5

Slide 5 text

What is self-hosting?

Slide 6

Slide 6 text

Self-hosting is... ● Implementing a programming language by itself ● For instance ○ Go compiler is written in Go. (since v1.5) ○ minruby ■ 遠藤侑介 (2017) 『RubyでつくるRuby』ラムダノート.

Slide 7

Slide 7 text

Example: ruby interpreter on ruby interpreter $ ruby a.rb $ ruby interp.rb a.rb $ ruby interp.rb interp.rb a.rb $ ruby interp.rb interp.rb interp.rb interp.rb interp.rb a.rb

Slide 8

Slide 8 text

What is Whitespace?

Slide 9

Slide 9 text

Do you know Whitespace language? ● Never mind if you don’t know! https://twitter.com/p_ck_/status/1123410375804768256 https://twitter.com/p_ck_/status/1123414869737263104

Slide 10

Slide 10 text

Whitespace ● Whitespace is an esoteric programming language (esolang) ○ Esolang example: Brainf**k, Piet, etc ● It consists only of space, tab and newline

Slide 11

Slide 11 text

hello.ws (Space → S, Tab → T, NewLine → N) SSSTSSTSSSNTNSSSSSTTSSTSTNTNSSSSSTTSTTSSNTNSSSS STTSTTSSNTNSSSSSTTSTTTTNTNSSSSSTSSSSSNTNSSSSSTS TSTTTNTNSSSSSTTSTTTTNTNSSSSSTTTSSTSNTNSSSSSTTS TTSSNTNSSSSSTTSSTSSNTNSSSSSSSSTSTSNTNSSNNN Space -> S, Tab -> T Ref: https://magazine.rubyist.net/articles/0022/0022-Le gwork.html

Slide 12

Slide 12 text

Inside of Whitespace ● It is a stack based language ● It has stack and heap for data store ● All value is a signed Integer ● It has goto and subroutine for ﬂow control ● It has 22 commands to manipulate stack and so on ○ All commands are encoded only with whitespace chars

Slide 13

Slide 13 text

Whitespace Commands (1) Stack commands ● stack.push(val) SS ● stack.pop() SNN ● stack.dup() SNS ○ stack.push(stack.last) ● stack.swap() SNT ○ stack[-1], stack[-2] = stack[-2], stack[-1] Heap commands ● heap.save(key, val) TTS ○ val = stack.pop; key = stack.pop heap[key] = val ● heap.load(key) TTT ○ stack.push heap[key.pop()]

Slide 14

Slide 14 text

Whitespace Commands (2) Calculations a = stack.pop; b = stack.pop stack.push(op(b, a)) ● add() TSSS ● sub() TSST ● multi() TSSN ● div() TSTS ● mod() TSTT IO ● write_char() TNSS ● write_number() TNST ○ print stack.pop() ● read_char() TNTS ● read_number() TNTT ○ heap[stack.pop] = read

Slide 15

Slide 15 text

Whitespace Commands (3) Flow ● deﬁne(label) NSS ○ deﬁne_method(label) ● call(label) NST ○ __send__(label) ● end NTN ● exit() NNN ● jump(label) NSN ○ goto label ● jump_if_zero(label) NTS ○ jump(label) if stack.pop == 0 ● jump_if_negative(label) NTT ○ jump(label) if stack.pop < 0

Slide 16

Slide 16 text

Command Example: Stack and IO SSSTSTSTSNTNST stack.push(42) write_number # => 42

Slide 17

Slide 17 text

Command Example: Heap SSSTTNSSSTSTSTSNTTSSSSTTNTTTTNST stack.push(3) stack.push(42) heap.save stack.push(3) heap.load write_number # 3 is address # 42 is a value # Save 42 to addr 3 # Load 42 from addr 3 # => 42

Slide 18

Slide 18 text

Self-hosted Interpreter

Slide 19

Slide 19 text

The interpreter is here ● https://gist.github.com/pocke/2847214a8713 9a0d1babd9338159d667 ● 72,367 lines ● 565,639 bytes

Slide 20

Slide 20 text

DEMO $ gows fizzbuzz.ws $ cat fizzbuzz.ws sep | gows whitespace.ws $ cat whitespace.ws sep fizzbuzz.ws sep | gows whitespace.ws gows is a Whitespace interpreter written in Go. https://github.com/pocke/gows

Slide 21

Slide 21 text

How implement the interpreter

Slide 22

Slide 22 text

Did I handwrote the 70,000+ lines Whitespace code? Deﬁnitely Not.

Slide 23

Slide 23 text

How implement Whitespace in Whitespace Handwriting Whitespace is difﬁcult, so 1. Write a compiler that compiles Ruby to Whitespace 2. Write Whitespace interpreter in Ruby 3. Generate Whitespace from Ruby!

Slide 24

Slide 24 text

1. Write a compiler that compiles Ruby to Whitespace

Slide 25

Slide 25 text

Akaza: A toolchain of Whitespace, written in Ruby ● https://github.com/pocke/akaza ● It has 3 features ○ Whitespace interpreter ○ Whitespace compiler: wsrb ← Today’s theme! ○ Embedded Whitespace in Ruby ■ https://pocke.hatenablog.com/entry/2019/04/28/183509

Slide 26

Slide 26 text

wsrb: Whitespace compiler and the language ● wsrb language is a subset of Ruby ● It supports ○ Local vars, method call and def, Integer, Array, Hash, true, false, nil, if, case, while ● It does not support ○ Most built-in methods, String, class, module, meta programming, and so on

Slide 27

Slide 27 text

wsrb implementation ● https://github.com/pocke/akaza/blob/master/li b/akaza/ruby2ws.rb ● It uses RubyVM::AST (since Ruby 2.6) and pattern matching (since Ruby 2.7) ○ Parse Ruby code to AST with RubyVM::AST ○ Translate the AST to Whitespace with pattern matching

Slide 28

Slide 28 text

2. Write Whitespace interpreter in Ruby

Slide 29

Slide 29 text

Interpreter implementation ● Simple parser and VM ● https://github.com/pocke/self-hosting-whitesp ace/blob/master/src/whitespace.ws.rb

Slide 30

Slide 30 text

3. Generate Whitespace from Ruby

Slide 31

Slide 31 text

Compile $ akaza wsrb src/whitespace.ws.rb DEMO

Slide 32

Slide 32 text

Implementation Tips

Slide 33

Slide 33 text

Value and Class

Slide 34

Slide 34 text

Value with Class ● Wsrb has Integer, Array and Hash classes ● Wsrb needs to distinguish classes ○ e.g. `hash[1]` and `array[1]` should call different method ● So, each value should have class information

Slide 35

Slide 35 text

Value with Class ● Wsrb uses lower 2 bits for types ● For example ○ Value is 42. (42.to_s(2) #=> 101010) ○ Integer class is 0b01 ○ In wsrb, the value is 10101001 see: http://i.loveruby.net/ja/rhg/book/object.html

Slide 36

Slide 36 text

Array

Slide 37

Slide 37 text

Array ● 3 meta data ○ Address of the ﬁrst item ○ Size: Same as Array#size ○ Capacity: Allocated items count (default: 10) ■ size <= capacity ■ If cap is too small, it’s re-allocated ● Items Addr of First Item Size Capacity Item 1 Item 2 Item 3 ...

Slide 38

Slide 38 text

Hash

Slide 39

Slide 39 text

Hash ● Use Hash table and linked list ● Hash table key is `key mod HASH_SIZE` ○ Default HASH_SIZE: 11 ● When conﬂict, it uses linked list

Slide 40

Slide 40 text

Hash Addr of FIrst Item Key1 Value1 Addr to next key Key2 Value2 Addr to next item ...(3*HASH _SIZE) Key1’ Value1’ Addr to next key Key1’’ Value1’’ Addr to next key

Slide 41

Slide 41 text

Differences between Go and Ruby

Slide 42

Slide 42 text

Differences between Go and Ruby ● I found the differences when I was implementing gows ● Division and modulo have differences https://play.golang.org/p/_LGKTmlMAGJ

Slide 43

Slide 43 text

Division difference ● -5 / 3 ○ Ruby: -2 ○ Go: -1 ● Absorb the difference by ○ return int(math.Floor(float64(left) / float64(right)))

Slide 44

Slide 44 text

Modulo differene ● -5 % 3 ○ Ruby: 1 ○ Go: -2 ● Absorb the difference by ○ res := left % right if res < 0 { return j + res } else { return res }

Slide 45

Slide 45 text

Conclusion

Slide 46

Slide 46 text

Conclusion ● We learned how to implement a programing language ● Try implementing Whitespace, or other esolangs! ● Try self-hosting! ● It is not productive, but it is fun! I recommend it

Slide 47

Slide 47 text

pp self ● Masataka “pocke” Kuwabara ● Work for Bit Journey, Inc. / Kibela ● Language: Ruby, Go, TypeScript Whitespace ← New! ● github.com/pocke, twitter.com/p_ck_ Thank you for listening!