Self-hosting Whitespace

https://docs.google.com/presentation/d/1BeNJ_E_KOLjjdM4Bd3u_96tiIGiRdG9J9orhglcxziw/edit?usp=sharing

Transcript

1. Self-hosting Whitespace 24th, May. 2019 Roppongi.rb#10 https://roppongirb.connpass.com/event/129603/
2. ⚠ IT NEEDS RUBY TRUNK (2.7) ⚠
3. Agenda I implemented a self-hosted Whitespace interpreter. I’ll talk about the interpreter.
4. Agenda • Prerequisite knowledge ◦ What is self-hosting? ◦ What is Whitespace? • Self-hosted interpreter ◦ How implement the interpreter ◦ Implementation Tips
5. What is self-hosting?
6. Self-hosting is... • Implementing a programming language by itself • For instance ◦ Go compiler is written in Go. (since v1.5) ◦ minruby ▪ 遠藤侑介 (2017) 『RubyでつくるRuby』ラムダノート.
7. Example: ruby interpreter on ruby interpreter $ ruby a.rb $ ruby interp.rb a.rb $ ruby interp.rb interp.rb a.rb $ ruby interp.rb interp.rb interp.rb interp.rb interp.rb a.rb
8. What is Whitespace?
9. Do you know Whitespace language? • Never mind if you don’t know! https://twitter.com/p_ck_/status/1123410375804768256 https://twitter.com/p_ck_/status/1123414869737263104
10. Whitespace • Whitespace is an esoteric programming language (esolang) ◦ Esolang example: Brainf**k, Piet, etc • It consists only of space, tab and newline
11. hello.ws (Space → S, Tab → T, NewLine → N) SSSTSSTSSSNTNSSSSSTTSSTSTNTNSSSSSTTSTTSSNTNSSSS STTSTTSSNTNSSSSSTTSTTTTNTNSSSSSTSSSSSNTNSSSSSTS TSTTTNTNSSSSSTTSTTTTNTNSSSSSTTTSSTSNTNSSSSSTTS TTSSNTNSSSSSTTSSTSSNTNSSSSSSSSTSTSNTNSSNNN Space -> S, Tab -> T Ref: https://magazine.rubyist.net/articles/0022/0022-Le gwork.html
12. Inside of Whitespace • It is a stack based language • It has stack and heap for data store • All value is a signed Integer • It has goto and subroutine for flow control • It has 22 commands to manipulate stack and so on ◦ All commands are encoded only with whitespace chars
13. Whitespace Commands (1) Stack commands • stack.push(val) SS • stack.pop() SNN • stack.dup() SNS ◦ stack.push(stack.last) • stack.swap() SNT ◦ stack[-1], stack[-2] = stack[-2], stack[-1] Heap commands • heap.save(key, val) TTS ◦ val = stack.pop; key = stack.pop heap[key] = val • heap.load(key) TTT ◦ stack.push heap[key.pop()]
14. Whitespace Commands (2) Calculations a = stack.pop; b = stack.pop stack.push(op(b, a)) • add() TSSS • sub() TSST • multi() TSSN • div() TSTS • mod() TSTT IO • write_char() TNSS • write_number() TNST ◦ print stack.pop() • read_char() TNTS • read_number() TNTT ◦ heap[stack.pop] = read
15. Whitespace Commands (3) Flow • define(label) NSS ◦ define_method(label) • call(label) NST ◦ __send__(label) • end NTN • exit() NNN • jump(label) NSN ◦ goto label • jump_if_zero(label) NTS ◦ jump(label) if stack.pop == 0 • jump_if_negative(label) NTT ◦ jump(label) if stack.pop < 0
16. Command Example: Stack and IO SSSTSTSTSNTNST stack.push(42) write_number # => 42
17. Command Example: Heap SSSTTNSSSTSTSTSNTTSSSSTTNTTTTNST stack.push(3) stack.push(42) heap.save stack.push(3) heap.load write_number # 3 is address # 42 is a value # Save 42 to addr 3 # Load 42 from addr 3 # => 42
18. Self-hosted Interpreter
19. The interpreter is here • https://gist.github.com/pocke/2847214a8713 9a0d1babd9338159d667 • 72,367 lines • 565,639 bytes
20. DEMO $ gows fizzbuzz.ws $ cat fizzbuzz.ws sep | gows whitespace.ws $ cat whitespace.ws sep fizzbuzz.ws sep | gows whitespace.ws gows is a Whitespace interpreter written in Go. https://github.com/pocke/gows
21. How implement the interpreter
22. Did I handwrote the 70,000+ lines Whitespace code? Definitely Not.
23. How implement Whitespace in Whitespace Handwriting Whitespace is difficult, so 1. Write a compiler that compiles Ruby to Whitespace 2. Write Whitespace interpreter in Ruby 3. Generate Whitespace from Ruby!
24. 1. Write a compiler that compiles Ruby to Whitespace
25. Akaza: A toolchain of Whitespace, written in Ruby • https://github.com/pocke/akaza • It has 3 features ◦ Whitespace interpreter ◦ Whitespace compiler: wsrb ← Today’s theme! ◦ Embedded Whitespace in Ruby ▪ https://pocke.hatenablog.com/entry/2019/04/28/183509
26. wsrb: Whitespace compiler and the language • wsrb language is a subset of Ruby • It supports ◦ Local vars, method call and def, Integer, Array, Hash, true, false, nil, if, case, while • It does not support ◦ Most built-in methods, String, class, module, meta programming, and so on
27. wsrb implementation • https://github.com/pocke/akaza/blob/master/li b/akaza/ruby2ws.rb • It uses RubyVM::AST (since Ruby 2.6) and pattern matching (since Ruby 2.7) ◦ Parse Ruby code to AST with RubyVM::AST ◦ Translate the AST to Whitespace with pattern matching
28. 2. Write Whitespace interpreter in Ruby
29. Interpreter implementation • Simple parser and VM • https://github.com/pocke/self-hosting-whitesp ace/blob/master/src/whitespace.ws.rb
30. 3. Generate Whitespace from Ruby
31. Compile $ akaza wsrb src/whitespace.ws.rb DEMO
32. Implementation Tips
33. Value and Class
34. Value with Class • Wsrb has Integer, Array and Hash classes • Wsrb needs to distinguish classes ◦ e.g. `hash[1]` and `array[1]` should call different method • So, each value should have class information
35. Value with Class • Wsrb uses lower 2 bits for types • For example ◦ Value is 42. (42.to_s(2) #=> 101010) ◦ Integer class is 0b01 ◦ In wsrb, the value is 10101001 see: http://i.loveruby.net/ja/rhg/book/object.html
36. Array
37. Array • 3 meta data ◦ Address of the first item ◦ Size: Same as Array#size ◦ Capacity: Allocated items count (default: 10) ▪ size <= capacity ▪ If cap is too small, it’s re-allocated • Items Addr of First Item Size Capacity Item 1 Item 2 Item 3 ...
38. Hash
39. Hash • Use Hash table and linked list • Hash table key is `key mod HASH_SIZE` ◦ Default HASH_SIZE: 11 • When conflict, it uses linked list
40. Hash Addr of FIrst Item Key1 Value1 Addr to next key Key2 Value2 Addr to next item ...(3*HASH _SIZE) Key1’ Value1’ Addr to next key Key1’’ Value1’’ Addr to next key
41. Differences between Go and Ruby
42. Differences between Go and Ruby • I found the differences when I was implementing gows • Division and modulo have differences https://play.golang.org/p/_LGKTmlMAGJ
43. Division difference • -5 / 3 ◦ Ruby: -2 ◦ Go: -1 • Absorb the difference by ◦ return int(math.Floor(float64(left) / float64(right)))
44. Modulo differene • -5 % 3 ◦ Ruby: 1 ◦ Go: -2 • Absorb the difference by ◦ res := left % right if res < 0 { return j + res } else { return res }
45. Conclusion
46. Conclusion • We learned how to implement a programing language • Try implementing Whitespace, or other esolangs! • Try self-hosting! • It is not productive, but it is fun! I recommend it
47. pp self • Masataka “pocke” Kuwabara • Work for Bit Journey, Inc. / Kibela • Language: Ruby, Go, TypeScript Whitespace ← New! • github.com/pocke, twitter.com/p_ck_ Thank you for listening!