Self-hosting Whitespace

https://docs.google.com/presentation/d/1BeNJ_E_KOLjjdM4Bd3u_96tiIGiRdG9J9orhglcxziw/edit?usp=sharing

Transcript

1. Self-hosting Whitespace
   24th, May. 2019
   Roppongi.rb#10
   https://roppongirb.connpass.com/event/129603/
   

   View Slide

2. ⚠ IT NEEDS RUBY TRUNK (2.7) ⚠
   

   View Slide

3. Agenda
   I implemented a self-hosted Whitespace
   interpreter.
   I’ll talk about the interpreter.
   

   View Slide

4. Agenda
   ● Prerequisite knowledge
   ○ What is self-hosting?
   ○ What is Whitespace?
   ● Self-hosted interpreter
   ○ How implement the interpreter
   ○ Implementation Tips
   

   View Slide

5. What is self-hosting?
   

   View Slide

6. Self-hosting is...
   ● Implementing a programming language by itself
   ● For instance
   ○ Go compiler is written in Go. (since v1.5)
   ○ minruby
   ■ 遠藤侑介 (2017) 『RubyでつくるRuby』ラムダノート.
   

   View Slide

7. Example: ruby interpreter on ruby interpreter
   $ ruby a.rb
   $ ruby interp.rb a.rb
   $ ruby interp.rb interp.rb a.rb
   $ ruby interp.rb interp.rb interp.rb interp.rb
   interp.rb a.rb
   

   View Slide

8. What is Whitespace?
   

   View Slide

9. Do you know Whitespace language?
   ● Never mind if you don’t know!
   https://twitter.com/p_ck_/status/1123410375804768256
   https://twitter.com/p_ck_/status/1123414869737263104
   

   View Slide

10. Whitespace
    ● Whitespace is an esoteric programming
    language (esolang)
    ○ Esolang example: Brainf**k, Piet, etc
    ● It consists only of space, tab and newline
    

    View Slide

11. hello.ws (Space → S, Tab → T, NewLine → N)
    SSSTSSTSSSNTNSSSSSTTSSTSTNTNSSSSSTTSTTSSNTNSSSS
    STTSTTSSNTNSSSSSTTSTTTTNTNSSSSSTSSSSSNTNSSSSSTS
    TSTTTNTNSSSSSTTSTTTTNTNSSSSSTTTSSTSNTNSSSSSTTS
    TTSSNTNSSSSSTTSSTSSNTNSSSSSSSSTSTSNTNSSNNN
    Space -> S, Tab -> T
    Ref:
    https://magazine.rubyist.net/articles/0022/0022-Le
    gwork.html
    

    View Slide

12. Inside of Whitespace
    ● It is a stack based language
    ● It has stack and heap for data store
    ● All value is a signed Integer
    ● It has goto and subroutine for flow control
    ● It has 22 commands to manipulate stack and so
    on
    ○ All commands are encoded only with whitespace chars
    

    View Slide

13. Whitespace Commands (1)
    Stack commands
    ● stack.push(val) SS
    ● stack.pop() SNN
    ● stack.dup() SNS
    ○ stack.push(stack.last)
    ● stack.swap() SNT
    ○ stack[-1], stack[-2] =
    stack[-2], stack[-1]
    Heap commands
    ● heap.save(key, val) TTS
    ○ val = stack.pop; key =
    stack.pop
    heap[key] = val
    ● heap.load(key) TTT
    ○ stack.push heap[key.pop()]
    

    View Slide

14. Whitespace Commands (2)
    Calculations
    a = stack.pop; b = stack.pop
    stack.push(op(b, a))
    ● add() TSSS
    ● sub() TSST
    ● multi() TSSN
    ● div() TSTS
    ● mod() TSTT
    IO
    ● write_char() TNSS
    ● write_number() TNST
    ○ print stack.pop()
    ● read_char() TNTS
    ● read_number() TNTT
    ○ heap[stack.pop] = read
    

    View Slide

15. Whitespace Commands (3)
    Flow
    ● define(label) NSS
    ○ define_method(label)
    ● call(label) NST
    ○ __send__(label)
    ● end NTN
    ● exit() NNN
    ● jump(label) NSN
    ○ goto label
    ● jump_if_zero(label) NTS
    ○ jump(label) if stack.pop == 0
    ● jump_if_negative(label) NTT
    ○ jump(label) if stack.pop < 0
    

    View Slide

16. Command Example: Stack and IO
    SSSTSTSTSNTNST
    stack.push(42)
    write_number # => 42
    

    View Slide

17. Command Example: Heap
    SSSTTNSSSTSTSTSNTTSSSSTTNTTTTNST
    stack.push(3)
    stack.push(42)
    heap.save
    stack.push(3)
    heap.load
    write_number
    # 3 is address
    # 42 is a value
    # Save 42 to addr 3
    # Load 42 from addr 3
    # => 42
    

    View Slide

18. Self-hosted Interpreter
    

    View Slide

19. The interpreter is here
    ● https://gist.github.com/pocke/2847214a8713
    9a0d1babd9338159d667
    ● 72,367 lines
    ● 565,639 bytes
    

    View Slide

20. DEMO
    $ gows fizzbuzz.ws
    $ cat fizzbuzz.ws sep | gows whitespace.ws
    $ cat whitespace.ws sep fizzbuzz.ws sep | gows
    whitespace.ws
    gows is a Whitespace interpreter written in Go.
    https://github.com/pocke/gows
    

    View Slide

21. How implement the interpreter
    

    View Slide

22. Did I handwrote the
    70,000+ lines
    Whitespace code?
    Definitely Not.
    

    View Slide

23. How implement Whitespace in Whitespace
    Handwriting Whitespace is difficult, so
    1. Write a compiler that compiles Ruby to
    Whitespace
    2. Write Whitespace interpreter in Ruby
    3. Generate Whitespace from Ruby!
    

    View Slide

24. 1. Write a compiler that compiles Ruby to
    Whitespace
    

    View Slide

25. Akaza: A toolchain of Whitespace, written in Ruby
    ● https://github.com/pocke/akaza
    ● It has 3 features
    ○ Whitespace interpreter
    ○ Whitespace compiler: wsrb ← Today’s theme!
    ○ Embedded Whitespace in Ruby
    ■ https://pocke.hatenablog.com/entry/2019/04/28/183509
    

    View Slide

26. wsrb: Whitespace compiler and the language
    ● wsrb language is a subset of Ruby
    ● It supports
    ○ Local vars, method call and def, Integer, Array, Hash,
    true, false, nil, if, case, while
    ● It does not support
    ○ Most built-in methods, String, class, module, meta
    programming, and so on
    

    View Slide

27. wsrb implementation
    ● https://github.com/pocke/akaza/blob/master/li
    b/akaza/ruby2ws.rb
    ● It uses RubyVM::AST (since Ruby 2.6) and
    pattern matching (since Ruby 2.7)
    ○ Parse Ruby code to AST with RubyVM::AST
    ○ Translate the AST to Whitespace with pattern matching
    

    View Slide

28. 2. Write Whitespace interpreter in Ruby
    

    View Slide

29. Interpreter implementation
    ● Simple parser and VM
    ● https://github.com/pocke/self-hosting-whitesp
    ace/blob/master/src/whitespace.ws.rb
    

    View Slide

30. 3. Generate Whitespace from Ruby
    

    View Slide

31. Compile
    $ akaza wsrb src/whitespace.ws.rb
    DEMO
    

    View Slide

32. Implementation Tips
    

    View Slide

33. Value and Class
    

    View Slide

34. Value with Class
    ● Wsrb has Integer, Array and Hash classes
    ● Wsrb needs to distinguish classes
    ○ e.g. `hash[1]` and `array[1]` should call different method
    ● So, each value should have class information
    

    View Slide

35. Value with Class
    ● Wsrb uses lower 2 bits for types
    ● For example
    ○ Value is 42. (42.to_s(2) #=> 101010)
    ○ Integer class is 0b01
    ○ In wsrb, the value is 10101001
    see: http://i.loveruby.net/ja/rhg/book/object.html
    

    View Slide

36. Array
    

    View Slide

37. Array
    ● 3 meta data
    ○ Address of the first item
    ○ Size: Same as Array#size
    ○ Capacity: Allocated items count (default: 10)
    ■ size <= capacity
    ■ If cap is too small, it’s re-allocated
    ● Items
    Addr of
    First Item
    Size Capacity Item 1 Item 2 Item 3 ...
    

    View Slide

38. Hash
    

    View Slide

39. Hash
    ● Use Hash table and linked list
    ● Hash table key is `key mod HASH_SIZE`
    ○ Default HASH_SIZE: 11
    ● When conflict, it uses linked list
    

    View Slide

40. Hash
    Addr of
    FIrst Item
    Key1 Value1 Addr to
    next key
    Key2 Value2 Addr to
    next item
    ...(3*HASH
    _SIZE)
    Key1’ Value1’ Addr to
    next key
    Key1’’ Value1’’ Addr to
    next key
    

    View Slide

41. Differences
    between
    Go and Ruby
    

    View Slide

42. Differences between Go and Ruby
    ● I found the differences when I was
    implementing gows
    ● Division and modulo have differences
    https://play.golang.org/p/_LGKTmlMAGJ
    

    View Slide

43. Division difference
    ● -5 / 3
    ○ Ruby: -2
    ○ Go: -1
    ● Absorb the difference by
    ○ return int(math.Floor(float64(left) /
    float64(right)))
    

    View Slide

44. Modulo differene
    ● -5 % 3
    ○ Ruby: 1
    ○ Go: -2
    ● Absorb the difference by
    ○ res := left % right
    if res < 0 {
    return j + res
    } else { return res }
    

    View Slide

45. Conclusion
    

    View Slide

46. Conclusion
    ● We learned how to implement a programing
    language
    ● Try implementing Whitespace, or other
    esolangs!
    ● Try self-hosting!
    ● It is not productive, but it is fun! I recommend it
    

    View Slide

47. pp self
    ● Masataka “pocke” Kuwabara
    ● Work for Bit Journey, Inc. / Kibela
    ● Language: Ruby, Go, TypeScript
    Whitespace ← New!
    ● github.com/pocke, twitter.com/p_ck_
    Thank you for listening!
    

    View Slide