Designing and Building the Oden Programming Language

This first part of this talk will introduce Oden, an experimental, statically typed, functional programming language being built for the Go ecosystem. We will look at how Oden aims to leverage the great features of Go — static linking, cross-compilation, goroutines, channels and the great set of libraries and tools — and enable higher-level abstractions, generics and a safer yet more flexible type system.

The second part will delve more deeply into the implementation of the Oden compiler. Why was it first written in Racket and then rewritten in Haskell? What pros and cons are there in writing compilers in Haskell? We will look at how the type system can help us build safe and robust intermediate representations and transformations between them.

Transcript

1. Designing and Building
   The Oden Programming
   Language
   Oskar Wickström
   @owickstrom
   

   View Slide

2. Part One: Designing Oden
   • Introduction
   • Why Oden?
   • Project Goals
   • Current State
   • What’s next?
   

   View Slide

3. Part Two: Building Oden
   • Racket & miniKanren
   • Rewriting in Haskell
   • Explicit Modelling
   • AST, IR and Transformations
   • Libraries and Tools
   

   View Slide

4. Introduction
   

   View Slide

5. Me
   • Musical education
   • Started with PHP about 5 years ago
   • .NET, Java, some FP and a lot of Web
   • Haskell, languages and compilers on spare time
   

   View Slide

6. Project Background
   • Started off writing LISP interpreters
   • Interested in static type systems
   • Began exploring statically typed LISP
   • Implementing Hindley-Milner inference
   • Reusing an existing ecosystem
   

   View Slide

7. Why Oden?
   

   View Slide

8. The Go Programming Language
   • Simple language
   • Concurrency, channels
   • Garbage Collection
   • No class hierarchies
   • Static linking
   • Cross-compilation
   • Fast compiler
   • Gets shit done
   

   View Slide

9. The Ecosystem
   • Great standard library
   • Tools
   • fmt
   • Coverage
   • Linting
   • Profiling
   • Race detection
   • Lots of libraries
   • Lots of projects
   

   View Slide

10. What I’m Missing
    • Expressions rather than statements
    • Abstractions
    • Generics
    • Custom generic data structures
    • Higher-order functions
    • Error handling (example coming up)
    • nil checking
    • Better type inference
    

    View Slide

11. func (c *dropboxClient) GetFile(accessToken, dropboxSrc, localTarget string) error {
    url := "https://content.dropboxapi.com/1/files/auto" + dropboxSrc
    req, err := http.NewRequest("GET", url, nil)
    req.Header.Add("Authorization", "Bearer "+accessToken)
    if err != nil {
    return err
    }
    client := &http.Client{}
    res, err := client.Do(req)
    if err != nil {
    return err
    }
    if res.StatusCode == 404 {
    return ErrNotFound
    }
    if res.StatusCode < 200 || res.StatusCode >= 300 {
    return errors.New("Dropbox request failed: " + res.Status)
    }
    if err != nil {
    return err
    }
    f, err := os.Create(localTarget)
    if err != nil {
    return err
    }
    io.Copy(f, res.Body)
    if err != nil {
    return err
    }
    return nil
    }
    

    View Slide

12. func (c *dropboxClient) GetFile(accessToken, dropboxSrc, localTarget string) error {
    url := "https://content.dropboxapi.com/1/files/auto" + dropboxSrc
    req, err := http.NewRequest("GET", url, nil)
    req.Header.Add("Authorization", "Bearer "+accessToken)
    if err != nil {
    return err
    }
    client := &http.Client{}
    res, err := client.Do(req)
    if err != nil {
    return err
    }
    if res.StatusCode == 404 {
    return ErrNotFound
    }
    if res.StatusCode < 200 || res.StatusCode >= 300 {
    return errors.New("Dropbox request failed: " + res.Status)
    }
    if err != nil {
    return err
    }
    f, err := os.Create(localTarget)
    if err != nil {
    return err
    }
    io.Copy(f, res.Body)
    if err != nil {
    return err
    }
    return nil
    }
    

    View Slide

13. func (c *dropboxClient) GetFile(accessToken, dropboxSrc, localTarget string) error {
    url := "https://content.dropboxapi.com/1/files/auto" + dropboxSrc
    req, err := http.NewRequest("GET", url, nil)
    req.Header.Add("Authorization", "Bearer "+accessToken)
    if err != nil {
    return err
    }
    client := &http.Client{}
    res, err := client.Do(req)
    if err != nil {
    return err
    }
    if res.StatusCode == 404 {
    return ErrNotFound
    }
    if res.StatusCode < 200 || res.StatusCode >= 300 {
    return errors.New("Dropbox request failed: " + res.Status)
    }
    if err != nil {
    return err
    }
    f, err := os.Create(localTarget)
    if err != nil {
    return err
    }
    io.Copy(f, res.Body)
    if err != nil {
    return err
    }
    return nil
    }
    

    View Slide

14. Strategy:
    Use Go as a platform
    (compare with languages on JVM, Erlang etc)
    

    View Slide

15. Project Goals
    

    View Slide

16. Improve the Go drawbacks
    previously mentioned
    

    View Slide

17. Type Inference
    

    View Slide

18. https://en.wikipedia.org/wiki/Type_inference
    “Type inference refers to the automatic deduction
    of the data type of an expression in a
    programming language.”
    

    View Slide

19. Immutability By Default
    

    View Slide

20. http://www.dictionary.com/browse/immutable
    immutable
    adjective
    1.
    not mutable; unchangeable; changeless.
    

    View Slide

21. Pattern Matching
    

    View Slide

22. Erlang Pattern Matching
    dolphin() ->
    receive
    {From, do_a_flip} ->
    From ! "How about no?";
    {From, fish} ->
    From ! "So long and thanks for all the fish!";
    _ ->
    io:format("Heh, we're smarter than you humans.~n")
    end.
    http://learnyousomeerlang.com/the-hitchhikers-guide-to-concurrency
    

    View Slide

23. Exhaustiveness Checking
    

    View Slide

24. Haskell Exhaustiveness Checking
    data Message = DoAFlip | Fish | Ohai
    dolphin :: Message -> String
    dolphin msg = case msg of
    DoAFlip -> "How about no?"
    Fish -> "So long and thanks for all the fish!"
    

    View Slide

25. $ ghc -Wall Dolphin.hs
    ...
    Dolphin.hs:6:15: Warning:
    Pattern match(es) are non-exhaustive
    In a case alternative: Patterns not matched: Ohai
    

    View Slide

26. Covering all cases
    data Message = DoAFlip | Fish | Ohai
    dolphin :: Message -> String
    dolphin msg = case msg of
    DoAFlip -> "How about no?"
    Fish -> "So long and thanks for all the fish!"
    Ohai -> "Heh, we're smarter than you humans."
    

    View Slide

27. Simple Interoperability
    Avoiding cumbersome FFI
    㽊
    

    View Slide

28. Fun!
    

    View Slide

29. Current State
    

    View Slide

30. Hello World
    package main
    main() = println("hello world")
    

    View Slide

31. Literals
    0
    -1239
    true
    false
    "hello"
    "\nnewlines\nand stuff"
    

    View Slide

32. Operators
    1 + 2
    +y
    -(1 - 2)
    4 / 2
    (100 - 50) / 2
    1 == 2
    1 != 2
    !x && (true || (1 == 2))
    "Foo" ++ "Bar"
    

    View Slide

33. Anonymous Functions
    (x) -> x + 1
    (x, y) -> x + y
    

    View Slide

34. Top Level Functions
    plus1 = (x) -> x + 1
    plus = (x, y) -> x + y
    

    View Slide

35. Function Shorthand
    plus1(x) = x + 1
    plus(x, y) = x + y
    

    View Slide

36. Function Application
    f(a1
    , a2
    , a3
    , ..., aN
    )
    

    View Slide

37. Blocks
    answer = {
    log("Calculating the answer...")
    42
    }
    

    View Slide

38. Control Flow
    factorial(n) =
    if n < 2 then 1 else n * factorial(n - 1)
    

    View Slide

39. factorial(n) = {
    if n < 2 then {
    1
    } else {
    n * factorial(n - 1)
    }
    }
    Control Flow with Blocks
    

    View Slide

40. Curried Functions
    plus(x, y) = x + y
    plusTen = plus(10)
    thirty = plusTen(20)
    

    View Slide

41. Curried Function Type
    plus : int -> int -> int
    

    View Slide

42. Curried Function Type
    plus : int -> (int -> int)
    

    View Slide

43. Curried Higher-Order Functions
    plus(x, y) = x + y
    twice(f, x) = f(f(x))
    plusTwenty = twice(plus(10))
    forty = plusTwenty(20)
    

    View Slide

44. Slices
    names = []{"Sarah", "Joe", "Maxime"}
    tail = names[1:]
    greeting = "Hello, " ++ tail[1]
    

    View Slide

45. Tuples
    person : (string, int)
    

    View Slide

46. Tuple Literals
    jessica = ("Jessica", 31)
    frank = ("Frank", 26)
    people = []{jessica, frank, ("Maxime", 25)}
    

    View Slide

47. Type Signatures
    n : int
    n = 1
    f : forall a. a -> a
    f(x) = x
    

    View Slide

48. Imports from Go
    package main
    import html
    main() = {
    println(html.EscapeString("Simon & Garfunkel"))
    // Simon & Garfunkel
    }
    

    View Slide

49. Curried Foreign Functions
    // multi-parameter func in Go
    func PlusInGo(x, y int) int {
    return x + y
    }
    // curried in Oden
    incr = pkg.PlusInGo(1)
    

    View Slide

50. Constructing Records
    sweden = {
    official = "Kingdom Of Sweden",
    population = 9858794,
    capital = "Stockholm"
    }
    usa = {
    official = "United States of America",
    population = 322369319,
    capital = "Washington D.C."
    }
    

    View Slide

51. Record Type
    sweden : { official : string, population : int, capital : string }
    

    View Slide

52. Extensible Records
    Example
    

    View Slide

53. package main
    bonnie = { firstName = "Bonnie", lastName = "Swanson", age = 37 }
    peter = { firstName = "Peter", lastName = "Griffin" }
    fullName(person) =
    person.firstName ++ " " ++ person.lastName
    main() = {
    println(fullName(bonnie))
    println(fullName(peter))
    }
    

    View Slide

54. Adding Types for Persons
    bonnie : { firstName : string, lastName : string, age : int }
    bonnie = { firstName = "Bonnie", lastName = "Swanson", age = 37 }
    peter : { firstName : string, lastName : string }
    peter = { firstName = "Peter", lastName = "Griffin", age = 41 }
    

    View Slide

55. Inferring Record Types
    fullName(person) =
    person.firstName ++ " " ++ person.lastName
    

    View Slide

56. The Inferred Type of fullName
    fullName : forall r.
    { firstName : string, lastName : string | r }
    -> string
    fullName(person) =
    person.firstName ++ " " ++ person.lastName
    

    View Slide

57. UTF-8
    ~$ cat emoji.oden
    package main
    main() = println("ͩΩ΁ͷ΅, "!")
    ~$ oden run emoji.oden
    ͩΩ΁ͷ΅, "!
    

    View Slide

58. So, Oden is ready for production?
    No.
    

    View Slide

59. oden-lang.org
    

    View Slide

60. playground.oden-lang.org
    

    View Slide

61. What’s Next?
    • Record extension, better interoperability with Go structs
    • Protocols (like Haskell type classes, but integrating with Go interfaces)
    • Support all primitive types, channels, maps (requires protocols for
    overloaded operators)
    • Explicit type conversions (also based on protocols)
    • Support for requiring Oden packages
    • Better Go codegen (right now a lot of anonymous functions)
    • Pattern matching
    • Better validation and error messages
    

    View Slide

62. View Slide

63. Part 2
    

    View Slide

64. Racket & miniKanren
    

    View Slide

65. The Starting Point
    • Started digging in to Hindley-Milner type inference
    • Learned about miniKanren (http://minikanren.org/)
    • The Reasoned Schemer (Daniel P. Friedman, William
    E. Byrd and Oleg Kiselyov)
    • Hindley-Milner using Constraint Solving
    • Decided to use Racket (PLT Scheme)
    • First step: statically typed LISP
    

    View Slide

66. Super Fast Intro to miniKanren
    • relations are functions that returns goals
    • goals either fail or succeed
    • fresh creates new unbound logic variables
    • run gets answers for a relation
    • conde does logical disjunction (OR)
    • == unifies terms
    

    View Slide

67. Run Query Variable
    > (run 1 (q) (== q 10))
    '(10)
    

    View Slide

68. Unbound Variable
    > (run 1 (q)
    (fresh (x y z)
    (== 3 y)
    (== x z)))
    '(_.0)
    

    View Slide

69. Unifying Logic Variables
    > (run 1 (q)
    (fresh (x)
    (== 3 x)
    (== q x)))
    '(3)
    

    View Slide

70. conde
    > (run 2 (q)
    (conde
    [(== q 1)]
    [(== q 2)]))
    '(1 2)
    

    View Slide

71. conde
    > (run 2 (q)
    (== q 2)
    (conde
    [(== q 1)]
    [(== q 2)]))
    '(2)
    

    View Slide

72. Lists
    > (run 1 (q)
    (fresh (x)
    (== '(1 2 3) x)
    (== x (list 1 q 3))))
    '(2)
    

    View Slide

73. Scheme Quasiquotes
    > `(0 1 2)
    '(0 1 2)
    > `(1 ,(+ 1 2) 4)
    '(1 3 4)
    

    View Slide

74. Quasiquoting and Logic Variables
    > (run 1 (q)
    (fresh (x)
    (== '(1 2 3) x)
    (== x `(1 ,q 3))))
    '(2)
    

    View Slide

75. miniKanren Convention:
    Relation function names end with “o”
    

    View Slide

76. Some Built-in Relations
    • conso
    • symbolo
    • numbero
    • absento
    

    View Slide

77. pairo
    > (define (pairo f s o)
    (== `(,f ,s) o))
    > (run 1 (q)
    (fresh (x y)
    (pairo x y q)
    (== x 1)
    (== y 2)))
    '((1 2))
    

    View Slide

78. miniKanren Type Inferencer
    

    View Slide

79. https://en.wikipedia.org/wiki/Abstract_syntax_tree
    “In computer science, an abstract syntax tree
    (AST) […] is a tree representation of the
    abstract syntactic structure of source code
    written in a programming language. ”
    

    View Slide

80. AST: Regular LISP Data
    '(def identity (fn (x) x))
    

    View Slide

81. Input and Output
    > (infer 1)
    '(1 : int)
    

    View Slide

82. Input and Output
    > (infer 'some-variable)
    '(some-variable : string)
    

    View Slide

83. Input and Output
    > (infer '(fn (x) x))
    '((fn ((x : _.0)) (x : _.0)) : (_.0 -> _.0))
    

    View Slide

84. infero
    (define (infero expr env t)
    (conde
    [(fresh (x)
    (symbolo expr)
    (lookupo expr env x)
    (== `(,expr : ,x) t))]
    ...
    ))
    

    View Slide

85. infero (cont.)
    (define (infero expr env t)
    (conde
    ...
    [(fresh (f ft f-ignore a at a-ignore x et)
    (== `(,f ,a) expr)
    (infero f env ft)
    (infero a env at)
    (== `(,f-ignore : (,x -> ,et)) ft)
    (== `(,a-ignore : ,x) at)
    (== `((,ft ,at) : ,et) t))]
    ...
    ))
    

    View Slide

86. My miniKanren Experience
    • No, or very primitive, error handling
    • Type safety became a concern as it grew
    • No success with Typed Racket
    • The state of miniKanren
    • No canonical implementation (?)
    • Documentation
    • Missing and/or broken functions
    

    View Slide

87. I miss my Monads
    • miniKanren is basically a constraint solver monad
    • Cannot combine with other monads (error handling,
    state etc)
    • https://github.com/webyrd/error-handling-relational-
    interpreter/blob/master/error-interp-specific-all.scm
    

    View Slide

88. Racket Itches
    • Tooling
    • Racket REPL in Emacs (Geiser) slow
    • Crashed every now and then
    • Compiled program startup time: ~400ms
    

    View Slide

89. View Slide

90. Rewriting in Haskell
    

    View Slide

91. Write You A Haskell
    • Inferencer based on Write You A Haskell, by Stephen
    Diehl
    • Rewritten to return typed expressions
    http://dev.stephendiehl.com/fun/006_hindley_milner.html
    

    View Slide

92. Explicit Modelling
    

    View Slide

93. https://en.wikipedia.org/wiki/Intermediate_representation
    “An Intermediate representation (IR) is the data
    structure or code used internally by a compiler or
    virtual machine to represent source code.”
    

    View Slide

94. Explicit Modelling
    • Represent valid cases explicitly in data types
    • Prohibit invalid cases in data types
    • Distinct types for transformations inputs and outputs
    • Transformations will basically write themselves &
    • The type system will help you
    

    View Slide

95. Syntax Excerpt
    data Expr = Symbol SourceInfo Identifier
    | Application SourceInfo Expr [Expr]
    | Fn SourceInfo [NameBinding] Expr
    | Block SourceInfo [Expr]
    ...
    

    View Slide

96. Syntax Excerpt
    data Expr = Symbol SourceInfo Identifier
    | Application SourceInfo Expr [Expr]
    | Fn SourceInfo [NameBinding] Expr
    | Block SourceInfo [Expr]
    ...
    

    View Slide

97. Untyped IR
    data Expr = Symbol (Metadata SourceInfo) Identifier
    | Application (Metadata SourceInfo) Expr [Expr]
    | Fn (Metadata SourceInfo) NameBinding Expr
    | Block (Metadata SourceInfo) [Expr]
    ...
    

    View Slide

98. Untyped IR
    data Expr = Symbol (Metadata SourceInfo) Identifier
    | Application (Metadata SourceInfo) Expr [Expr]
    | Fn (Metadata SourceInfo) NameBinding Expr
    | Block (Metadata SourceInfo) [Expr]
    ...
    

    View Slide

99. AST, IR and Transformations
    

    View Slide

100. Curried Functions
     // In Oden we write:
     (x, y, z) -> x
     // ...but we get:
     (x) -> (y) -> (z) -> x
     

     View Slide

101. “Explode”
     explodeExpr :: Expr
     -> Either [ExplodeError] Untyped.Expr
     

     View Slide

102. Exploding Functions
     explodeExpr' (Fn sourceInfo (param:params) body) =
     Untyped.Fn
     (Metadata sourceInfo)
     (explodeNameBinding param)
     <$> explodeExpr' (Fn sourceInfo params body)
     ...
     

     View Slide

103. Other Examples
     

     View Slide

104. Example Code
     package identity
     // A polymorphic function.
     identity(x) = x
     // Use it with different types.
     three = identity(3)
     hello = identity("hello")
     

     View Slide

105. Resolve Imports
     resolveImports :: Importer
     -> Untyped.Package [Untyped.ImportReference]
     -> IO (Either PackageImportError
     (Untyped.Package [Core.ImportedPackage],
     [UnsupportedTypesWarning]))
     

     View Slide

106. Resolve Imports
     resolveImports :: Importer
     -> Untyped.Package [Untyped.ImportReference]
     -> IO (Either PackageImportError
     (Untyped.Package [Core.ImportedPackage],
     [UnsupportedTypesWarning]))
     

     View Slide

107. Inference
     inferPackage :: Untyped.Package [Core.ImportedPackage]
     -> Either TypeError Core.Package
     

     View Slide

108. After Type Inference
     ~$ oden print-inferred identity.oden
     package identity
     identity : forall a. a -> a
     identity = ((x) -> x)
     three : int
     three = identity(3)
     hello : string
     hello = identity("hello")
     

     View Slide

109. Monomorphization
     monomorphPackage :: Core.Package
     -> Either MonomorphError
     MonomorphedPackage
     

     View Slide

110. Monomorphization
     monomorph :: Core.Expr Poly.Type
     -> Monomorph (Core.Expr Mono.Type)
     

     View Slide

111. Monomorphization
     monomorph :: Core.Expr Poly.Type
     -> Monomorph (Core.Expr Mono.Type)
     

     View Slide

112. After Monomorphization
     ~$ oden print-compiled identity.oden
     package identity
     identity_inst_int_to_int : int -> int
     identity_inst_int_to_int = ((x) -> x)
     identity_inst_string_to_string : string -> string
     identity_inst_string_to_string = ((x) -> x)
     hello : string
     hello = identity_inst_string_to_string("hello")
     three : int
     three = identity_inst_int_to_int(3)
     

     View Slide

113. Codegen
     class Backend b where
     codegen :: b
     -> MonomorphedPackage
     -> Either CodegenError
     [CompiledFile]
     

     View Slide

114. After Codegen
     package identity
     import fmt "fmt"
     func print(x interface{}) {
     fmt.Print(x)
     }
     func println(x interface{}) {
     fmt.Println(x)
     }
     func identity_inst_int_to_int(x int) int {
     return x
     }
     func identity_inst_string_to_string(x string) string {
     return x
     }
     var hello string = identity_inst_string_to_string("hello")
     var three int = identity_inst_int_to_int(3)
     

     View Slide

115. After Codegen
     package identity
     import fmt "fmt"
     func print(x interface{}) {
     fmt.Print(x)
     }
     func println(x interface{}) {
     fmt.Println(x)
     }
     func identity_inst_int_to_int(x int) int {
     return x
     }
     func identity_inst_string_to_string(x string) string {
     return x
     }
     var hello string = identity_inst_string_to_string("hello")
     var three int = identity_inst_int_to_int(3)
     

     View Slide

116. Libraries & Tools
     

     View Slide

117. Haskell Libraries
     • parsec
     • Good to begin with, expressive
     • Might rewrite with alex and happy
     • mtl (monad transformers library)
     • Used throughout the compiler
     • wl-pprint
     • Better control over whitespace than standard Text.PrettyPrint
     

     View Slide

118. Haskell Tooling
     • hdevtools or ghc-mod with
     • Warnings
     • HLint
     • -Wall
     • HSpec with nodemon/guard
     

     View Slide

119. Summary
     

     View Slide

120. Summary
     • Oden: a functional programming language built for
     the Go ecosystem
     • The current state of Oden, what’s next
     • Racket, miniKanren and the Haskell rewrite
     • Explicit Modelling and Transformations in Haskell
     • Haskell libraries and tooling
     

     View Slide

121. Monads as Design Pattern
     The Door to Real-World Functional Programming
     Wednesday, April 27
     http://www.foocafe.org/event/monads-as-design-pattern
     

     View Slide

122. Contribute
     • Try it out
     • Discuss
     • https://groups.google.com/forum/#!forum/oden-lang
     • #oden at Freenode
     • Report issues on GitHub
     • Send pull requests
     

     View Slide

123. View Slide