Designing and Building the Oden Programming Language

This first part of this talk will introduce Oden, an experimental, statically typed, functional programming language being built for the Go ecosystem. We will look at how Oden aims to leverage the great features of Go — static linking, cross-compilation, goroutines, channels and the great set of libraries and tools — and enable higher-level abstractions, generics and a safer yet more flexible type system.

The second part will delve more deeply into the implementation of the Oden compiler. Why was it first written in Racket and then rewritten in Haskell? What pros and cons are there in writing compilers in Haskell? We will look at how the type system can help us build safe and robust intermediate representations and transformations between them.

Transcript

1. Designing and Building The Oden Programming Language Oskar Wickström @owickstrom
2. Part One: Designing Oden • Introduction • Why Oden? • Project Goals • Current State • What’s next?
3. Part Two: Building Oden • Racket & miniKanren • Rewriting in Haskell • Explicit Modelling • AST, IR and Transformations • Libraries and Tools
4. Introduction
5. Me • Musical education • Started with PHP about 5 years ago • .NET, Java, some FP and a lot of Web • Haskell, languages and compilers on spare time
6. Project Background • Started off writing LISP interpreters • Interested in static type systems • Began exploring statically typed LISP • Implementing Hindley-Milner inference • Reusing an existing ecosystem
7. Why Oden?
8. The Go Programming Language • Simple language • Concurrency, channels • Garbage Collection • No class hierarchies • Static linking • Cross-compilation • Fast compiler • Gets shit done
9. The Ecosystem • Great standard library • Tools • fmt • Coverage • Linting • Profiling • Race detection • Lots of libraries • Lots of projects
10. What I’m Missing • Expressions rather than statements • Abstractions • Generics • Custom generic data structures • Higher-order functions • Error handling (example coming up) • nil checking • Better type inference
11. func (c *dropboxClient) GetFile(accessToken, dropboxSrc, localTarget string) error { url := "https://content.dropboxapi.com/1/files/auto" + dropboxSrc req, err := http.NewRequest("GET", url, nil) req.Header.Add("Authorization", "Bearer "+accessToken) if err != nil { return err } client := &http.Client{} res, err := client.Do(req) if err != nil { return err } if res.StatusCode == 404 { return ErrNotFound } if res.StatusCode < 200 || res.StatusCode >= 300 { return errors.New("Dropbox request failed: " + res.Status) } if err != nil { return err } f, err := os.Create(localTarget) if err != nil { return err } io.Copy(f, res.Body) if err != nil { return err } return nil }
12. func (c *dropboxClient) GetFile(accessToken, dropboxSrc, localTarget string) error { url := "https://content.dropboxapi.com/1/files/auto" + dropboxSrc req, err := http.NewRequest("GET", url, nil) req.Header.Add("Authorization", "Bearer "+accessToken) if err != nil { return err } client := &http.Client{} res, err := client.Do(req) if err != nil { return err } if res.StatusCode == 404 { return ErrNotFound } if res.StatusCode < 200 || res.StatusCode >= 300 { return errors.New("Dropbox request failed: " + res.Status) } if err != nil { return err } f, err := os.Create(localTarget) if err != nil { return err } io.Copy(f, res.Body) if err != nil { return err } return nil }
13. func (c *dropboxClient) GetFile(accessToken, dropboxSrc, localTarget string) error { url := "https://content.dropboxapi.com/1/files/auto" + dropboxSrc req, err := http.NewRequest("GET", url, nil) req.Header.Add("Authorization", "Bearer "+accessToken) if err != nil { return err } client := &http.Client{} res, err := client.Do(req) if err != nil { return err } if res.StatusCode == 404 { return ErrNotFound } if res.StatusCode < 200 || res.StatusCode >= 300 { return errors.New("Dropbox request failed: " + res.Status) } if err != nil { return err } f, err := os.Create(localTarget) if err != nil { return err } io.Copy(f, res.Body) if err != nil { return err } return nil }
14. Strategy: Use Go as a platform (compare with languages on JVM, Erlang etc)
15. Project Goals
16. Improve the Go drawbacks previously mentioned
17. Type Inference
18. https://en.wikipedia.org/wiki/Type_inference “Type inference refers to the automatic deduction of the data type of an expression in a programming language.”
19. Immutability By Default
20. http://www.dictionary.com/browse/immutable immutable adjective 1. not mutable; unchangeable; changeless.
21. Pattern Matching
22. Erlang Pattern Matching dolphin() -> receive {From, do_a_flip} -> From ! "How about no?"; {From, fish} -> From ! "So long and thanks for all the fish!"; _ -> io:format("Heh, we're smarter than you humans.~n") end. http://learnyousomeerlang.com/the-hitchhikers-guide-to-concurrency
23. Exhaustiveness Checking
24. Haskell Exhaustiveness Checking data Message = DoAFlip | Fish | Ohai dolphin :: Message -> String dolphin msg = case msg of DoAFlip -> "How about no?" Fish -> "So long and thanks for all the fish!"
25. $ ghc -Wall Dolphin.hs ... Dolphin.hs:6:15: Warning: Pattern match(es) are non-exhaustive In a case alternative: Patterns not matched: Ohai
26. Covering all cases data Message = DoAFlip | Fish | Ohai dolphin :: Message -> String dolphin msg = case msg of DoAFlip -> "How about no?" Fish -> "So long and thanks for all the fish!" Ohai -> "Heh, we're smarter than you humans."
27. Simple Interoperability Avoiding cumbersome FFI 㽊
28. Fun!
29. Current State
30. Hello World package main main() = println("hello world")
31. Literals 0 -1239 true false "hello" "\nnewlines\nand stuff"
32. Operators 1 + 2 +y -(1 - 2) 4 / 2 (100 - 50) / 2 1 == 2 1 != 2 !x && (true || (1 == 2)) "Foo" ++ "Bar"
33. Anonymous Functions (x) -> x + 1 (x, y) -> x + y
34. Top Level Functions plus1 = (x) -> x + 1 plus = (x, y) -> x + y
35. Function Shorthand plus1(x) = x + 1 plus(x, y) = x + y
36. Function Application f(a1 , a2 , a3 , ..., aN )
37. Blocks answer = { log("Calculating the answer...") 42 }
38. Control Flow factorial(n) = if n < 2 then 1 else n * factorial(n - 1)
39. factorial(n) = { if n < 2 then { 1 } else { n * factorial(n - 1) } } Control Flow with Blocks
40. Curried Functions plus(x, y) = x + y plusTen = plus(10) thirty = plusTen(20)
41. Curried Function Type plus : int -> int -> int
42. Curried Function Type plus : int -> (int -> int)
43. Curried Higher-Order Functions plus(x, y) = x + y twice(f, x) = f(f(x)) plusTwenty = twice(plus(10)) forty = plusTwenty(20)
44. Slices names = []{"Sarah", "Joe", "Maxime"} tail = names[1:] greeting = "Hello, " ++ tail[1]
45. Tuples person : (string, int)
46. Tuple Literals jessica = ("Jessica", 31) frank = ("Frank", 26) people = []{jessica, frank, ("Maxime", 25)}
47. Type Signatures n : int n = 1 f : forall a. a -> a f(x) = x
48. Imports from Go package main import html main() = { println(html.EscapeString("Simon & Garfunkel")) // Simon &amp; Garfunkel }
49. Curried Foreign Functions // multi-parameter func in Go func PlusInGo(x, y int) int { return x + y } // curried in Oden incr = pkg.PlusInGo(1)
50. Constructing Records sweden = { official = "Kingdom Of Sweden", population = 9858794, capital = "Stockholm" } usa = { official = "United States of America", population = 322369319, capital = "Washington D.C." }
51. Record Type sweden : { official : string, population : int, capital : string }
52. Extensible Records Example
53. package main bonnie = { firstName = "Bonnie", lastName = "Swanson", age = 37 } peter = { firstName = "Peter", lastName = "Griffin" } fullName(person) = person.firstName ++ " " ++ person.lastName main() = { println(fullName(bonnie)) println(fullName(peter)) }
54. Adding Types for Persons bonnie : { firstName : string, lastName : string, age : int } bonnie = { firstName = "Bonnie", lastName = "Swanson", age = 37 } peter : { firstName : string, lastName : string } peter = { firstName = "Peter", lastName = "Griffin", age = 41 }
55. Inferring Record Types fullName(person) = person.firstName ++ " " ++ person.lastName
56. The Inferred Type of fullName fullName : forall r. { firstName : string, lastName : string | r } -> string fullName(person) = person.firstName ++ " " ++ person.lastName
57. UTF-8 ~$ cat emoji.oden package main main() = println("ͩΩ΁ͷ΅, "!") ~$ oden run emoji.oden ͩΩ΁ͷ΅, "!
58. So, Oden is ready for production? No.
59. oden-lang.org
60. playground.oden-lang.org
61. What’s Next? • Record extension, better interoperability with Go structs • Protocols (like Haskell type classes, but integrating with Go interfaces) • Support all primitive types, channels, maps (requires protocols for overloaded operators) • Explicit type conversions (also based on protocols) • Support for requiring Oden packages • Better Go codegen (right now a lot of anonymous functions) • Pattern matching • Better validation and error messages
63. Part 2
64. Racket & miniKanren
65. The Starting Point • Started digging in to Hindley-Milner type inference • Learned about miniKanren (http://minikanren.org/) • The Reasoned Schemer (Daniel P. Friedman, William E. Byrd and Oleg Kiselyov) • Hindley-Milner using Constraint Solving • Decided to use Racket (PLT Scheme) • First step: statically typed LISP
66. Super Fast Intro to miniKanren • relations are functions that returns goals • goals either fail or succeed • fresh creates new unbound logic variables • run gets answers for a relation • conde does logical disjunction (OR) • == unifies terms
67. Run Query Variable > (run 1 (q) (== q 10)) '(10)
68. Unbound Variable > (run 1 (q) (fresh (x y z) (== 3 y) (== x z))) '(_.0)
69. Unifying Logic Variables > (run 1 (q) (fresh (x) (== 3 x) (== q x))) '(3)
70. conde > (run 2 (q) (conde [(== q 1)] [(== q 2)])) '(1 2)
71. conde > (run 2 (q) (== q 2) (conde [(== q 1)] [(== q 2)])) '(2)
72. Lists > (run 1 (q) (fresh (x) (== '(1 2 3) x) (== x (list 1 q 3)))) '(2)
73. Scheme Quasiquotes > `(0 1 2) '(0 1 2) > `(1 ,(+ 1 2) 4) '(1 3 4)
74. Quasiquoting and Logic Variables > (run 1 (q) (fresh (x) (== '(1 2 3) x) (== x `(1 ,q 3)))) '(2)
75. miniKanren Convention: Relation function names end with “o”
76. Some Built-in Relations • conso • symbolo • numbero • absento
77. pairo > (define (pairo f s o) (== `(,f ,s) o)) > (run 1 (q) (fresh (x y) (pairo x y q) (== x 1) (== y 2))) '((1 2))
78. miniKanren Type Inferencer
79. https://en.wikipedia.org/wiki/Abstract_syntax_tree “In computer science, an abstract syntax tree (AST) […] is a tree representation of the abstract syntactic structure of source code written in a programming language. ”
80. AST: Regular LISP Data '(def identity (fn (x) x))
81. Input and Output > (infer 1) '(1 : int)
82. Input and Output > (infer 'some-variable) '(some-variable : string)
83. Input and Output > (infer '(fn (x) x)) '((fn ((x : _.0)) (x : _.0)) : (_.0 -> _.0))
84. infero (define (infero expr env t) (conde [(fresh (x) (symbolo expr) (lookupo expr env x) (== `(,expr : ,x) t))] ... ))
85. infero (cont.) (define (infero expr env t) (conde ... [(fresh (f ft f-ignore a at a-ignore x et) (== `(,f ,a) expr) (infero f env ft) (infero a env at) (== `(,f-ignore : (,x -> ,et)) ft) (== `(,a-ignore : ,x) at) (== `((,ft ,at) : ,et) t))] ... ))
86. My miniKanren Experience • No, or very primitive, error handling • Type safety became a concern as it grew • No success with Typed Racket • The state of miniKanren • No canonical implementation (?) • Documentation • Missing and/or broken functions
87. I miss my Monads • miniKanren is basically a constraint solver monad • Cannot combine with other monads (error handling, state etc) • https://github.com/webyrd/error-handling-relational- interpreter/blob/master/error-interp-specific-all.scm
88. Racket Itches • Tooling • Racket REPL in Emacs (Geiser) slow • Crashed every now and then • Compiled program startup time: ~400ms
90. Rewriting in Haskell
91. Write You A Haskell • Inferencer based on Write You A Haskell, by Stephen Diehl • Rewritten to return typed expressions http://dev.stephendiehl.com/fun/006_hindley_milner.html
92. Explicit Modelling
93. https://en.wikipedia.org/wiki/Intermediate_representation “An Intermediate representation (IR) is the data structure or code used internally by a compiler or virtual machine to represent source code.”
94. Explicit Modelling • Represent valid cases explicitly in data types • Prohibit invalid cases in data types • Distinct types for transformations inputs and outputs • Transformations will basically write themselves & • The type system will help you
95. Syntax Excerpt data Expr = Symbol SourceInfo Identifier | Application SourceInfo Expr [Expr] | Fn SourceInfo [NameBinding] Expr | Block SourceInfo [Expr] ...
96. Syntax Excerpt data Expr = Symbol SourceInfo Identifier | Application SourceInfo Expr [Expr] | Fn SourceInfo [NameBinding] Expr | Block SourceInfo [Expr] ...
97. Untyped IR data Expr = Symbol (Metadata SourceInfo) Identifier | Application (Metadata SourceInfo) Expr [Expr] | Fn (Metadata SourceInfo) NameBinding Expr | Block (Metadata SourceInfo) [Expr] ...
98. Untyped IR data Expr = Symbol (Metadata SourceInfo) Identifier | Application (Metadata SourceInfo) Expr [Expr] | Fn (Metadata SourceInfo) NameBinding Expr | Block (Metadata SourceInfo) [Expr] ...
99. AST, IR and Transformations
100. Curried Functions // In Oden we write: (x, y, z) -> x // ...but we get: (x) -> (y) -> (z) -> x
101. “Explode” explodeExpr :: Expr -> Either [ExplodeError] Untyped.Expr
102. Exploding Functions explodeExpr' (Fn sourceInfo (param:params) body) = Untyped.Fn (Metadata sourceInfo) (explodeNameBinding param) <$> explodeExpr' (Fn sourceInfo params body) ...
103. Other Examples
104. Example Code package identity // A polymorphic function. identity(x) = x // Use it with different types. three = identity(3) hello = identity("hello")
105. Resolve Imports resolveImports :: Importer -> Untyped.Package [Untyped.ImportReference] -> IO (Either PackageImportError (Untyped.Package [Core.ImportedPackage], [UnsupportedTypesWarning]))
106. Resolve Imports resolveImports :: Importer -> Untyped.Package [Untyped.ImportReference] -> IO (Either PackageImportError (Untyped.Package [Core.ImportedPackage], [UnsupportedTypesWarning]))
107. Inference inferPackage :: Untyped.Package [Core.ImportedPackage] -> Either TypeError Core.Package
108. After Type Inference ~$ oden print-inferred identity.oden package identity identity : forall a. a -> a identity = ((x) -> x) three : int three = identity(3) hello : string hello = identity("hello")
109. Monomorphization monomorphPackage :: Core.Package -> Either MonomorphError MonomorphedPackage
110. Monomorphization monomorph :: Core.Expr Poly.Type -> Monomorph (Core.Expr Mono.Type)
111. Monomorphization monomorph :: Core.Expr Poly.Type -> Monomorph (Core.Expr Mono.Type)
112. After Monomorphization ~$ oden print-compiled identity.oden package identity identity_inst_int_to_int : int -> int identity_inst_int_to_int = ((x) -> x) identity_inst_string_to_string : string -> string identity_inst_string_to_string = ((x) -> x) hello : string hello = identity_inst_string_to_string("hello") three : int three = identity_inst_int_to_int(3)
113. Codegen class Backend b where codegen :: b -> MonomorphedPackage -> Either CodegenError [CompiledFile]
114. After Codegen package identity import fmt "fmt" func print(x interface{}) { fmt.Print(x) } func println(x interface{}) { fmt.Println(x) } func identity_inst_int_to_int(x int) int { return x } func identity_inst_string_to_string(x string) string { return x } var hello string = identity_inst_string_to_string("hello") var three int = identity_inst_int_to_int(3)
115. After Codegen package identity import fmt "fmt" func print(x interface{}) { fmt.Print(x) } func println(x interface{}) { fmt.Println(x) } func identity_inst_int_to_int(x int) int { return x } func identity_inst_string_to_string(x string) string { return x } var hello string = identity_inst_string_to_string("hello") var three int = identity_inst_int_to_int(3)
116. Libraries & Tools
117. Haskell Libraries • parsec • Good to begin with, expressive • Might rewrite with alex and happy • mtl (monad transformers library) • Used throughout the compiler • wl-pprint • Better control over whitespace than standard Text.PrettyPrint
118. Haskell Tooling • hdevtools or ghc-mod with <insert editor here> • Warnings • HLint • -Wall • HSpec with nodemon/guard
119. Summary
120. Summary • Oden: a functional programming language built for the Go ecosystem • The current state of Oden, what’s next • Racket, miniKanren and the Haskell rewrite • Explicit Modelling and Transformations in Haskell • Haskell libraries and tooling
121. Monads as Design Pattern The Door to Real-World Functional Programming Wednesday, April 27 http://www.foocafe.org/event/monads-as-design-pattern
122. Contribute • Try it out • Discuss • https://groups.google.com/forum/#!forum/oden-lang • #oden at Freenode • Report issues on GitHub • Send pull requests