Designing and Building the Oden Programming Language

Designing and Building the Oden Programming Language

This first part of this talk will introduce Oden, an experimental, statically typed, functional programming language being built for the Go ecosystem. We will look at how Oden aims to leverage the great features of Go — static linking, cross-compilation, goroutines, channels and the great set of libraries and tools — and enable higher-level abstractions, generics and a safer yet more flexible type system.

The second part will delve more deeply into the implementation of the Oden compiler. Why was it first written in Racket and then rewritten in Haskell? What pros and cons are there in writing compilers in Haskell? We will look at how the type system can help us build safe and robust intermediate representations and transformations between them.

Be279c8e94a0825d7c982e745a3b3ba9?s=128

Oskar Wickström

April 13, 2016
Tweet

Transcript

  1. Designing and Building The Oden Programming Language Oskar Wickström @owickstrom

  2. Part One: Designing Oden • Introduction • Why Oden? •

    Project Goals • Current State • What’s next?
  3. Part Two: Building Oden • Racket & miniKanren • Rewriting

    in Haskell • Explicit Modelling • AST, IR and Transformations • Libraries and Tools
  4. Introduction

  5. Me • Musical education • Started with PHP about 5

    years ago • .NET, Java, some FP and a lot of Web • Haskell, languages and compilers on spare time
  6. Project Background • Started off writing LISP interpreters • Interested

    in static type systems • Began exploring statically typed LISP • Implementing Hindley-Milner inference • Reusing an existing ecosystem
  7. Why Oden?

  8. The Go Programming Language • Simple language • Concurrency, channels

    • Garbage Collection • No class hierarchies • Static linking • Cross-compilation • Fast compiler • Gets shit done
  9. The Ecosystem • Great standard library • Tools • fmt

    • Coverage • Linting • Profiling • Race detection • Lots of libraries • Lots of projects
  10. What I’m Missing • Expressions rather than statements • Abstractions

    • Generics • Custom generic data structures • Higher-order functions • Error handling (example coming up) • nil checking • Better type inference
  11. func (c *dropboxClient) GetFile(accessToken, dropboxSrc, localTarget string) error { url

    := "https://content.dropboxapi.com/1/files/auto" + dropboxSrc req, err := http.NewRequest("GET", url, nil) req.Header.Add("Authorization", "Bearer "+accessToken) if err != nil { return err } client := &http.Client{} res, err := client.Do(req) if err != nil { return err } if res.StatusCode == 404 { return ErrNotFound } if res.StatusCode < 200 || res.StatusCode >= 300 { return errors.New("Dropbox request failed: " + res.Status) } if err != nil { return err } f, err := os.Create(localTarget) if err != nil { return err } io.Copy(f, res.Body) if err != nil { return err } return nil }
  12. func (c *dropboxClient) GetFile(accessToken, dropboxSrc, localTarget string) error { url

    := "https://content.dropboxapi.com/1/files/auto" + dropboxSrc req, err := http.NewRequest("GET", url, nil) req.Header.Add("Authorization", "Bearer "+accessToken) if err != nil { return err } client := &http.Client{} res, err := client.Do(req) if err != nil { return err } if res.StatusCode == 404 { return ErrNotFound } if res.StatusCode < 200 || res.StatusCode >= 300 { return errors.New("Dropbox request failed: " + res.Status) } if err != nil { return err } f, err := os.Create(localTarget) if err != nil { return err } io.Copy(f, res.Body) if err != nil { return err } return nil }
  13. func (c *dropboxClient) GetFile(accessToken, dropboxSrc, localTarget string) error { url

    := "https://content.dropboxapi.com/1/files/auto" + dropboxSrc req, err := http.NewRequest("GET", url, nil) req.Header.Add("Authorization", "Bearer "+accessToken) if err != nil { return err } client := &http.Client{} res, err := client.Do(req) if err != nil { return err } if res.StatusCode == 404 { return ErrNotFound } if res.StatusCode < 200 || res.StatusCode >= 300 { return errors.New("Dropbox request failed: " + res.Status) } if err != nil { return err } f, err := os.Create(localTarget) if err != nil { return err } io.Copy(f, res.Body) if err != nil { return err } return nil }
  14. Strategy: Use Go as a platform (compare with languages on

    JVM, Erlang etc)
  15. Project Goals

  16. Improve the Go drawbacks previously mentioned

  17. Type Inference

  18. https://en.wikipedia.org/wiki/Type_inference “Type inference refers to the automatic deduction of the

    data type of an expression in a programming language.”
  19. Immutability By Default

  20. http://www.dictionary.com/browse/immutable immutable adjective 1. not mutable; unchangeable; changeless.

  21. Pattern Matching

  22. Erlang Pattern Matching dolphin() -> receive {From, do_a_flip} -> From

    ! "How about no?"; {From, fish} -> From ! "So long and thanks for all the fish!"; _ -> io:format("Heh, we're smarter than you humans.~n") end. http://learnyousomeerlang.com/the-hitchhikers-guide-to-concurrency
  23. Exhaustiveness Checking

  24. Haskell Exhaustiveness Checking data Message = DoAFlip | Fish |

    Ohai dolphin :: Message -> String dolphin msg = case msg of DoAFlip -> "How about no?" Fish -> "So long and thanks for all the fish!"
  25. $ ghc -Wall Dolphin.hs ... Dolphin.hs:6:15: Warning: Pattern match(es) are

    non-exhaustive In a case alternative: Patterns not matched: Ohai
  26. Covering all cases data Message = DoAFlip | Fish |

    Ohai dolphin :: Message -> String dolphin msg = case msg of DoAFlip -> "How about no?" Fish -> "So long and thanks for all the fish!" Ohai -> "Heh, we're smarter than you humans."
  27. Simple Interoperability Avoiding cumbersome FFI 㽊

  28. Fun!

  29. Current State

  30. Hello World package main main() = println("hello world")

  31. Literals 0 -1239 true false "hello" "\nnewlines\nand stuff"

  32. Operators 1 + 2 +y -(1 - 2) 4 /

    2 (100 - 50) / 2 1 == 2 1 != 2 !x && (true || (1 == 2)) "Foo" ++ "Bar"
  33. Anonymous Functions (x) -> x + 1 (x, y) ->

    x + y
  34. Top Level Functions plus1 = (x) -> x + 1

    plus = (x, y) -> x + y
  35. Function Shorthand plus1(x) = x + 1 plus(x, y) =

    x + y
  36. Function Application f(a1 , a2 , a3 , ..., aN

    )
  37. Blocks answer = { log("Calculating the answer...") 42 }

  38. Control Flow factorial(n) = if n < 2 then 1

    else n * factorial(n - 1)
  39. factorial(n) = { if n < 2 then { 1

    } else { n * factorial(n - 1) } } Control Flow with Blocks
  40. Curried Functions plus(x, y) = x + y plusTen =

    plus(10) thirty = plusTen(20)
  41. Curried Function Type plus : int -> int -> int

  42. Curried Function Type plus : int -> (int -> int)

  43. Curried Higher-Order Functions plus(x, y) = x + y twice(f,

    x) = f(f(x)) plusTwenty = twice(plus(10)) forty = plusTwenty(20)
  44. Slices names = []{"Sarah", "Joe", "Maxime"} tail = names[1:] greeting

    = "Hello, " ++ tail[1]
  45. Tuples person : (string, int)

  46. Tuple Literals jessica = ("Jessica", 31) frank = ("Frank", 26)

    people = []{jessica, frank, ("Maxime", 25)}
  47. Type Signatures n : int n = 1 f :

    forall a. a -> a f(x) = x
  48. Imports from Go package main import html main() = {

    println(html.EscapeString("Simon & Garfunkel")) // Simon &amp; Garfunkel }
  49. Curried Foreign Functions // multi-parameter func in Go func PlusInGo(x,

    y int) int { return x + y } // curried in Oden incr = pkg.PlusInGo(1)
  50. Constructing Records sweden = { official = "Kingdom Of Sweden",

    population = 9858794, capital = "Stockholm" } usa = { official = "United States of America", population = 322369319, capital = "Washington D.C." }
  51. Record Type sweden : { official : string, population :

    int, capital : string }
  52. Extensible Records Example

  53. package main bonnie = { firstName = "Bonnie", lastName =

    "Swanson", age = 37 } peter = { firstName = "Peter", lastName = "Griffin" } fullName(person) = person.firstName ++ " " ++ person.lastName main() = { println(fullName(bonnie)) println(fullName(peter)) }
  54. Adding Types for Persons bonnie : { firstName : string,

    lastName : string, age : int } bonnie = { firstName = "Bonnie", lastName = "Swanson", age = 37 } peter : { firstName : string, lastName : string } peter = { firstName = "Peter", lastName = "Griffin", age = 41 }
  55. Inferring Record Types fullName(person) = person.firstName ++ " " ++

    person.lastName
  56. The Inferred Type of fullName fullName : forall r. {

    firstName : string, lastName : string | r } -> string fullName(person) = person.firstName ++ " " ++ person.lastName
  57. UTF-8 ~$ cat emoji.oden package main main() = println("ͩΩ΁ͷ΅, "!")

    ~$ oden run emoji.oden ͩΩ΁ͷ΅, "!
  58. So, Oden is ready for production? No.

  59. oden-lang.org

  60. playground.oden-lang.org

  61. What’s Next? • Record extension, better interoperability with Go structs

    • Protocols (like Haskell type classes, but integrating with Go interfaces) • Support all primitive types, channels, maps (requires protocols for overloaded operators) • Explicit type conversions (also based on protocols) • Support for requiring Oden packages • Better Go codegen (right now a lot of anonymous functions) • Pattern matching • Better validation and error messages
  62. None
  63. Part 2

  64. Racket & miniKanren

  65. The Starting Point • Started digging in to Hindley-Milner type

    inference • Learned about miniKanren (http://minikanren.org/) • The Reasoned Schemer (Daniel P. Friedman, William E. Byrd and Oleg Kiselyov) • Hindley-Milner using Constraint Solving • Decided to use Racket (PLT Scheme) • First step: statically typed LISP
  66. Super Fast Intro to miniKanren • relations are functions that

    returns goals • goals either fail or succeed • fresh creates new unbound logic variables • run gets answers for a relation • conde does logical disjunction (OR) • == unifies terms
  67. Run Query Variable > (run 1 (q) (== q 10))

    '(10)
  68. Unbound Variable > (run 1 (q) (fresh (x y z)

    (== 3 y) (== x z))) '(_.0)
  69. Unifying Logic Variables > (run 1 (q) (fresh (x) (==

    3 x) (== q x))) '(3)
  70. conde > (run 2 (q) (conde [(== q 1)] [(==

    q 2)])) '(1 2)
  71. conde > (run 2 (q) (== q 2) (conde [(==

    q 1)] [(== q 2)])) '(2)
  72. Lists > (run 1 (q) (fresh (x) (== '(1 2

    3) x) (== x (list 1 q 3)))) '(2)
  73. Scheme Quasiquotes > `(0 1 2) '(0 1 2) >

    `(1 ,(+ 1 2) 4) '(1 3 4)
  74. Quasiquoting and Logic Variables > (run 1 (q) (fresh (x)

    (== '(1 2 3) x) (== x `(1 ,q 3)))) '(2)
  75. miniKanren Convention: Relation function names end with “o”

  76. Some Built-in Relations • conso • symbolo • numbero •

    absento
  77. pairo > (define (pairo f s o) (== `(,f ,s)

    o)) > (run 1 (q) (fresh (x y) (pairo x y q) (== x 1) (== y 2))) '((1 2))
  78. miniKanren Type Inferencer

  79. https://en.wikipedia.org/wiki/Abstract_syntax_tree “In computer science, an abstract syntax tree (AST) […]

    is a tree representation of the abstract syntactic structure of source code written in a programming language. ”
  80. AST: Regular LISP Data '(def identity (fn (x) x))

  81. Input and Output > (infer 1) '(1 : int)

  82. Input and Output > (infer 'some-variable) '(some-variable : string)

  83. Input and Output > (infer '(fn (x) x)) '((fn ((x

    : _.0)) (x : _.0)) : (_.0 -> _.0))
  84. infero (define (infero expr env t) (conde [(fresh (x) (symbolo

    expr) (lookupo expr env x) (== `(,expr : ,x) t))] ... ))
  85. infero (cont.) (define (infero expr env t) (conde ... [(fresh

    (f ft f-ignore a at a-ignore x et) (== `(,f ,a) expr) (infero f env ft) (infero a env at) (== `(,f-ignore : (,x -> ,et)) ft) (== `(,a-ignore : ,x) at) (== `((,ft ,at) : ,et) t))] ... ))
  86. My miniKanren Experience • No, or very primitive, error handling

    • Type safety became a concern as it grew • No success with Typed Racket • The state of miniKanren • No canonical implementation (?) • Documentation • Missing and/or broken functions
  87. I miss my Monads • miniKanren is basically a constraint

    solver monad • Cannot combine with other monads (error handling, state etc) • https://github.com/webyrd/error-handling-relational- interpreter/blob/master/error-interp-specific-all.scm
  88. Racket Itches • Tooling • Racket REPL in Emacs (Geiser)

    slow • Crashed every now and then • Compiled program startup time: ~400ms
  89. None
  90. Rewriting in Haskell

  91. Write You A Haskell • Inferencer based on Write You

    A Haskell, by Stephen Diehl • Rewritten to return typed expressions http://dev.stephendiehl.com/fun/006_hindley_milner.html
  92. Explicit Modelling

  93. https://en.wikipedia.org/wiki/Intermediate_representation “An Intermediate representation (IR) is the data structure or

    code used internally by a compiler or virtual machine to represent source code.”
  94. Explicit Modelling • Represent valid cases explicitly in data types

    • Prohibit invalid cases in data types • Distinct types for transformations inputs and outputs • Transformations will basically write themselves & • The type system will help you
  95. Syntax Excerpt data Expr = Symbol SourceInfo Identifier | Application

    SourceInfo Expr [Expr] | Fn SourceInfo [NameBinding] Expr | Block SourceInfo [Expr] ...
  96. Syntax Excerpt data Expr = Symbol SourceInfo Identifier | Application

    SourceInfo Expr [Expr] | Fn SourceInfo [NameBinding] Expr | Block SourceInfo [Expr] ...
  97. Untyped IR data Expr = Symbol (Metadata SourceInfo) Identifier |

    Application (Metadata SourceInfo) Expr [Expr] | Fn (Metadata SourceInfo) NameBinding Expr | Block (Metadata SourceInfo) [Expr] ...
  98. Untyped IR data Expr = Symbol (Metadata SourceInfo) Identifier |

    Application (Metadata SourceInfo) Expr [Expr] | Fn (Metadata SourceInfo) NameBinding Expr | Block (Metadata SourceInfo) [Expr] ...
  99. AST, IR and Transformations

  100. Curried Functions // In Oden we write: (x, y, z)

    -> x // ...but we get: (x) -> (y) -> (z) -> x
  101. “Explode” explodeExpr :: Expr -> Either [ExplodeError] Untyped.Expr

  102. Exploding Functions explodeExpr' (Fn sourceInfo (param:params) body) = Untyped.Fn (Metadata

    sourceInfo) (explodeNameBinding param) <$> explodeExpr' (Fn sourceInfo params body) ...
  103. Other Examples

  104. Example Code package identity // A polymorphic function. identity(x) =

    x // Use it with different types. three = identity(3) hello = identity("hello")
  105. Resolve Imports resolveImports :: Importer -> Untyped.Package [Untyped.ImportReference] -> IO

    (Either PackageImportError (Untyped.Package [Core.ImportedPackage], [UnsupportedTypesWarning]))
  106. Resolve Imports resolveImports :: Importer -> Untyped.Package [Untyped.ImportReference] -> IO

    (Either PackageImportError (Untyped.Package [Core.ImportedPackage], [UnsupportedTypesWarning]))
  107. Inference inferPackage :: Untyped.Package [Core.ImportedPackage] -> Either TypeError Core.Package

  108. After Type Inference ~$ oden print-inferred identity.oden package identity identity

    : forall a. a -> a identity = ((x) -> x) three : int three = identity(3) hello : string hello = identity("hello")
  109. Monomorphization monomorphPackage :: Core.Package -> Either MonomorphError MonomorphedPackage

  110. Monomorphization monomorph :: Core.Expr Poly.Type -> Monomorph (Core.Expr Mono.Type)

  111. Monomorphization monomorph :: Core.Expr Poly.Type -> Monomorph (Core.Expr Mono.Type)

  112. After Monomorphization ~$ oden print-compiled identity.oden package identity identity_inst_int_to_int :

    int -> int identity_inst_int_to_int = ((x) -> x) identity_inst_string_to_string : string -> string identity_inst_string_to_string = ((x) -> x) hello : string hello = identity_inst_string_to_string("hello") three : int three = identity_inst_int_to_int(3)
  113. Codegen class Backend b where codegen :: b -> MonomorphedPackage

    -> Either CodegenError [CompiledFile]
  114. After Codegen package identity import fmt "fmt" func print(x interface{})

    { fmt.Print(x) } func println(x interface{}) { fmt.Println(x) } func identity_inst_int_to_int(x int) int { return x } func identity_inst_string_to_string(x string) string { return x } var hello string = identity_inst_string_to_string("hello") var three int = identity_inst_int_to_int(3)
  115. After Codegen package identity import fmt "fmt" func print(x interface{})

    { fmt.Print(x) } func println(x interface{}) { fmt.Println(x) } func identity_inst_int_to_int(x int) int { return x } func identity_inst_string_to_string(x string) string { return x } var hello string = identity_inst_string_to_string("hello") var three int = identity_inst_int_to_int(3)
  116. Libraries & Tools

  117. Haskell Libraries • parsec • Good to begin with, expressive

    • Might rewrite with alex and happy • mtl (monad transformers library) • Used throughout the compiler • wl-pprint • Better control over whitespace than standard Text.PrettyPrint
  118. Haskell Tooling • hdevtools or ghc-mod with <insert editor here>

    • Warnings • HLint • -Wall • HSpec with nodemon/guard
  119. Summary

  120. Summary • Oden: a functional programming language built for the

    Go ecosystem • The current state of Oden, what’s next • Racket, miniKanren and the Haskell rewrite • Explicit Modelling and Transformations in Haskell • Haskell libraries and tooling
  121. Monads as Design Pattern The Door to Real-World Functional Programming

    Wednesday, April 27 http://www.foocafe.org/event/monads-as-design-pattern
  122. Contribute • Try it out • Discuss • https://groups.google.com/forum/#!forum/oden-lang •

    #oden at Freenode • Report issues on GitHub • Send pull requests
  123. None