Gore: A Tale of Go REPL

Go Conference Summer 2015

Transcript

1. A Tale of Go REPL
   motemen
   Go Conference 2015 Summer @ Tokyo
   

   View Slide

2. About Me
   - @motemen
   - Web engineer
   - Chief engineer at Hatena, Kyoto
   

   View Slide

3. Mackerel mackerel.io
   

   View Slide

4. Agenda
   - Introduction of ! motemen/gore
   - Gore’s architecture
   - Implementation details
   

   View Slide

5. Gore /gɔː/
   

   View Slide

6. Case: Learning New Languages
   - Playing with language features
   - Trying libraries
   

   View Slide

7. My Case — Perl
   - perl ~/sketch.pl
   - Edit and Run
   

   View Slide

8. My Case — Ruby
   - irb
   - pry
   - A “REPL”

   (Read-Eval-Print-Loop)
   

   View Slide

9. My Case — Go?
   - tmp.go
   - “packge main” “func main()”
   - Removing unused variables by hand
   - Question: Can gophers haz a REPL?
   

   View Slide

10. Go "REPL" impls (AFAIK)
    - ! sbinet/igo — built upon ! sbinet/go-eval, with liner
    - ! emicklei/rango — importing, statements, listing variables
    - ! vito/go-repl — importing, statements, useful commands
    - ! d4l3k/go-pry — attaches to running program, uses reflection
    - (play.golang.org)
    

    View Slide

11. Needed a REPL that:
    - Evaluates statements
    - Imports any module
    - Pretty prints values automatically
    - Completes code
    

    View Slide

12. The Difficulties
    - How can we achieve Go semantics?
    - No APIs provided
    - Re-implement Go compiler?
    

    View Slide

13. Solution: “go run”
    

    View Slide

14. Solution: “go run”
    - An “eval” to Go
    - Perfect implementation & Always up-to-date
    - Dead simple
    - In: a program Out: the result
    

    View Slide

15. Gore: A REPL using “go run”
    go get github.com/motemen/gore
    

    View Slide

16. Caveats
    - Gore runs all code input for each run
    - Code with side effects will be repeated
    - eg. Printing output / Sleeping
    

    View Slide

17. Stories Inside Gore
    

    View Slide

18. Overview: not a “REPL” exactly
    - REPL: Read-Eval-Print-Loop
    - No “Eval” and “Print” phase
    - As we do them by “go run”
    - Much like: Read-Generate-Run-Loop
    

    View Slide

19. Read-Gen-Run-Loop
    1. Read user input to evaluate
    2. Generate a .go file that prints result
    3. “go run” it
    4. Back to 1.
    

    View Slide

20. Step: Read
    - Codes go under AST (abstract syntax tree) form
    - Syntax checking
    - Easier to manipulate
    - Input String → AST → .go → “go run”
    

    View Slide

21. Read: Input
    - ! peterh/liner
    - Line editing
    - History
    - Supports Windows
    

    View Slide

22. Read: Input to AST
    - package go/parser
    - Go has rich support for parsing/typing its code
    - Easy to generate & manipulate
    

    View Slide

23. Read: Input to AST
    - Parse input as an expression
    - If failed: a statement
    - Otherwise: continue input (multi-line)
    

    View Slide

24. Read: Expression
    - Easy
    - Dedicated API
    - parser.ParseExpr(in)
    

    View Slide

25. Read: Statements
    - $
    func (s *Session) evalStmt(in string) error {
    src := fmt.Sprintf("package P; func F() { %s }", in)
    f, err := parser.ParseFile(s.Fset, "stmt.go", src, 0)
    if err != nil {
    return err
    }
    enclosingFunc := f.Scope.Lookup("F").Decl.
    (*ast.FuncDecl)
    stmts := enclosingFunc.Body.List
    }
    

    View Slide

26. Generate: The Delicious Part
    - Append the input AST nodes to our main()
    - Add the code to print the value
    

    View Slide

27. Printing Values
    - The generated program prints values (not gore)
    - “New Values” should be printed
    - Expression: Its resulting value
    - Statement: Values assigned
    

    View Slide

28. Printing Values: Expression
    - User: foo.Bar(a+1)
    - Gore: __gore_p(foo.Bar(a+1))
    

    View Slide

29. Printing Values: Statements
    - ast.AssignStmt stands for assign/define
    - User: a, b := foo()
    - Gore: a, b := foo(); __gore_p(a, b)
    

    View Slide

30. Printing Values: __gore_p()
    - “Pretty prints” values
    - Defined along with the main()
    - Depends on installed packages
    - ! k0kubun/pp, ! davecgh/go-spew or plain old %#v
    

    View Slide

31. The Initial .go File (pp)
    package main
    import "github.com/k0kubun/pp"
    func __gore_p(xx ...interface{}) {
    for _, x := range xx {
    pp.Println(x)
    }
    }
    func main() {
    // User input goes here
    }
    

    View Slide

32. The Initial .go File (go-spew)
    package main
    import "github.com/davecgh/go-spew/spew"
    func __gore_p(xx ...interface{}) {
    for _, x := range xx {
    spew.Printf("%#v\n", x)
    }
    }
    func main() {
    // User input goes here
    }
    

    View Slide

33. The Initial .go File (%#v)
    package main
    import "fmt"
    func __gore_p(xx ...interface{}) {
    for _, x := range xx {
    fmt.Printf("%#v\n", x)
    }
    }
    func main() {
    // User input goes here
    }
    

    View Slide

34. Generate: Append to main()
    - Just append those generated statements

    s.mainBody.List = append(s.mainBody.List, stmts…)
    - Now we’ve got the working code!
    - Really? No! %
    

    View Slide

35. Go compiler complaints (you know)
    - “x declared but not used”
    - “p imported but not used”
    - “no new variables of left side of :=“
    - & Should remove these to a successful go run
    

    View Slide

36. Quick Fixing Erroneous Program
    - “x declared but not used”
    - “p imported but not used”
    - “no new variables on left side of :=“
    Use it!!
    Anonymize it!!
    Make it `=‘ !!
    

    View Slide

37. “declared but not used”
    package P
    func main() {
    s := "hello"
    }
    package P
    func main() {
    s := "hello"
    _ = s
    }
    

    View Slide

38. ”imported but not used”
    package P
    import "fmt"
    func main() {
    }
    package P
    import _ "fmt"
    func main() {
    }
    

    View Slide

39. “no new variables on left side of :=“
    package P
    func main() {
    var a int
    a := 1
    }
    package P
    func main() {
    var a int
    a = 1
    }
    

    View Slide

40. ! motemen/go-quickfix to do the work
    - Type check source code using go/types.Check()
    - Catch errors and modify AST
    - Packed with a bin
    - goquickfix -w main.go
    

    View Slide

41. Gore specific quickfix
    - __gore_p() for non-value expressions
    

    View Slide

42. Running
    - Output the AST to file — go/printer
    - Then go run
    - If failed to run, revert the last input
    

    View Slide

43. Recap: Read-Gen-Quickfix-Run-Loop
    1. Read and parse the input to obtain AST
    2. Add printing function call __gore_p()
    3. Append the code to main()
    4. Quickfix it so that it compiles well
    5. go run
    

    View Slide

44. PRs welcome!
    - ! motemen/gore
    - ! motemen/go-quickfix
    

    View Slide

45. htn.to/intern2015
    

    View Slide

46. A Tale Gore
    motemen
    Go Conference 2015 Summer @ Tokyo
    

    View Slide

47. Appendix:
    More Features
    

    View Slide

48. Code Completion
    - liner has support for completion
    - Great tool for editors: ! nsf/gocode
    - Server/client model
    - And an interface to it: motemen/gore/gocode
    - If gocode binary can be located, use it
    

    View Slide

49. Commands
    - :print
    - :write <file>
    - :import
    - :doc
    

    View Slide

50. :import
    - Imports arbitrary packages
    - Just append import statement to the file
    - Unused imports are handled by quickfix
    

    View Slide

51. :doc
    - Invokes go doc for given expression
    - Package: :doc json
    - Function: :doc json.Marshal
    - Method: :doc json.NewEncoder().Encode
    

    View Slide