How to write your own Go tool

Go tools are very powerful and yet simple to use. But how are Go tools created? In this talk I’m going to answer this question by showing the various Go parser family packages (go/token, go/scanner, go/parser, etc…) and how to use them to create your own Go tool from scratch.

1. How to write your own Go tool Fatih Arslan @DigitalOcean GothamGo - 2016
2. One year ago, I gave a talk about the state of Go tools
3. I wanted to write a tool myself But where do I begin?
4. After reading lots of documentation, code, etc... I wrote a tool called "motion"
5. Some of features in motion: • Jump to function or type declaration • Jump to next or previous function declaration • Select function content (delete/copy/change...) • ...
6. Jump to function declarations
7. Let’s start from the beginning ...
8. The Lexer & Parser Family
9. go/token go/scanner go/parser go/ast go/printer
10. go/token Package token defines constants representing the lexical tokens of the Go programming language and basic operations on tokens (printing, predicates).
14. token.VAR token.IDENT token.IDENT token.ASSIGN token.INT token.ADD token.INT var sum int = 3 + 2
17. go/scanner Package scanner implements a scanner for Go source text. It takes a []byte as source which can then be tokenized through repeated calls to the Scan method.
18. var sum int = 3 + 2 Run through a scanner go/scanner token.VAR token.IDENT token.EOF . . .
19. How do we scan?
20. func main() { src := []byte(`var sum int = 3 + 2`) // Initialize the scanner. var s scanner.Scanner fset := token.NewFileSet() file := fset.AddFile("", fset.Base(), len(src)) s.Init(file, src, nil, scanner.ScanComments) // run the scanner for { pos, tok, lit := s.Scan() if tok == token.EOF { break } fmt.Printf("%s\t%s\t%q\n", fset.Position(pos), tok, lit) } }
21. func main() { src := []byte(`var sum int = 3 + 2`) // Initialize the scanner. var s scanner.Scanner fset := token.NewFileSet() file := fset.AddFile("", fset.Base(), len(src)) s.Init(file, src, nil, scanner.ScanComments) // run the scanner for { pos, tok, lit := s.Scan() if tok == token.EOF { break } fmt.Printf("%s\t%s\t%q\n", fset.Position(pos), tok, lit) } }
22. func main() { src := []byte(`var sum int = 3 + 2`) // Initialize the scanner. var s scanner.Scanner fset := token.NewFileSet() file := fset.AddFile("", fset.Base(), len(src)) s.Init(file, src, nil, scanner.ScanComments) // run the scanner for { pos, tok, lit := s.Scan() if tok == token.EOF { break } fmt.Printf("%s\t%s\t%q\n", fset.Position(pos), tok, lit) } }
23. var sum int = 3 + 2 go/scanner
24. var sum int = 3 + 2 Pos Token Literal 1:1 VAR "var" 1:5 IDENT "sum" 1:9 IDENT "int" 1:13 ASSIGN "=" 1:15 INT "3" 1:19 INT "2" 1:17 ADD "+" 1:20 EOF "" go/scanner
25. go/ast Package ast declares the types used to represent syntax trees for Go packages.
26. ast.Node
27. 3 token.INT token.ADD token.INT + 2
28. *ast.BinaryExpr
29. *ast.BinaryExpr token.INT token.ADD token.INT
30. 3 token.INT token.ADD token.INT + 2 *ast.BinaryExpr
31. 3 + 2 *ast.BinaryExpr
32. 3 + 2 *ast.BinaryExpr *ast.BasicLit 3
33. 3 + 2 *ast.BinaryExpr *ast.BasicLit 2
34. 3 + 2 *ast.BinaryExpr *ast.BasicLit *ast.BasicLit 3 2 +
35. 3 + (7 - 5) *ast.BinaryExpr *ast.BasicLit 3 + *ast.BinaryExpr *ast.BasicLit *ast.BasicLit 5 - 7
36. 3 + (7 - 5) *ast.BinaryExpr *ast.BasicLit 3 + *ast.BinaryExpr *ast.BasicLit *ast.BasicLit 5 - 7
37. var sum int = 3 + 2
39. var sum int = 3 + 2 *ast.ValueSpec
40. var sum int = 3 + 2 *ast.ValueSpec *ast.Ident sum var
41. var sum int = 3 + 2 *ast.ValueSpec *ast.Ident int var
42. var sum int = 3 + 2 *ast.ValueSpec *ast.BinaryExpr *ast.BasicLit *ast.BasicLit 3 2 + var
43. var sum int = 3 + 2 *ast.ValueSpec *ast.Ident *ast.Ident *ast.BinaryExpr *ast.BasicLit *ast.BasicLit sum int 3 2 + var =
44. go/parser Package parser implements a parser for Go source files. Input may be provided in a variety of forms; the output is an abstract syntax tree (AST) representing the Go source.
45. var sum int = 3 + 2 go/parser *ast.ValueSpec *ast.Ident *ast.Ident *ast.BinaryExpr *ast.BasicLit *ast.BasicLit
46. var sum int = 3 + 2 go/parser *ast.ValueSpec *ast.Ident *ast.Ident *ast.BinaryExpr *ast.BasicLit *ast.BasicLit go/scanner go/token
47. How do we parse?
48. func main() { src := []byte(`package main var sum int = 3 + 2`) fset := token.NewFileSet() node, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Fprint(os.Stdout, fset, node, nil) }
49. func main() { src := []byte(`package main var sum int = 3 + 2`) fset := token.NewFileSet() node, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Fprint(os.Stdout, fset, node, nil) }
50. func main() { src := []byte(`package main var sum int = 3 + 2`) fset := token.NewFileSet() node, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Fprint(os.Stdout, fset, node, nil) }
52. How do we traverse the tree?
53. Let us find a line and column number package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) }
54. package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) } *ast.BinaryExpr Let us find a line and column number
55. src := []byte(`package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) }`) fset := token.NewFileSet() node, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } // start searching for the main() function declaration for _, decl := range node.Decls { ...
56. src := []byte(`package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) }`) fset := token.NewFileSet() node, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } // start searching for the main() function declaration for _, decl := range node.Decls { ...
57. src := []byte(`package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) }`) fset := token.NewFileSet() node, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } // start searching for the main() function declaration for _, decl := range node.Decls { ... package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) } 1 2 3 4 5 6 7 8
58. for _, decl := range node.Decls { // search for main function fn, ok := decl.(*ast.FuncDecl) if !ok { continue } // inside the main function body for _, stmt := range fn.Body.List { // search through statements for a declaration... declStmt, ok := stmt.(*ast.DeclStmt) if !ok { continue } // continue with declStmt.Decl // ... } } package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) } 1 2 3 4 5 6 7 8
59. for _, decl := range node.Decls { // search for main function fn, ok := decl.(*ast.FuncDecl) if !ok { continue } // inside the main function body for _, stmt := range fn.Body.List { // search through statements for a declaration... declStmt, ok := stmt.(*ast.DeclStmt) if !ok { continue } // continue with declStmt.Decl // ... } } package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) } 1 2 3 4 5 6 7 8
60. // continue with declStmt.Decl genDecl, ok := declStmt.Decl.(*ast.GenDecl) if !ok { continue } // declarations can have multiple specs, // search for a valuespec for _, spec := range genDecl.Specs { valueSpec, ok := spec.(*ast.ValueSpec) if !ok { continue } // continue with valueSpec.Values // ... } package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) } 1 2 3 4 5 6 7 8
61. // continue with declStmt.Decl genDecl, ok := declStmt.Decl.(*ast.GenDecl) if !ok { continue } // declarations can have multiple specs, // search for a valuespec for _, spec := range genDecl.Specs { valueSpec, ok := spec.(*ast.ValueSpec) if !ok { continue } // continue with valueSpec.Values // ... } package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) } 1 2 3 4 5 6 7 8
62. // continue with valueSpec.Values for _, expr := range valueSpec.Values { // search for a binary expr binaryExpr, ok := expr.(*ast.BinaryExpr) if !ok { continue } // found it! fmt.Printf("Found binary expression at: %d:%d\n", fset.Position(binaryExpr.Pos()).Line, fset.Position(binaryExpr.Pos()).Column, ) } package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) } 1 2 3 4 5 6 7 8
63. // continue with valueSpec.Values for _, expr := range valueSpec.Values { // search for a binary expr binaryExpr, ok := expr.(*ast.BinaryExpr) if !ok { continue } // found it! fmt.Printf("Found binary expression at: %d:%d\n", fset.Position(binaryExpr.Pos()).Line, fset.Position(binaryExpr.Pos()).Column, ) } package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) } 1 2 3 4 5 6 7 8
64. // continue with valueSpec.Values for _, expr := range valueSpec.Values { // search for a binary expr binaryExpr, ok := expr.(*ast.BinaryExpr) if !ok { continue } // found it! fmt.Printf("Found binary expression at: %d:%d\n", fset.Position(binaryExpr.Pos()).Line, fset.Position(binaryExpr.Pos()).Column, ) } package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) } Output: Found binary expression at: 6:17 1 2 3 4 5 6 7 8
65. This is a tedious process
66. Is there an easier way?
67. Is there an easier way? Yes!
68. ast.Inspect()
69. visitorFunc := func(n ast.Node) bool { binaryExpr, ok := n.(*ast.BinaryExpr) if !ok { return true } fmt.Printf("Found binary expression at: %d:%d\n", fset.Position(binaryExpr.Pos()).Line, fset.Position(binaryExpr.Pos()).Column, ) return true } // walk trough all nodes and run the // given function for each node ast.Inspect(node, visitorFunc)
70. visitorFunc := func(n ast.Node) bool { binaryExpr, ok := n.(*ast.BinaryExpr) if !ok { return true } fmt.Printf("Found binary expression at: %d:%d\n", fset.Position(binaryExpr.Pos()).Line, fset.Position(binaryExpr.Pos()).Column, ) return true } // walk trough all nodes and run the // given function for each node ast.Inspect(node, visitorFunc) Output: Found binary expression at: 6:17
71. ast.Inspect() for the President!
72. go/printer Package printer implements printing of AST nodes.
73. go/ast go/printer package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) }
74. How do we print?
75. func main() { src := []byte(`package main var sum int = 3 + 2`) fset := token.NewFileSet() node, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } printer.Fprint(os.Stdout, fset, node) }
76. func main() { src := []byte(`package main var sum int = 3 + 2`) fset := token.NewFileSet() node, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } printer.Fprint(os.Stdout, fset, node) }
77. func main() { src := []byte(`package main var sum int = 3 + 2`) fset := token.NewFileSet() node, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } printer.Fprint(os.Stdout, fset, node) } Output: package main var sum int = 3 + 2
78. Back to the motion tool
79. Basic rules for a simple tool 1. Read (go/parser) 2. Inspect (go/ast) 3. Write (go/printer) or custom format (JSON, etc...) 4. Create a CLI
80. Case study: motion 1. Read (go/parser) 2. Inspect (go/ast) 3. Write (go/printer) or custom format (JSON, etc...) 4. Create a CLI
81. func NewParser(opts *ParserOptions) (*Parser, error) { // ... switch { case opts.File != "": p.file, err = parser.ParseFile(fset, opts.File, nil, mode) case opts.Dir != "": p.pkgs, err = parser.ParseDir(fset, opts.Dir, nil, mode) case opts.Src != nil: p.file, err = parser.ParseFile(fset, "src.go", opts.Src, mode) default: return nil, errors.New("file, src or dir is not specified") } return p, nil }
82. func NewParser(opts *ParserOptions) (*Parser, error) { // ... switch { case opts.File != "": p.file, err = parser.ParseFile(fset, opts.File, nil, mode) case opts.Dir != "": p.pkgs, err = parser.ParseDir(fset, opts.Dir, nil, mode) case opts.Src != nil: p.file, err = parser.ParseFile(fset, "src.go", opts.Src, mode) default: return nil, errors.New("file, src or dir is not specified") } return p, nil }
83. func NewParser(opts *ParserOptions) (*Parser, error) { // ... switch { case opts.File != "": p.file, err = parser.ParseFile(fset, opts.File, nil, mode) case opts.Dir != "": p.pkgs, err = parser.ParseDir(fset, opts.Dir, nil, mode) case opts.Src != nil: p.file, err = parser.ParseFile(fset, "src.go", opts.Src, mode) default: return nil, errors.New("file, src or dir is not specified") } return p, nil }
84. func NewParser(opts *ParserOptions) (*Parser, error) { // ... switch { case opts.File != "": p.file, err = parser.ParseFile(fset, opts.File, nil, mode) case opts.Dir != "": p.pkgs, err = parser.ParseDir(fset, opts.Dir, nil, mode) case opts.Src != nil: p.file, err = parser.ParseFile(fset, "src.go", opts.Src, mode) default: return nil, errors.New("file, src or dir is not specified") } return p, nil }
85. Case study: motion 1. Read (go/parser) 2. Inspect (go/ast) 3. Write (go/printer) or custom format (JSON, etc...) 4. Create a CLI
86. func (p *Parser) collectFunctions() []ast.Node { var funcs []*ast.FuncDecl visitorFunc := func(n ast.Node) bool { fn, ok := n.(*ast.FuncDecl) if !ok { return true } funcs = append(funcs, fn) return true } ast.Inspect(p.file, visitorFunc) return funcs }
87. func (p *Parser) collectFunctions() []ast.Node { var funcs []*ast.FuncDecl visitorFunc := func(n ast.Node) bool { fn, ok := n.(*ast.FuncDecl) if !ok { return true } funcs = append(funcs, fn) return true } ast.Inspect(p.file, visitorFunc) return funcs }
88. func (p *Parser) collectFunctions() []ast.Node { var funcs []*ast.FuncDecl visitorFunc := func(n ast.Node) bool { fn, ok := n.(*ast.FuncDecl) if !ok { return true } funcs = append(funcs, fn) return true } ast.Inspect(p.file, visitorFunc) return funcs }
89. func (p *Parser) collectFunctions() []ast.Node { var funcs []*ast.FuncDecl visitorFunc := func(n ast.Node) bool { fn, ok := n.(*ast.FuncDecl) if !ok { return true } funcs = append(funcs, fn) return true } ast.Inspect(p.file, visitorFunc) return funcs }
90. func (p *Parser) collectFunctions() []ast.Node { var funcs []*ast.FuncDecl visitorFunc := func(n ast.Node) bool { fn, ok := n.(*ast.FuncDecl) if !ok { return true } funcs = append(funcs, fn) return true } ast.Inspect(p.file, visitorFunc) return funcs }
91. Case study: motion 1. Read (go/parser) 2. Inspect (go/ast) 3. Write (go/printer) or custom format (JSON, etc...) 4. Create a CLI
92. func formatFunctions(funcs []*ast.FuncDecl) string { out := new(bytes.Buffer) for _, fn := range funcs { fnPos := p.fset.Position(fn.Type.Func) // foo.go:line:column filePos := fmt.Sprintf("%s:%d:%d", fnPos.Filename, fnPos.Line, fnPos.Column) // foo.go:line:column | func name | func signature() fmt.Fprintf(out, "%s | %s | %s\n", filePos, fn.Name.Name, funcSignature(fn)) } return out.String() }
93. func formatFunctions(funcs []*ast.FuncDecl) string { out := new(bytes.Buffer) for _, fn := range funcs { fnPos := p.fset.Position(fn.Type.Func) // foo.go:line:column filePos := fmt.Sprintf("%s:%d:%d", fnPos.Filename, fnPos.Line, fnPos.Column) // foo.go:line:column | func name | func signature() fmt.Fprintf(out, "%s | %s | %s\n", filePos, fn.Name.Name, funcSignature(fn)) } return out.String() }
94. func formatFunctions(funcs []*ast.FuncDecl) string { out := new(bytes.Buffer) for _, fn := range funcs { fnPos := p.fset.Position(fn.Type.Func) // foo.go:line:column filePos := fmt.Sprintf("%s:%d:%d", fnPos.Filename, fnPos.Line, fnPos.Column) // foo.go:line:column | func name | func signature() fmt.Fprintf(out, "%s | %s | %s\n", filePos, fn.Name.Name, funcSignature(fn)) } return out.String() }
95. func formatFunctions(funcs []*ast.FuncDecl) string { out := new(bytes.Buffer) for _, fn := range funcs { fnPos := p.fset.Position(fn.Type.Func) // foo.go:line:column filePos := fmt.Sprintf("%s:%d:%d", fnPos.Filename, fnPos.Line, fnPos.Column) // foo.go:line:column | func name | func signature() fmt.Fprintf(out, "%s | %s | %s\n", filePos, fn.Name.Name, funcSignature(fn)) } return out.String() }
96. Case study: motion 1. Read (go/parser) 2. Inspect (go/ast) 3. Write (go/printer) or custom format (JSON, etc...) 4. Create a CLI
97. func main() { flagFile = flag.String("file", "", "Filename to be parsed") flagMode = flag.String("mode", "", "Running mode") flag.Parse() opts := &ParserOptions{File: *flagFile} parser, err := NewParser(opts) if err != nil { panic(err) } if *flagMode == "funcs" { funcs := parser.collectFunctions() out := formatFunctions(funcs) fmt.Println(out) } // ... }
98. func main() { flagFile = flag.String("file", "", "Filename to be parsed") flagMode = flag.String("mode", "", "Running mode") flag.Parse() opts := &ParserOptions{File: *flagFile} parser, err := NewParser(opts) if err != nil { panic(err) } if *flagMode == "funcs" { funcs := parser.collectFunctions() out := formatFunctions(funcs) fmt.Println(out) } // ... }
99. func main() { flagFile = flag.String("file", "", "Filename to be parsed") flagMode = flag.String("mode", "", "Running mode") flag.Parse() opts := &ParserOptions{File: *flagFile} parser, err := NewParser(opts) if err != nil { panic(err) } if *flagMode == "funcs" { funcs := parser.collectFunctions() out := formatFunctions(funcs) fmt.Println(out) } // ... }
100. func main() { flagFile = flag.String("file", "", "Filename to be parsed") flagMode = flag.String("mode", "", "Running mode") flag.Parse() opts := &ParserOptions{File: *flagFile} parser, err := NewParser(opts) if err != nil { panic(err) } if *flagMode == "funcs" { funcs := parser.collectFunctions() out := formatFunctions(funcs) fmt.Println(out) } // ... }
101. $ cat main.go package main import "fmt" func main() { a := 5 incr := func(x int) int { fmt.Printf("incremeting %d by one\n", x) return x + 1 } b := incr(a) fmt.Println(b, check(b)) } // check checks if the given integer // is smaller than 10 func check(a int) bool { return a < 10 } // decr decrements the given integer func decr(x int) int { return x - 1 }
102. $ motion -mode funcs -file main.go main.go:5:1 | main | func main() main.go:19:1 | check | func check(a int) bool main.go:24:1 | decr | func decr(x int) int
105. Another case study gofmt
106. Case study: gofmt 1. Read (go/parser) 2. Inspect (go/ast) 3. Write (go/printer) or custom format (JSON, etc...) 4. Create a CLI
107. Case study: gofmt package main import "fmt" func main(){ var sum int = 3+2 fmt.Println( sum ) }
108. Case study: gofmt go/parser package main import "fmt" func main(){ var sum int = 3+2 fmt.Println( sum ) }
109. Case study: gofmt go/parser go/ast package main import "fmt" func main(){ var sum int = 3+2 fmt.Println( sum ) }
110. func processFile(filename string, in io.Reader, out io.Writer, stdin bool) error { // ... src, err := ioutil.ReadAll(in) if err != nil { return err } file, sourceAdj, indentAdj, err := parse(fileSet, filename, src, stdin) if err != nil { return err } // ...
111. func processFile(filename string, in io.Reader, out io.Writer, stdin bool) error { // ... src, err := ioutil.ReadAll(in) if err != nil { return err } file, sourceAdj, indentAdj, err := parse(fileSet, filename, src, stdin) if err != nil { return err } // ...
112. Case study: gofmt 1. Read (go/parser) 2. Inspect (go/ast) 3. Write (go/printer) or custom format (JSON, etc...) 4. Create a CLI
113. Case study: gofmt go/parser go/ast sort simplify rewrite package main import "fmt" func main(){ var sum int = 3+2 fmt.Println( sum ) }
114. func processFile(filename string, in io.Reader, out io.Writer, stdin bool) error { // ... if rewrite != nil { if sourceAdj == nil { file = rewrite(file) } else { fmt.Fprintf(os.Stderr, "warning: rewrite ignored for incomplete programs\n") } } ast.SortImports(fileSet, file) if *simplifyAST { simplify(file) } // ...
115. Case study: gofmt 1. Read (go/parser) 2. Inspect (go/ast) 3. Write (go/printer) or custom format (JSON, etc...) 4. Create a CLI
116. Case study: gofmt go/parser go/ast sort simplify rewrite go/printer package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) } package main import "fmt" func main(){ var sum int = 3+2 fmt.Println( sum ) }
117. func processFile(filename string, in io.Reader, out io.Writer, stdin bool) error { // ... res, err := format(fileSet, file, sourceAdj, indentAdj, src, printer.Config{Mode: printerMode, Tabwidth: tabWidth}) if err != nil { return err } if !*list && !*write && !*doDiff { _, err = out.Write(res) } // ...
118. Case study: gofmt 1. Read (go/parser) 2. Inspect (go/ast) 3. Write (go/printer) or custom format (JSON, etc...) 4. Create a CLI
119. func main() { // ... for i := 0; i < flag.NArg(); i++ { path := flag.Arg(i) switch dir, err := os.Stat(path); { case err != nil: report(err) case dir.IsDir(): walkDir(path) default: if err := processFile(path, nil, os.Stdout, false); err != nil { report(err) } } } // ... }
120. $ cat main.go package main import "fmt" func main() { var sum int = 3+2 fmt.Println( sum ) }
121. $ cat main.go package main import "fmt" func main() { var sum int = 3+2 fmt.Println( sum ) } $ gofmt main.go package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) }
122. Recap
123. package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) } go/parser go/ast go/printer package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) }
124. package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) } go/ast package main import "fmt" func main() { var sum int = 3 + 2 fmt.Println(sum) } go/printer go/parser go/scanner go/token go/parser
125. Next steps
126. Source: https://talks.golang.org/2014/static-analysis.slide#8
127. Thanks Fatih Arslan fatih fatih fatih@digitalocean.com