Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to write your own Go tool

Fatih Arslan
November 18, 2016

How to write your own Go tool

Go tools are very powerful and yet simple to use. But how are Go tools created? In this talk I’m going to answer this question by showing the various Go parser family packages (go/token, go/scanner, go/parser, etc…) and how to use them to create your own Go tool from scratch.

Fatih Arslan

November 18, 2016
Tweet

More Decks by Fatih Arslan

Other Decks in Programming

Transcript

  1. How to write your own Go tool
    Fatih Arslan @DigitalOcean
    GothamGo - 2016

    View full-size slide

  2. One year ago, I gave a talk
    about the state of Go tools

    View full-size slide

  3. I wanted to write a tool myself
    But where do I begin?

    View full-size slide

  4. After reading lots of
    documentation, code, etc...
    I wrote a tool called "motion"

    View full-size slide

  5. Some of features in motion:
    • Jump to function or type declaration
    • Jump to next or previous function declaration
    • Select function content (delete/copy/change...)
    • ...

    View full-size slide

  6. Jump to
    function declarations

    View full-size slide

  7. Let’s start from the beginning ...

    View full-size slide

  8. The Lexer & Parser
    Family

    View full-size slide

  9. go/token go/scanner
    go/parser
    go/ast
    go/printer

    View full-size slide

  10. go/token
    Package token defines constants representing the lexical tokens of
    the Go programming language and basic operations on tokens
    (printing, predicates).

    View full-size slide

  11. token.VAR
    token.IDENT
    token.IDENT
    token.ASSIGN
    token.INT
    token.ADD
    token.INT
    var sum int = 3 + 2

    View full-size slide

  12. go/scanner
    Package scanner implements a scanner for Go source text. It
    takes a []byte as source which can then be tokenized through
    repeated calls to the Scan method.

    View full-size slide

  13. var sum int = 3 + 2
    Run through a scanner
    go/scanner
    token.VAR
    token.IDENT
    token.EOF
    .
    .
    .

    View full-size slide

  14. How do we scan?

    View full-size slide

  15. func main() {
    src := []byte(`var sum int = 3 + 2`)
    // Initialize the scanner.
    var s scanner.Scanner
    fset := token.NewFileSet()
    file := fset.AddFile("", fset.Base(), len(src))
    s.Init(file, src, nil, scanner.ScanComments)
    // run the scanner
    for {
    pos, tok, lit := s.Scan()
    if tok == token.EOF {
    break
    }
    fmt.Printf("%s\t%s\t%q\n", fset.Position(pos), tok, lit)
    }
    }

    View full-size slide

  16. func main() {
    src := []byte(`var sum int = 3 + 2`)
    // Initialize the scanner.
    var s scanner.Scanner
    fset := token.NewFileSet()
    file := fset.AddFile("", fset.Base(), len(src))
    s.Init(file, src, nil, scanner.ScanComments)
    // run the scanner
    for {
    pos, tok, lit := s.Scan()
    if tok == token.EOF {
    break
    }
    fmt.Printf("%s\t%s\t%q\n", fset.Position(pos), tok, lit)
    }
    }

    View full-size slide

  17. func main() {
    src := []byte(`var sum int = 3 + 2`)
    // Initialize the scanner.
    var s scanner.Scanner
    fset := token.NewFileSet()
    file := fset.AddFile("", fset.Base(), len(src))
    s.Init(file, src, nil, scanner.ScanComments)
    // run the scanner
    for {
    pos, tok, lit := s.Scan()
    if tok == token.EOF {
    break
    }
    fmt.Printf("%s\t%s\t%q\n", fset.Position(pos), tok, lit)
    }
    }

    View full-size slide

  18. var sum int = 3 + 2
    go/scanner

    View full-size slide

  19. var sum int = 3 + 2 Pos Token Literal
    1:1 VAR "var"
    1:5 IDENT "sum"
    1:9 IDENT "int"
    1:13 ASSIGN "="
    1:15 INT "3"
    1:19 INT "2"
    1:17 ADD "+"
    1:20 EOF ""
    go/scanner

    View full-size slide

  20. go/ast
    Package ast declares the types used to represent syntax
    trees for Go packages.

    View full-size slide

  21. 3
    token.INT
    token.ADD
    token.INT
    + 2

    View full-size slide

  22. *ast.BinaryExpr

    View full-size slide

  23. *ast.BinaryExpr
    token.INT
    token.ADD
    token.INT

    View full-size slide

  24. 3
    token.INT
    token.ADD
    token.INT
    + 2
    *ast.BinaryExpr

    View full-size slide

  25. 3 + 2
    *ast.BinaryExpr

    View full-size slide

  26. 3 + 2
    *ast.BinaryExpr
    *ast.BasicLit
    3

    View full-size slide

  27. 3 + 2
    *ast.BinaryExpr
    *ast.BasicLit
    2

    View full-size slide

  28. 3 + 2
    *ast.BinaryExpr
    *ast.BasicLit *ast.BasicLit
    3 2
    +

    View full-size slide

  29. 3 + (7 - 5)
    *ast.BinaryExpr
    *ast.BasicLit
    3
    +
    *ast.BinaryExpr
    *ast.BasicLit *ast.BasicLit
    5
    -
    7

    View full-size slide

  30. 3 + (7 - 5)
    *ast.BinaryExpr
    *ast.BasicLit
    3
    +
    *ast.BinaryExpr
    *ast.BasicLit *ast.BasicLit
    5
    -
    7

    View full-size slide

  31. var sum int = 3 + 2

    View full-size slide

  32. var sum int = 3 + 2
    *ast.ValueSpec

    View full-size slide

  33. var sum int = 3 + 2
    *ast.ValueSpec
    *ast.Ident
    sum
    var

    View full-size slide

  34. var sum int = 3 + 2
    *ast.ValueSpec
    *ast.Ident
    int
    var

    View full-size slide

  35. var sum int = 3 + 2
    *ast.ValueSpec
    *ast.BinaryExpr
    *ast.BasicLit *ast.BasicLit
    3 2
    +
    var

    View full-size slide

  36. var sum int = 3 + 2
    *ast.ValueSpec
    *ast.Ident *ast.Ident *ast.BinaryExpr
    *ast.BasicLit *ast.BasicLit
    sum int
    3 2
    +
    var
    =

    View full-size slide

  37. go/parser
    Package parser implements a parser for Go source files.
    Input may be provided in a variety of forms; the output is an
    abstract syntax tree (AST) representing the Go source.

    View full-size slide

  38. var sum int = 3 + 2
    go/parser
    *ast.ValueSpec
    *ast.Ident *ast.Ident
    *ast.BinaryExpr
    *ast.BasicLit *ast.BasicLit

    View full-size slide

  39. var sum int = 3 + 2
    go/parser
    *ast.ValueSpec
    *ast.Ident *ast.Ident
    *ast.BinaryExpr
    *ast.BasicLit *ast.BasicLit
    go/scanner
    go/token

    View full-size slide

  40. How do we parse?

    View full-size slide

  41. func main() {
    src := []byte(`package main
    var sum int = 3 + 2`)
    fset := token.NewFileSet()
    node, err := parser.ParseFile(fset, "demo", src, parser.ParseComments)
    if err != nil {
    panic(err)
    }
    ast.Fprint(os.Stdout, fset, node, nil)
    }

    View full-size slide

  42. func main() {
    src := []byte(`package main
    var sum int = 3 + 2`)
    fset := token.NewFileSet()
    node, err := parser.ParseFile(fset, "demo", src, parser.ParseComments)
    if err != nil {
    panic(err)
    }
    ast.Fprint(os.Stdout, fset, node, nil)
    }

    View full-size slide

  43. func main() {
    src := []byte(`package main
    var sum int = 3 + 2`)
    fset := token.NewFileSet()
    node, err := parser.ParseFile(fset, "demo", src, parser.ParseComments)
    if err != nil {
    panic(err)
    }
    ast.Fprint(os.Stdout, fset, node, nil)
    }

    View full-size slide

  44. How do we traverse the tree?

    View full-size slide

  45. Let us find a line and column number
    package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }

    View full-size slide

  46. package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }
    *ast.BinaryExpr
    Let us find a line and column number

    View full-size slide

  47. src := []byte(`package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }`)
    fset := token.NewFileSet()
    node, err := parser.ParseFile(fset, "demo", src, parser.ParseComments)
    if err != nil {
    panic(err)
    }
    // start searching for the main() function declaration
    for _, decl := range node.Decls {
    ...

    View full-size slide

  48. src := []byte(`package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }`)
    fset := token.NewFileSet()
    node, err := parser.ParseFile(fset, "demo", src, parser.ParseComments)
    if err != nil {
    panic(err)
    }
    // start searching for the main() function declaration
    for _, decl := range node.Decls {
    ...

    View full-size slide

  49. src := []byte(`package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }`)
    fset := token.NewFileSet()
    node, err := parser.ParseFile(fset, "demo", src, parser.ParseComments)
    if err != nil {
    panic(err)
    }
    // start searching for the main() function declaration
    for _, decl := range node.Decls {
    ...
    package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }
    1
    2
    3
    4
    5
    6
    7
    8

    View full-size slide

  50. for _, decl := range node.Decls {
    // search for main function
    fn, ok := decl.(*ast.FuncDecl)
    if !ok {
    continue
    }
    // inside the main function body
    for _, stmt := range fn.Body.List {
    // search through statements for a declaration...
    declStmt, ok := stmt.(*ast.DeclStmt)
    if !ok {
    continue
    }
    // continue with declStmt.Decl
    // ...
    }
    }
    package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }
    1
    2
    3
    4
    5
    6
    7
    8

    View full-size slide

  51. for _, decl := range node.Decls {
    // search for main function
    fn, ok := decl.(*ast.FuncDecl)
    if !ok {
    continue
    }
    // inside the main function body
    for _, stmt := range fn.Body.List {
    // search through statements for a declaration...
    declStmt, ok := stmt.(*ast.DeclStmt)
    if !ok {
    continue
    }
    // continue with declStmt.Decl
    // ...
    }
    }
    package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }
    1
    2
    3
    4
    5
    6
    7
    8

    View full-size slide

  52. // continue with declStmt.Decl
    genDecl, ok := declStmt.Decl.(*ast.GenDecl)
    if !ok {
    continue
    }
    // declarations can have multiple specs,
    // search for a valuespec
    for _, spec := range genDecl.Specs {
    valueSpec, ok := spec.(*ast.ValueSpec)
    if !ok {
    continue
    }
    // continue with valueSpec.Values
    // ...
    }
    package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }
    1
    2
    3
    4
    5
    6
    7
    8

    View full-size slide

  53. // continue with declStmt.Decl
    genDecl, ok := declStmt.Decl.(*ast.GenDecl)
    if !ok {
    continue
    }
    // declarations can have multiple specs,
    // search for a valuespec
    for _, spec := range genDecl.Specs {
    valueSpec, ok := spec.(*ast.ValueSpec)
    if !ok {
    continue
    }
    // continue with valueSpec.Values
    // ...
    }
    package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }
    1
    2
    3
    4
    5
    6
    7
    8

    View full-size slide

  54. // continue with valueSpec.Values
    for _, expr := range valueSpec.Values {
    // search for a binary expr
    binaryExpr, ok := expr.(*ast.BinaryExpr)
    if !ok {
    continue
    }
    // found it!
    fmt.Printf("Found binary expression at: %d:%d\n",
    fset.Position(binaryExpr.Pos()).Line,
    fset.Position(binaryExpr.Pos()).Column,
    )
    }
    package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }
    1
    2
    3
    4
    5
    6
    7
    8

    View full-size slide

  55. // continue with valueSpec.Values
    for _, expr := range valueSpec.Values {
    // search for a binary expr
    binaryExpr, ok := expr.(*ast.BinaryExpr)
    if !ok {
    continue
    }
    // found it!
    fmt.Printf("Found binary expression at: %d:%d\n",
    fset.Position(binaryExpr.Pos()).Line,
    fset.Position(binaryExpr.Pos()).Column,
    )
    }
    package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }
    1
    2
    3
    4
    5
    6
    7
    8

    View full-size slide

  56. // continue with valueSpec.Values
    for _, expr := range valueSpec.Values {
    // search for a binary expr
    binaryExpr, ok := expr.(*ast.BinaryExpr)
    if !ok {
    continue
    }
    // found it!
    fmt.Printf("Found binary expression at: %d:%d\n",
    fset.Position(binaryExpr.Pos()).Line,
    fset.Position(binaryExpr.Pos()).Column,
    )
    }
    package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }
    Output:
    Found binary expression at: 6:17
    1
    2
    3
    4
    5
    6
    7
    8

    View full-size slide

  57. This is a tedious process

    View full-size slide

  58. Is there an easier way?

    View full-size slide

  59. Is there an easier way?
    Yes!

    View full-size slide

  60. ast.Inspect()

    View full-size slide

  61. visitorFunc := func(n ast.Node) bool {
    binaryExpr, ok := n.(*ast.BinaryExpr)
    if !ok {
    return true
    }
    fmt.Printf("Found binary expression at: %d:%d\n",
    fset.Position(binaryExpr.Pos()).Line,
    fset.Position(binaryExpr.Pos()).Column,
    )
    return true
    }
    // walk trough all nodes and run the
    // given function for each node
    ast.Inspect(node, visitorFunc)

    View full-size slide

  62. visitorFunc := func(n ast.Node) bool {
    binaryExpr, ok := n.(*ast.BinaryExpr)
    if !ok {
    return true
    }
    fmt.Printf("Found binary expression at: %d:%d\n",
    fset.Position(binaryExpr.Pos()).Line,
    fset.Position(binaryExpr.Pos()).Column,
    )
    return true
    }
    // walk trough all nodes and run the
    // given function for each node
    ast.Inspect(node, visitorFunc)
    Output:
    Found binary expression at: 6:17

    View full-size slide

  63. ast.Inspect()
    for the President!

    View full-size slide

  64. go/printer
    Package printer implements printing of AST nodes.

    View full-size slide

  65. go/ast go/printer
    package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }

    View full-size slide

  66. How do we print?

    View full-size slide

  67. func main() {
    src := []byte(`package main
    var sum int = 3 + 2`)
    fset := token.NewFileSet()
    node, err := parser.ParseFile(fset, "demo", src, parser.ParseComments)
    if err != nil {
    panic(err)
    }
    printer.Fprint(os.Stdout, fset, node)
    }

    View full-size slide

  68. func main() {
    src := []byte(`package main
    var sum int = 3 + 2`)
    fset := token.NewFileSet()
    node, err := parser.ParseFile(fset, "demo", src, parser.ParseComments)
    if err != nil {
    panic(err)
    }
    printer.Fprint(os.Stdout, fset, node)
    }

    View full-size slide

  69. func main() {
    src := []byte(`package main
    var sum int = 3 + 2`)
    fset := token.NewFileSet()
    node, err := parser.ParseFile(fset, "demo", src, parser.ParseComments)
    if err != nil {
    panic(err)
    }
    printer.Fprint(os.Stdout, fset, node)
    }
    Output:
    package main
    var sum int = 3 + 2

    View full-size slide

  70. Back to the motion tool

    View full-size slide

  71. Basic rules for a simple tool
    1. Read (go/parser)
    2. Inspect (go/ast)
    3. Write (go/printer) or custom format (JSON, etc...)
    4. Create a CLI

    View full-size slide

  72. Case study: motion
    1. Read (go/parser)
    2. Inspect (go/ast)
    3. Write (go/printer) or custom format (JSON, etc...)
    4. Create a CLI

    View full-size slide

  73. func NewParser(opts *ParserOptions) (*Parser, error) {
    // ...
    switch {
    case opts.File != "":
    p.file, err = parser.ParseFile(fset, opts.File, nil, mode)
    case opts.Dir != "":
    p.pkgs, err = parser.ParseDir(fset, opts.Dir, nil, mode)
    case opts.Src != nil:
    p.file, err = parser.ParseFile(fset, "src.go", opts.Src, mode)
    default:
    return nil, errors.New("file, src or dir is not specified")
    }
    return p, nil
    }

    View full-size slide

  74. func NewParser(opts *ParserOptions) (*Parser, error) {
    // ...
    switch {
    case opts.File != "":
    p.file, err = parser.ParseFile(fset, opts.File, nil, mode)
    case opts.Dir != "":
    p.pkgs, err = parser.ParseDir(fset, opts.Dir, nil, mode)
    case opts.Src != nil:
    p.file, err = parser.ParseFile(fset, "src.go", opts.Src, mode)
    default:
    return nil, errors.New("file, src or dir is not specified")
    }
    return p, nil
    }

    View full-size slide

  75. func NewParser(opts *ParserOptions) (*Parser, error) {
    // ...
    switch {
    case opts.File != "":
    p.file, err = parser.ParseFile(fset, opts.File, nil, mode)
    case opts.Dir != "":
    p.pkgs, err = parser.ParseDir(fset, opts.Dir, nil, mode)
    case opts.Src != nil:
    p.file, err = parser.ParseFile(fset, "src.go", opts.Src, mode)
    default:
    return nil, errors.New("file, src or dir is not specified")
    }
    return p, nil
    }

    View full-size slide

  76. func NewParser(opts *ParserOptions) (*Parser, error) {
    // ...
    switch {
    case opts.File != "":
    p.file, err = parser.ParseFile(fset, opts.File, nil, mode)
    case opts.Dir != "":
    p.pkgs, err = parser.ParseDir(fset, opts.Dir, nil, mode)
    case opts.Src != nil:
    p.file, err = parser.ParseFile(fset, "src.go", opts.Src, mode)
    default:
    return nil, errors.New("file, src or dir is not specified")
    }
    return p, nil
    }

    View full-size slide

  77. Case study: motion
    1. Read (go/parser)
    2. Inspect (go/ast)
    3. Write (go/printer) or custom format (JSON, etc...)
    4. Create a CLI

    View full-size slide

  78. func (p *Parser) collectFunctions() []ast.Node {
    var funcs []*ast.FuncDecl
    visitorFunc := func(n ast.Node) bool {
    fn, ok := n.(*ast.FuncDecl)
    if !ok {
    return true
    }
    funcs = append(funcs, fn)
    return true
    }
    ast.Inspect(p.file, visitorFunc)
    return funcs
    }

    View full-size slide

  79. func (p *Parser) collectFunctions() []ast.Node {
    var funcs []*ast.FuncDecl
    visitorFunc := func(n ast.Node) bool {
    fn, ok := n.(*ast.FuncDecl)
    if !ok {
    return true
    }
    funcs = append(funcs, fn)
    return true
    }
    ast.Inspect(p.file, visitorFunc)
    return funcs
    }

    View full-size slide

  80. func (p *Parser) collectFunctions() []ast.Node {
    var funcs []*ast.FuncDecl
    visitorFunc := func(n ast.Node) bool {
    fn, ok := n.(*ast.FuncDecl)
    if !ok {
    return true
    }
    funcs = append(funcs, fn)
    return true
    }
    ast.Inspect(p.file, visitorFunc)
    return funcs
    }

    View full-size slide

  81. func (p *Parser) collectFunctions() []ast.Node {
    var funcs []*ast.FuncDecl
    visitorFunc := func(n ast.Node) bool {
    fn, ok := n.(*ast.FuncDecl)
    if !ok {
    return true
    }
    funcs = append(funcs, fn)
    return true
    }
    ast.Inspect(p.file, visitorFunc)
    return funcs
    }

    View full-size slide

  82. func (p *Parser) collectFunctions() []ast.Node {
    var funcs []*ast.FuncDecl
    visitorFunc := func(n ast.Node) bool {
    fn, ok := n.(*ast.FuncDecl)
    if !ok {
    return true
    }
    funcs = append(funcs, fn)
    return true
    }
    ast.Inspect(p.file, visitorFunc)
    return funcs
    }

    View full-size slide

  83. Case study: motion
    1. Read (go/parser)
    2. Inspect (go/ast)
    3. Write (go/printer) or custom format (JSON, etc...)
    4. Create a CLI

    View full-size slide

  84. func formatFunctions(funcs []*ast.FuncDecl) string {
    out := new(bytes.Buffer)
    for _, fn := range funcs {
    fnPos := p.fset.Position(fn.Type.Func)
    // foo.go:line:column
    filePos := fmt.Sprintf("%s:%d:%d",
    fnPos.Filename, fnPos.Line, fnPos.Column)
    // foo.go:line:column | func name | func signature()
    fmt.Fprintf(out, "%s | %s | %s\n",
    filePos, fn.Name.Name, funcSignature(fn))
    }
    return out.String()
    }

    View full-size slide

  85. func formatFunctions(funcs []*ast.FuncDecl) string {
    out := new(bytes.Buffer)
    for _, fn := range funcs {
    fnPos := p.fset.Position(fn.Type.Func)
    // foo.go:line:column
    filePos := fmt.Sprintf("%s:%d:%d",
    fnPos.Filename, fnPos.Line, fnPos.Column)
    // foo.go:line:column | func name | func signature()
    fmt.Fprintf(out, "%s | %s | %s\n",
    filePos, fn.Name.Name, funcSignature(fn))
    }
    return out.String()
    }

    View full-size slide

  86. func formatFunctions(funcs []*ast.FuncDecl) string {
    out := new(bytes.Buffer)
    for _, fn := range funcs {
    fnPos := p.fset.Position(fn.Type.Func)
    // foo.go:line:column
    filePos := fmt.Sprintf("%s:%d:%d",
    fnPos.Filename, fnPos.Line, fnPos.Column)
    // foo.go:line:column | func name | func signature()
    fmt.Fprintf(out, "%s | %s | %s\n",
    filePos, fn.Name.Name, funcSignature(fn))
    }
    return out.String()
    }

    View full-size slide

  87. func formatFunctions(funcs []*ast.FuncDecl) string {
    out := new(bytes.Buffer)
    for _, fn := range funcs {
    fnPos := p.fset.Position(fn.Type.Func)
    // foo.go:line:column
    filePos := fmt.Sprintf("%s:%d:%d",
    fnPos.Filename, fnPos.Line, fnPos.Column)
    // foo.go:line:column | func name | func signature()
    fmt.Fprintf(out, "%s | %s | %s\n",
    filePos, fn.Name.Name, funcSignature(fn))
    }
    return out.String()
    }

    View full-size slide

  88. Case study: motion
    1. Read (go/parser)
    2. Inspect (go/ast)
    3. Write (go/printer) or custom format (JSON, etc...)
    4. Create a CLI

    View full-size slide

  89. func main() {
    flagFile = flag.String("file", "", "Filename to be parsed")
    flagMode = flag.String("mode", "", "Running mode")
    flag.Parse()
    opts := &ParserOptions{File: *flagFile}
    parser, err := NewParser(opts)
    if err != nil {
    panic(err)
    }
    if *flagMode == "funcs" {
    funcs := parser.collectFunctions()
    out := formatFunctions(funcs)
    fmt.Println(out)
    }
    // ...
    }

    View full-size slide

  90. func main() {
    flagFile = flag.String("file", "", "Filename to be parsed")
    flagMode = flag.String("mode", "", "Running mode")
    flag.Parse()
    opts := &ParserOptions{File: *flagFile}
    parser, err := NewParser(opts)
    if err != nil {
    panic(err)
    }
    if *flagMode == "funcs" {
    funcs := parser.collectFunctions()
    out := formatFunctions(funcs)
    fmt.Println(out)
    }
    // ...
    }

    View full-size slide

  91. func main() {
    flagFile = flag.String("file", "", "Filename to be parsed")
    flagMode = flag.String("mode", "", "Running mode")
    flag.Parse()
    opts := &ParserOptions{File: *flagFile}
    parser, err := NewParser(opts)
    if err != nil {
    panic(err)
    }
    if *flagMode == "funcs" {
    funcs := parser.collectFunctions()
    out := formatFunctions(funcs)
    fmt.Println(out)
    }
    // ...
    }

    View full-size slide

  92. func main() {
    flagFile = flag.String("file", "", "Filename to be parsed")
    flagMode = flag.String("mode", "", "Running mode")
    flag.Parse()
    opts := &ParserOptions{File: *flagFile}
    parser, err := NewParser(opts)
    if err != nil {
    panic(err)
    }
    if *flagMode == "funcs" {
    funcs := parser.collectFunctions()
    out := formatFunctions(funcs)
    fmt.Println(out)
    }
    // ...
    }

    View full-size slide

  93. $ cat main.go
    package main
    import "fmt"
    func main() {
    a := 5
    incr := func(x int) int {
    fmt.Printf("incremeting %d by one\n", x)
    return x + 1
    }
    b := incr(a)
    fmt.Println(b, check(b))
    }
    // check checks if the given integer
    // is smaller than 10
    func check(a int) bool {
    return a < 10
    }
    // decr decrements the given integer
    func decr(x int) int {
    return x - 1
    }

    View full-size slide

  94. $ motion -mode funcs -file main.go
    main.go:5:1 | main | func main()
    main.go:19:1 | check | func check(a int) bool
    main.go:24:1 | decr | func decr(x int) int

    View full-size slide

  95. Another case study
    gofmt

    View full-size slide

  96. Case study: gofmt
    1. Read (go/parser)
    2. Inspect (go/ast)
    3. Write (go/printer) or custom format (JSON, etc...)
    4. Create a CLI

    View full-size slide

  97. Case study: gofmt
    package main
    import "fmt"
    func main(){
    var sum int = 3+2
    fmt.Println( sum )
    }

    View full-size slide

  98. Case study: gofmt
    go/parser
    package main
    import "fmt"
    func main(){
    var sum int = 3+2
    fmt.Println( sum )
    }

    View full-size slide

  99. Case study: gofmt
    go/parser
    go/ast
    package main
    import "fmt"
    func main(){
    var sum int = 3+2
    fmt.Println( sum )
    }

    View full-size slide

  100. func processFile(filename string, in io.Reader, out io.Writer, stdin bool)
    error {
    // ...
    src, err := ioutil.ReadAll(in)
    if err != nil {
    return err
    }
    file, sourceAdj, indentAdj, err := parse(fileSet, filename, src, stdin)
    if err != nil {
    return err
    }
    // ...

    View full-size slide

  101. func processFile(filename string, in io.Reader, out io.Writer, stdin bool)
    error {
    // ...
    src, err := ioutil.ReadAll(in)
    if err != nil {
    return err
    }
    file, sourceAdj, indentAdj, err := parse(fileSet, filename, src, stdin)
    if err != nil {
    return err
    }
    // ...

    View full-size slide

  102. Case study: gofmt
    1. Read (go/parser)
    2. Inspect (go/ast)
    3. Write (go/printer) or custom format (JSON, etc...)
    4. Create a CLI

    View full-size slide

  103. Case study: gofmt
    go/parser
    go/ast
    sort
    simplify
    rewrite
    package main
    import "fmt"
    func main(){
    var sum int = 3+2
    fmt.Println( sum )
    }

    View full-size slide

  104. func processFile(filename string, in io.Reader, out io.Writer, stdin bool)
    error {
    // ...
    if rewrite != nil {
    if sourceAdj == nil {
    file = rewrite(file)
    } else {
    fmt.Fprintf(os.Stderr, "warning: rewrite ignored for incomplete
    programs\n")
    }
    }
    ast.SortImports(fileSet, file)
    if *simplifyAST {
    simplify(file)
    }
    // ...

    View full-size slide

  105. Case study: gofmt
    1. Read (go/parser)
    2. Inspect (go/ast)
    3. Write (go/printer) or custom format (JSON, etc...)
    4. Create a CLI

    View full-size slide

  106. Case study: gofmt
    go/parser
    go/ast
    sort
    simplify
    rewrite
    go/printer
    package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }
    package main
    import "fmt"
    func main(){
    var sum int = 3+2
    fmt.Println( sum )
    }

    View full-size slide

  107. func processFile(filename string, in io.Reader, out io.Writer, stdin bool)
    error {
    // ...
    res, err := format(fileSet, file, sourceAdj, indentAdj, src,
    printer.Config{Mode: printerMode, Tabwidth: tabWidth})
    if err != nil {
    return err
    }
    if !*list && !*write && !*doDiff {
    _, err = out.Write(res)
    }
    // ...

    View full-size slide

  108. Case study: gofmt
    1. Read (go/parser)
    2. Inspect (go/ast)
    3. Write (go/printer) or custom format (JSON, etc...)
    4. Create a CLI

    View full-size slide

  109. func main() {
    // ...
    for i := 0; i < flag.NArg(); i++ {
    path := flag.Arg(i)
    switch dir, err := os.Stat(path); {
    case err != nil:
    report(err)
    case dir.IsDir():
    walkDir(path)
    default:
    if err := processFile(path, nil, os.Stdout, false); err != nil {
    report(err)
    }
    }
    }
    // ...
    }

    View full-size slide

  110. $ cat main.go
    package main
    import "fmt"
    func main() {
    var sum int = 3+2
    fmt.Println( sum )
    }

    View full-size slide

  111. $ cat main.go
    package main
    import "fmt"
    func main() {
    var sum int = 3+2
    fmt.Println( sum )
    }
    $ gofmt main.go
    package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }

    View full-size slide

  112. package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }
    go/parser go/ast
    go/printer
    package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }

    View full-size slide

  113. package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }
    go/ast
    package main
    import "fmt"
    func main() {
    var sum int = 3 + 2
    fmt.Println(sum)
    }
    go/printer
    go/parser
    go/scanner
    go/token
    go/parser

    View full-size slide

  114. Source: https://talks.golang.org/2014/static-analysis.slide#8

    View full-size slide

  115. Thanks
    Fatih Arslan
    fatih
    fatih
    [email protected]

    View full-size slide