Reading Go Tools - GoCon 2016 Spring

Transcript

1. Reading Go Tools
   GoCon 2016 Spring

   motemen
   

   View Slide

2. About myself
   • @motemen
   • Chief engineer at Hatena co, ltd.
   • Writing Go at 90% of hobby
   • Likes to write programs in/for Go
   

   View Slide

3. Go tools I have written
   • gore — Go REPL (-like thingy)
   • gompatible — diff package API changes
   • goiferr — automatically insert “if err != nil” statements
   • gofind — find struct field/method usages
   

   View Slide

4. Reading Go tools
   • gofmt, godoc, …
   • To write your own Go tools
   • To know about standard go/* packages
   • Meta programming powers Go
   • eg. “go generate”
   

   View Slide

5. Tools for reading code
   • jstemmer/gotags — generates ctags-compatible tags file
   • x/tools/cmd/guru — query Go program in various ways
   • motemen/gofind — search for a specific usage of types
   % gofind -s golang.org/x/tools/cmd/stringer.Package.defs \
   golang.org/x/tools/cmd/stringer
   stringer.go:262:6: pkg.defs = make(map[*ast.Ident]types.Object)
   stringer.go:265:13: Defs: pkg.defs,
   stringer.go:433:21: obj, ok := f.pkg.defs[name]
   

   View Slide

6. Tools to read today
   • x/tools/cmd/goimports
   • golang/gddo
   

   View Slide

7. goimports
   

   View Slide

8. goimports
   • Updates import decls; adds missing, removes unused
   • Can replace gofmt
   • go get golang.org/x/tools/cmd/goimports
   

   View Slide

9. goimports: usage
   • goimports -w file.go
   package p
   import (
   "fmt"
   "log"
   )
   func f() {
   fmt.Println(math.E)
   }
   package p
   import (
   "fmt"
   "math"
   )
   func f() {
   fmt.Println(math.E)
   }
   
   

   View Slide

10. goimports: steps
    Main logic: golang.org/x/tools/imports.Process()
    • Parse source files
    • Resolve imports
    • Find unused/missing imports
    • Generate modified source code
    

    View Slide

11. goimports: parsing codes
    • go/parser.ParseFile() to obtain *go/ast.File
    • ast.Node: An “abstract syntax tree” for a Go source file
    • Every program handling Go source should call this
    

    View Slide

12. goimports: resolving imports
    • Traverse tree by ast.Walk()
    • Packages are imported by “import” declarations
    • ast.ImportSpec
    • Packages are referred in “pkg.name” form (“selectors”)
    • ast.SelectorExpr
    

    View Slide

13. goimports: what name is provided?
    • import “fmt” introduces “fmt”
    • import “gopkg.in/yaml.v2” introduces “yaml”
    • You cannot determine until you parse the source code
    • importPathToName()
    

    View Slide

14. goimports: importPathToName()
    • Use go/build.Import to get information from package path
    • Determines source files as “go build path/to/pkg” do
    • Supports GOOS, GOARCH, -tag=…
    • go/build.Package struct
    • Dir / Name / GoFiles / TestGoFiles / …
    

    View Slide

15. goimports: find package selector
    var math struct { E string }; math.E
    • Does not require package math
    • Go is lexically scoped
    • Each ast.Ident has “Obj ast.Object”, named language entity
    • Find identifiers with Obj == nil
    

    View Slide

16. goimports: find package missing
    • Check package names and their exports
    • First, match against prebuilt stdlib table
    • Second, search user-installed libs, i.e. under GOPATH
    • loadPkgIndex()
    • Traverse go/build.Default.SrcDirs
    

    View Slide

17. goimports: generating source
    • Modify AST to insert imports & Insert blank lines to group imports
    • go/printer API to get []byte from ast.File
    • go/format.Source to re-format []byte
    • Same code is used inside gofmt
    • “This file and the file src/cmd/gofmt/internal.go are the same
    (but for this comment and the package name).”
    

    View Slide

18. gddo
    

    View Slide

19. gddo
    • gddo = “Go Doc Dot Org”
    • https://godoc.org/
    • go get github.com/golang/gddo/gddo-server
    

    View Slide

20. gddo
    • https://godoc.org/fmt
    • https://godoc.org/github.com/motemen/go-astmanip
    

    View Slide

21. gddo: steps
    Main logic: gddo-server.servePackage()
    • Retrieve cache or fetch source codes on remote VCS
    • Parse them and obtain documentations
    • (Render them to HTML)
    

    View Slide

22. gddo: retrieve document
    • On GET /path/to/pkg, retrieve doc from db
    • gddo/doc.Package
    • If stale, fetch source codes from remote
    • gddo/doc.Get(client, importPath, etag)
    (*doc.Package, error)
    

    View Slide

23. gddo: gddo/doc.Get()
    • gddo/gosrc.Get() to obtain virtual source directory
    • gosrc.File { Data []byte … }
    • gosrc.Directory { Files []*File … }
    • gddo/doc.newPackage() converts the dir to document
    

    View Slide

24. gddo: gddo/gosrc.Get()
    • Well-known services have their own handler
    • GitHub, BitBucket, …
    • e.g. getGitHubDir() uses GitHub API
    • GET https://api.github.com/repos/{owner}/{repo}/contents/{dir}
    • Less-known ones can provide go-source meta tags
    

    View Slide

25. gddo: gddo/doc.newPacakge()
    • Builds documents from gosrc.Directory
    • go/doc API to generate documents from AST
    • go/build.Context API to list files to list doc sources
    • Avoid mixing *_linux.go and *_windows.go together
    

    View Slide

26. go/build.Context
    • Can handle not only local filesystems
    • Supports custom fs via fields like OpenFile, ReadDir, etc.
    type Context struct {
    // OpenFile opens a file (not a directory) for reading.
    // If OpenFile is nil, Import uses os.Open.
    OpenFile func(path string) (r io.ReadCloser, err error)
    …
    }
    

    View Slide

27. Standard libs we saw today
    • go/ast, go/parser — syntactic parsing source code
    • go/build — deciding package location
    • go/doc — extracting documentations
    • go/format, go/printer — source code formatting
    

    View Slide

28. Conclusion
    • Powerful standard Go tools are written using std libs
    • We also can use go/* to write our Go tools
    • Other tools to read:gofmt, godoc, x/tools/cmd/*
    

    View Slide

29. View Slide

30. View Slide

31. Appendix
    

    View Slide

32. stringer
    

    View Slide

33. stringer
    • Generates String() methods of constants for humans
    • go get golang.org/x/tools/cmd/stringer
    

    View Slide

34. stringer: usage
    • stringer -type Sushi sushi.go
    package sushi
    type Sushi uint
    const (
    Maguro Sushi = iota
    Ikura
    Uni
    Tamago
    )
    // Code generated by "stringer -type Sushi sush
    package sushi
    import "fmt"
    const _Sushi_name = "MaguroIkuraUniTamago"
    var _Sushi_index = [...]uint8{0, 6, 11, 14, 20}
    func (i Sushi) String() string {
    if i >= Sushi(len(_Sushi_index)-1) {
    return fmt.Sprintf("Sushi(%d)", i)
    }
    return _Sushi_name[_Sushi_index[i]:_Sushi_i
    }
    
    

    View Slide

35. stringer: steps
    • Parse given source files
    • Find constants with specified types
    • Generate source codes to stringify them
    

    View Slide

36. stringer: type checking
    • Uses go/types API
    • go/types.Config.Check
    • Obtain “Defs”, map[*ast.Ident]types.Object
    • Maps each identifier to its type information
    

    View Slide

37. stringer: find constants
    • Traverse AST for ast.GenDecl
    • Collect the constants’ names along with their (int) values
    

    View Slide

38. stringer: generate source
    • Generate source code using fmt.Sprintf
    • Then format by go/format.Source
    // Arguments to format are:
    // [1]: type name
    // [2]: size of index element (8 for uint8 etc.)
    // [3]: less than zero check (for signed types)
    const stringOneRun = `func (i %[1]s) String() string {
    if %[3]si >= %[1]s(len(_%[1]s_index)-1) {
    return fmt.Sprintf("%[1]s(%%d)", i)
    }
    return _%[1]s_name[_%[1]s_index[i]:_%[1]s_index[i+1]]
    }
    `
    

    View Slide

39. CLI Interface
    

    View Slide

40. Common interfaces
    • eg. gofmt [flags] [path…]
    • Accept files
    • Accept directories (process their children)
    • Otherwise, use stdin
    

    View Slide

41. Common interfaces
    • Output to stdout by default
    • Overwrite input files if -w given
    • Output a diff if -d given (simply executing “diff” command)
    • List files which are to be updated if -l given
    

    View Slide