Reading Go Tools - GoCon 2016 Spring

9b741203feda475cbeae8b384de9f415?s=47 motemen
April 23, 2016

Reading Go Tools - GoCon 2016 Spring

9b741203feda475cbeae8b384de9f415?s=128

motemen

April 23, 2016
Tweet

Transcript

  1. 2.

    About myself • @motemen • Chief engineer at Hatena co,

    ltd. • Writing Go at 90% of hobby • Likes to write programs in/for Go
  2. 3.

    Go tools I have written • gore — Go REPL

    (-like thingy) • gompatible — diff package API changes • goiferr — automatically insert “if err != nil” statements • gofind — find struct field/method usages
  3. 4.

    Reading Go tools • gofmt, godoc, … • To write

    your own Go tools • To know about standard go/* packages • Meta programming powers Go • eg. “go generate”
  4. 5.

    Tools for reading code • jstemmer/gotags — generates ctags-compatible tags

    file • x/tools/cmd/guru — query Go program in various ways • motemen/gofind — search for a specific usage of types % gofind -s golang.org/x/tools/cmd/stringer.Package.defs \ golang.org/x/tools/cmd/stringer stringer.go:262:6: pkg.defs = make(map[*ast.Ident]types.Object) stringer.go:265:13: Defs: pkg.defs, stringer.go:433:21: obj, ok := f.pkg.defs[name]
  5. 8.

    goimports • Updates import decls; adds missing, removes unused •

    Can replace gofmt • go get golang.org/x/tools/cmd/goimports
  6. 9.

    goimports: usage • goimports -w file.go package p import (

    "fmt" "log" ) func f() { fmt.Println(math.E) } package p import ( "fmt" "math" ) func f() { fmt.Println(math.E) }
  7. 10.

    goimports: steps Main logic: golang.org/x/tools/imports.Process() • Parse source files •

    Resolve imports • Find unused/missing imports • Generate modified source code
  8. 11.

    goimports: parsing codes • go/parser.ParseFile() to obtain *go/ast.File • ast.Node:

    An “abstract syntax tree” for a Go source file • Every program handling Go source should call this
  9. 12.

    goimports: resolving imports • Traverse tree by ast.Walk() • Packages

    are imported by “import” declarations • ast.ImportSpec • Packages are referred in “pkg.name” form (“selectors”) • ast.SelectorExpr
  10. 13.

    goimports: what name is provided? • import “fmt” introduces “fmt”

    • import “gopkg.in/yaml.v2” introduces “yaml” • You cannot determine until you parse the source code • importPathToName()
  11. 14.

    goimports: importPathToName() • Use go/build.Import to get information from package

    path • Determines source files as “go build path/to/pkg” do • Supports GOOS, GOARCH, -tag=… • go/build.Package struct • Dir / Name / GoFiles / TestGoFiles / …
  12. 15.

    goimports: find package selector var math struct { E string

    }; math.E • Does not require package math • Go is lexically scoped • Each ast.Ident has “Obj ast.Object”, named language entity • Find identifiers with Obj == nil
  13. 16.

    goimports: find package missing • Check package names and their

    exports • First, match against prebuilt stdlib table • Second, search user-installed libs, i.e. under GOPATH • loadPkgIndex() • Traverse go/build.Default.SrcDirs
  14. 17.

    goimports: generating source • Modify AST to insert imports &

    Insert blank lines to group imports • go/printer API to get []byte from ast.File • go/format.Source to re-format []byte • Same code is used inside gofmt • “This file and the file src/cmd/gofmt/internal.go are the same (but for this comment and the package name).”
  15. 18.
  16. 19.

    gddo • gddo = “Go Doc Dot Org” • https://godoc.org/

    • go get github.com/golang/gddo/gddo-server
  17. 21.

    gddo: steps Main logic: gddo-server.servePackage() • Retrieve cache or fetch

    source codes on remote VCS • Parse them and obtain documentations • (Render them to HTML)
  18. 22.

    gddo: retrieve document • On GET /path/to/pkg, retrieve doc from

    db • gddo/doc.Package • If stale, fetch source codes from remote • gddo/doc.Get(client, importPath, etag) (*doc.Package, error)
  19. 23.

    gddo: gddo/doc.Get() • gddo/gosrc.Get() to obtain virtual source directory •

    gosrc.File { Data []byte … } • gosrc.Directory { Files []*File … } • gddo/doc.newPackage() converts the dir to document
  20. 24.

    gddo: gddo/gosrc.Get() • Well-known services have their own handler •

    GitHub, BitBucket, … • e.g. getGitHubDir() uses GitHub API • GET https://api.github.com/repos/{owner}/{repo}/contents/{dir} • Less-known ones can provide go-source meta tags
  21. 25.

    gddo: gddo/doc.newPacakge() • Builds documents from gosrc.Directory • go/doc API

    to generate documents from AST • go/build.Context API to list files to list doc sources • Avoid mixing *_linux.go and *_windows.go together
  22. 26.

    go/build.Context • Can handle not only local filesystems • Supports

    custom fs via fields like OpenFile, ReadDir, etc. type Context struct { // OpenFile opens a file (not a directory) for reading. // If OpenFile is nil, Import uses os.Open. OpenFile func(path string) (r io.ReadCloser, err error) … }
  23. 27.

    Standard libs we saw today • go/ast, go/parser — syntactic

    parsing source code • go/build — deciding package location • go/doc — extracting documentations • go/format, go/printer — source code formatting
  24. 28.

    Conclusion • Powerful standard Go tools are written using std

    libs • We also can use go/* to write our Go tools • Other tools to read:gofmt, godoc, x/tools/cmd/*
  25. 29.
  26. 30.
  27. 31.
  28. 32.
  29. 33.
  30. 34.

    stringer: usage • stringer -type Sushi sushi.go package sushi type

    Sushi uint const ( Maguro Sushi = iota Ikura Uni Tamago ) // Code generated by "stringer -type Sushi sush package sushi import "fmt" const _Sushi_name = "MaguroIkuraUniTamago" var _Sushi_index = [...]uint8{0, 6, 11, 14, 20} func (i Sushi) String() string { if i >= Sushi(len(_Sushi_index)-1) { return fmt.Sprintf("Sushi(%d)", i) } return _Sushi_name[_Sushi_index[i]:_Sushi_i }
  31. 35.

    stringer: steps • Parse given source files • Find constants

    with specified types • Generate source codes to stringify them
  32. 36.

    stringer: type checking • Uses go/types API • go/types.Config.Check •

    Obtain “Defs”, map[*ast.Ident]types.Object • Maps each identifier to its type information
  33. 37.

    stringer: find constants • Traverse AST for ast.GenDecl • Collect

    the constants’ names along with their (int) values
  34. 38.

    stringer: generate source • Generate source code using fmt.Sprintf •

    Then format by go/format.Source // Arguments to format are: // [1]: type name // [2]: size of index element (8 for uint8 etc.) // [3]: less than zero check (for signed types) const stringOneRun = `func (i %[1]s) String() string { if %[3]si >= %[1]s(len(_%[1]s_index)-1) { return fmt.Sprintf("%[1]s(%%d)", i) } return _%[1]s_name[_%[1]s_index[i]:_%[1]s_index[i+1]] } `
  35. 40.

    Common interfaces • eg. gofmt [flags] [path…] • Accept files

    • Accept directories (process their children) • Otherwise, use stdin
  36. 41.

    Common interfaces • Output to stdout by default • Overwrite

    input files if -w given • Output a diff if -d given (simply executing “diff” command) • List files which are to be updated if -l given