Reading Go Tools - GoCon 2016 Spring

Transcript

1. 1.

   Reading Go Tools GoCon 2016 Spring
 motemen

2. 2.

   About myself • @motemen • Chief engineer at Hatena co,

   ltd. • Writing Go at 90% of hobby • Likes to write programs in/for Go
3. 3.

   Go tools I have written • gore — Go REPL

   (-like thingy) • gompatible — diff package API changes • goiferr — automatically insert “if err != nil” statements • gofind — find struct field/method usages
4. 4.

   Reading Go tools • gofmt, godoc, … • To write

   your own Go tools • To know about standard go/* packages • Meta programming powers Go • eg. “go generate”
5. 5.

   Tools for reading code • jstemmer/gotags — generates ctags-compatible tags

   file • x/tools/cmd/guru — query Go program in various ways • motemen/gofind — search for a specific usage of types % gofind -s golang.org/x/tools/cmd/stringer.Package.defs \ golang.org/x/tools/cmd/stringer stringer.go:262:6: pkg.defs = make(map[*ast.Ident]types.Object) stringer.go:265:13: Defs: pkg.defs, stringer.go:433:21: obj, ok := f.pkg.defs[name]
6. 6.

   Tools to read today • x/tools/cmd/goimports • golang/gddo

7. 7.

   goimports

8. 8.

   goimports • Updates import decls; adds missing, removes unused •

   Can replace gofmt • go get golang.org/x/tools/cmd/goimports
9. 9.

   goimports: usage • goimports -w file.go package p import (

   "fmt" "log" ) func f() { fmt.Println(math.E) } package p import ( "fmt" "math" ) func f() { fmt.Println(math.E) }
10. 10.

    goimports: steps Main logic: golang.org/x/tools/imports.Process() • Parse source files •

    Resolve imports • Find unused/missing imports • Generate modified source code
11. 11.

    goimports: parsing codes • go/parser.ParseFile() to obtain *go/ast.File • ast.Node:

    An “abstract syntax tree” for a Go source file • Every program handling Go source should call this
12. 12.

    goimports: resolving imports • Traverse tree by ast.Walk() • Packages

    are imported by “import” declarations • ast.ImportSpec • Packages are referred in “pkg.name” form (“selectors”) • ast.SelectorExpr
13. 13.

    goimports: what name is provided? • import “fmt” introduces “fmt”

    • import “gopkg.in/yaml.v2” introduces “yaml” • You cannot determine until you parse the source code • importPathToName()
14. 14.

    goimports: importPathToName() • Use go/build.Import to get information from package

    path • Determines source files as “go build path/to/pkg” do • Supports GOOS, GOARCH, -tag=… • go/build.Package struct • Dir / Name / GoFiles / TestGoFiles / …
15. 15.

    goimports: find package selector var math struct { E string

    }; math.E • Does not require package math • Go is lexically scoped • Each ast.Ident has “Obj ast.Object”, named language entity • Find identifiers with Obj == nil
16. 16.

    goimports: find package missing • Check package names and their

    exports • First, match against prebuilt stdlib table • Second, search user-installed libs, i.e. under GOPATH • loadPkgIndex() • Traverse go/build.Default.SrcDirs
17. 17.

    goimports: generating source • Modify AST to insert imports &

    Insert blank lines to group imports • go/printer API to get []byte from ast.File • go/format.Source to re-format []byte • Same code is used inside gofmt • “This file and the file src/cmd/gofmt/internal.go are the same (but for this comment and the package name).”
18. 18.

    gddo

19. 19.

    gddo • gddo = “Go Doc Dot Org” • https://godoc.org/

    • go get github.com/golang/gddo/gddo-server
20. 20.

    gddo • https://godoc.org/fmt • https://godoc.org/github.com/motemen/go-astmanip

21. 21.

    gddo: steps Main logic: gddo-server.servePackage() • Retrieve cache or fetch

    source codes on remote VCS • Parse them and obtain documentations • (Render them to HTML)
22. 22.

    gddo: retrieve document • On GET /path/to/pkg, retrieve doc from

    db • gddo/doc.Package • If stale, fetch source codes from remote • gddo/doc.Get(client, importPath, etag) (*doc.Package, error)
23. 23.

    gddo: gddo/doc.Get() • gddo/gosrc.Get() to obtain virtual source directory •

    gosrc.File { Data []byte … } • gosrc.Directory { Files []*File … } • gddo/doc.newPackage() converts the dir to document
24. 24.

    gddo: gddo/gosrc.Get() • Well-known services have their own handler •

    GitHub, BitBucket, … • e.g. getGitHubDir() uses GitHub API • GET https://api.github.com/repos/{owner}/{repo}/contents/{dir} • Less-known ones can provide go-source meta tags
25. 25.

    gddo: gddo/doc.newPacakge() • Builds documents from gosrc.Directory • go/doc API

    to generate documents from AST • go/build.Context API to list files to list doc sources • Avoid mixing *_linux.go and *_windows.go together
26. 26.

    go/build.Context • Can handle not only local filesystems • Supports

    custom fs via fields like OpenFile, ReadDir, etc. type Context struct { // OpenFile opens a file (not a directory) for reading. // If OpenFile is nil, Import uses os.Open. OpenFile func(path string) (r io.ReadCloser, err error) … }
27. 27.

    Standard libs we saw today • go/ast, go/parser — syntactic

    parsing source code • go/build — deciding package location • go/doc — extracting documentations • go/format, go/printer — source code formatting
28. 28.

    Conclusion • Powerful standard Go tools are written using std

    libs • We also can use go/* to write our Go tools • Other tools to read:gofmt, godoc, x/tools/cmd/*
29. 29.

    None
30. 30.

    None
31. 31.

    Appendix

32. 32.

    stringer

33. 33.

    stringer • Generates String() methods of constants for humans •

    go get golang.org/x/tools/cmd/stringer
34. 34.

    stringer: usage • stringer -type Sushi sushi.go package sushi type

    Sushi uint const ( Maguro Sushi = iota Ikura Uni Tamago ) // Code generated by "stringer -type Sushi sush package sushi import "fmt" const _Sushi_name = "MaguroIkuraUniTamago" var _Sushi_index = [...]uint8{0, 6, 11, 14, 20} func (i Sushi) String() string { if i >= Sushi(len(_Sushi_index)-1) { return fmt.Sprintf("Sushi(%d)", i) } return _Sushi_name[_Sushi_index[i]:_Sushi_i }
35. 35.

    stringer: steps • Parse given source files • Find constants

    with specified types • Generate source codes to stringify them
36. 36.

    stringer: type checking • Uses go/types API • go/types.Config.Check •

    Obtain “Defs”, map[*ast.Ident]types.Object • Maps each identifier to its type information
37. 37.

    stringer: find constants • Traverse AST for ast.GenDecl • Collect

    the constants’ names along with their (int) values
38. 38.

    stringer: generate source • Generate source code using fmt.Sprintf •

    Then format by go/format.Source // Arguments to format are: // [1]: type name // [2]: size of index element (8 for uint8 etc.) // [3]: less than zero check (for signed types) const stringOneRun = `func (i %[1]s) String() string { if %[3]si >= %[1]s(len(_%[1]s_index)-1) { return fmt.Sprintf("%[1]s(%%d)", i) } return _%[1]s_name[_%[1]s_index[i]:_%[1]s_index[i+1]] } `
39. 39.

    CLI Interface

40. 40.

    Common interfaces • eg. gofmt [flags] [path…] • Accept files

    • Accept directories (process their children) • Otherwise, use stdin
41. 41.

    Common interfaces • Output to stdout by default • Overwrite

    input files if -w given • Output a diff if -d given (simply executing “diff” command) • List files which are to be updated if -l given