Reading Go Tools - GoCon 2016 Spring

Transcript

1. Reading Go Tools GoCon 2016 Spring
 motemen

2. About myself • @motemen • Chief engineer at Hatena co,

   ltd. • Writing Go at 90% of hobby • Likes to write programs in/for Go

3. Go tools I have written • gore — Go REPL

   (-like thingy) • gompatible — diff package API changes • goiferr — automatically insert “if err != nil” statements • gofind — find struct field/method usages

4. Reading Go tools • gofmt, godoc, … • To write

   your own Go tools • To know about standard go/* packages • Meta programming powers Go • eg. “go generate”

5. Tools for reading code • jstemmer/gotags — generates ctags-compatible tags

   file • x/tools/cmd/guru — query Go program in various ways • motemen/gofind — search for a specific usage of types % gofind -s golang.org/x/tools/cmd/stringer.Package.defs \ golang.org/x/tools/cmd/stringer stringer.go:262:6: pkg.defs = make(map[*ast.Ident]types.Object) stringer.go:265:13: Defs: pkg.defs, stringer.go:433:21: obj, ok := f.pkg.defs[name]

6. Tools to read today • x/tools/cmd/goimports • golang/gddo

7. goimports

8. goimports • Updates import decls; adds missing, removes unused •

   Can replace gofmt • go get golang.org/x/tools/cmd/goimports

9. goimports: usage • goimports -w file.go package p import (

   "fmt" "log" ) func f() { fmt.Println(math.E) } package p import ( "fmt" "math" ) func f() { fmt.Println(math.E) }

10. goimports: steps Main logic: golang.org/x/tools/imports.Process() • Parse source files •

    Resolve imports • Find unused/missing imports • Generate modified source code

11. goimports: parsing codes • go/parser.ParseFile() to obtain *go/ast.File • ast.Node:

    An “abstract syntax tree” for a Go source file • Every program handling Go source should call this

12. goimports: resolving imports • Traverse tree by ast.Walk() • Packages

    are imported by “import” declarations • ast.ImportSpec • Packages are referred in “pkg.name” form (“selectors”) • ast.SelectorExpr

13. goimports: what name is provided? • import “fmt” introduces “fmt”

    • import “gopkg.in/yaml.v2” introduces “yaml” • You cannot determine until you parse the source code • importPathToName()

14. goimports: importPathToName() • Use go/build.Import to get information from package

    path • Determines source files as “go build path/to/pkg” do • Supports GOOS, GOARCH, -tag=… • go/build.Package struct • Dir / Name / GoFiles / TestGoFiles / …

15. goimports: find package selector var math struct { E string

    }; math.E • Does not require package math • Go is lexically scoped • Each ast.Ident has “Obj ast.Object”, named language entity • Find identifiers with Obj == nil

16. goimports: find package missing • Check package names and their

    exports • First, match against prebuilt stdlib table • Second, search user-installed libs, i.e. under GOPATH • loadPkgIndex() • Traverse go/build.Default.SrcDirs

17. goimports: generating source • Modify AST to insert imports &

    Insert blank lines to group imports • go/printer API to get []byte from ast.File • go/format.Source to re-format []byte • Same code is used inside gofmt • “This file and the file src/cmd/gofmt/internal.go are the same (but for this comment and the package name).”

18. gddo

19. gddo • gddo = “Go Doc Dot Org” • https://godoc.org/

    • go get github.com/golang/gddo/gddo-server

20. gddo • https://godoc.org/fmt • https://godoc.org/github.com/motemen/go-astmanip

21. gddo: steps Main logic: gddo-server.servePackage() • Retrieve cache or fetch

    source codes on remote VCS • Parse them and obtain documentations • (Render them to HTML)

22. gddo: retrieve document • On GET /path/to/pkg, retrieve doc from

    db • gddo/doc.Package • If stale, fetch source codes from remote • gddo/doc.Get(client, importPath, etag) (*doc.Package, error)

23. gddo: gddo/doc.Get() • gddo/gosrc.Get() to obtain virtual source directory •

    gosrc.File { Data []byte … } • gosrc.Directory { Files []*File … } • gddo/doc.newPackage() converts the dir to document

24. gddo: gddo/gosrc.Get() • Well-known services have their own handler •

    GitHub, BitBucket, … • e.g. getGitHubDir() uses GitHub API • GET https://api.github.com/repos/{owner}/{repo}/contents/{dir} • Less-known ones can provide go-source meta tags

25. gddo: gddo/doc.newPacakge() • Builds documents from gosrc.Directory • go/doc API

    to generate documents from AST • go/build.Context API to list files to list doc sources • Avoid mixing *_linux.go and *_windows.go together

26. go/build.Context • Can handle not only local filesystems • Supports

    custom fs via fields like OpenFile, ReadDir, etc. type Context struct { // OpenFile opens a file (not a directory) for reading. // If OpenFile is nil, Import uses os.Open. OpenFile func(path string) (r io.ReadCloser, err error) … }

27. Standard libs we saw today • go/ast, go/parser — syntactic

    parsing source code • go/build — deciding package location • go/doc — extracting documentations • go/format, go/printer — source code formatting

28. Conclusion • Powerful standard Go tools are written using std

    libs • We also can use go/* to write our Go tools • Other tools to read:gofmt, godoc, x/tools/cmd/*
29. None
30. None

31. Appendix

32. stringer

33. stringer • Generates String() methods of constants for humans •

    go get golang.org/x/tools/cmd/stringer

34. stringer: usage • stringer -type Sushi sushi.go package sushi type

    Sushi uint const ( Maguro Sushi = iota Ikura Uni Tamago ) // Code generated by "stringer -type Sushi sush package sushi import "fmt" const _Sushi_name = "MaguroIkuraUniTamago" var _Sushi_index = [...]uint8{0, 6, 11, 14, 20} func (i Sushi) String() string { if i >= Sushi(len(_Sushi_index)-1) { return fmt.Sprintf("Sushi(%d)", i) } return _Sushi_name[_Sushi_index[i]:_Sushi_i }

35. stringer: steps • Parse given source files • Find constants

    with specified types • Generate source codes to stringify them

36. stringer: type checking • Uses go/types API • go/types.Config.Check •

    Obtain “Defs”, map[*ast.Ident]types.Object • Maps each identifier to its type information

37. stringer: find constants • Traverse AST for ast.GenDecl • Collect

    the constants’ names along with their (int) values

38. stringer: generate source • Generate source code using fmt.Sprintf •

    Then format by go/format.Source // Arguments to format are: // [1]: type name // [2]: size of index element (8 for uint8 etc.) // [3]: less than zero check (for signed types) const stringOneRun = `func (i %[1]s) String() string { if %[3]si >= %[1]s(len(_%[1]s_index)-1) { return fmt.Sprintf("%[1]s(%%d)", i) } return _%[1]s_name[_%[1]s_index[i]:_%[1]s_index[i+1]] } `

39. CLI Interface

40. Common interfaces • eg. gofmt [flags] [path…] • Accept files

    • Accept directories (process their children) • Otherwise, use stdin

41. Common interfaces • Output to stdout by default • Overwrite

    input files if -w given • Output a diff if -d given (simply executing “diff” command) • List files which are to be updated if -l given