Reading Go Tools GoCon 2016 Spring
 motemen

About myself • @motemen • Chief engineer at Hatena co, ltd. • Writing Go at 90% of hobby • Likes to write programs in/for Go

Go tools I have written • gore — Go REPL (-like thingy) • gompatible — diff package API changes • goiferr — automatically insert “if err != nil” statements • gofind — find struct field/method usages

Reading Go tools • gofmt, godoc, … • To write your own Go tools • To know about standard go/* packages • Meta programming powers Go • eg. “go generate”

Tools for reading code • jstemmer/gotags — generates ctags-compatible tags file • x/tools/cmd/guru — query Go program in various ways • motemen/gofind — search for a specific usage of types % gofind -s golang.org/x/tools/cmd/stringer.Package.defs \ golang.org/x/tools/cmd/stringer stringer.go:262:6: pkg.defs = make(map[*ast.Ident]types.Object) stringer.go:265:13: Defs: pkg.defs, stringer.go:433:21: obj, ok := f.pkg.defs[name]

Tools to read today • x/tools/cmd/goimports • golang/gddo

goimports

goimports • Updates import decls; adds missing, removes unused • Can replace gofmt • go get golang.org/x/tools/cmd/goimports

goimports: usage • goimports -w file.go package p import ( "fmt" "log" ) func f() { fmt.Println(math.E) } package p import ( "fmt" "math" ) func f() { fmt.Println(math.E) }

goimports: steps Main logic: golang.org/x/tools/imports.Process() • Parse source files • Resolve imports • Find unused/missing imports • Generate modified source code

goimports: parsing codes • go/parser.ParseFile() to obtain *go/ast.File • ast.Node: An “abstract syntax tree” for a Go source file • Every program handling Go source should call this

goimports: resolving imports • Traverse tree by ast.Walk() • Packages are imported by “import” declarations • ast.ImportSpec • Packages are referred in “pkg.name” form (“selectors”) • ast.SelectorExpr

goimports: what name is provided? • import “fmt” introduces “fmt” • import “gopkg.in/yaml.v2” introduces “yaml” • You cannot determine until you parse the source code • importPathToName()

goimports: importPathToName() • Use go/build.Import to get information from package path • Determines source files as “go build path/to/pkg” do • Supports GOOS, GOARCH, -tag=… • go/build.Package struct • Dir / Name / GoFiles / TestGoFiles / …

goimports: find package selector var math struct { E string }; math.E • Does not require package math • Go is lexically scoped • Each ast.Ident has “Obj ast.Object”, named language entity • Find identifiers with Obj == nil

goimports: find package missing • Check package names and their exports • First, match against prebuilt stdlib table • Second, search user-installed libs, i.e. under GOPATH • loadPkgIndex() • Traverse go/build.Default.SrcDirs

goimports: generating source • Modify AST to insert imports & Insert blank lines to group imports • go/printer API to get []byte from ast.File • go/format.Source to re-format []byte • Same code is used inside gofmt • “This file and the file src/cmd/gofmt/internal.go are the same (but for this comment and the package name).”

gddo

gddo • gddo = “Go Doc Dot Org” • https://godoc.org/ • go get github.com/golang/gddo/gddo-server

gddo • https://godoc.org/fmt • https://godoc.org/github.com/motemen/go-astmanip

gddo: steps Main logic: gddo-server.servePackage() • Retrieve cache or fetch source codes on remote VCS • Parse them and obtain documentations • (Render them to HTML)

gddo: retrieve document • On GET /path/to/pkg, retrieve doc from db • gddo/doc.Package • If stale, fetch source codes from remote • gddo/doc.Get(client, importPath, etag) (*doc.Package, error)

gddo: gddo/doc.Get() • gddo/gosrc.Get() to obtain virtual source directory • gosrc.File { Data []byte … } • gosrc.Directory { Files []*File … } • gddo/doc.newPackage() converts the dir to document

gddo: gddo/gosrc.Get() • Well-known services have their own handler • GitHub, BitBucket, … • e.g. getGitHubDir() uses GitHub API • GET https://api.github.com/repos/{owner}/{repo}/contents/{dir} • Less-known ones can provide go-source meta tags

gddo: gddo/doc.newPacakge() • Builds documents from gosrc.Directory • go/doc API to generate documents from AST • go/build.Context API to list files to list doc sources • Avoid mixing *_linux.go and *_windows.go together

go/build.Context • Can handle not only local filesystems • Supports custom fs via fields like OpenFile, ReadDir, etc. type Context struct { // OpenFile opens a file (not a directory) for reading. // If OpenFile is nil, Import uses os.Open. OpenFile func(path string) (r io.ReadCloser, err error) … }

Standard libs we saw today • go/ast, go/parser — syntactic parsing source code • go/build — deciding package location • go/doc — extracting documentations • go/format, go/printer — source code formatting

Conclusion • Powerful standard Go tools are written using std libs • We also can use go/* to write our Go tools • Other tools to read:gofmt, godoc, x/tools/cmd/*

No content

No content

Appendix

stringer

stringer • Generates String() methods of constants for humans • go get golang.org/x/tools/cmd/stringer

stringer: usage • stringer -type Sushi sushi.go package sushi type Sushi uint const ( Maguro Sushi = iota Ikura Uni Tamago ) // Code generated by "stringer -type Sushi sush package sushi import "fmt" const _Sushi_name = "MaguroIkuraUniTamago" var _Sushi_index = [...]uint8{0, 6, 11, 14, 20} func (i Sushi) String() string { if i >= Sushi(len(_Sushi_index)-1) { return fmt.Sprintf("Sushi(%d)", i) } return _Sushi_name[_Sushi_index[i]:_Sushi_i }

stringer: steps • Parse given source files • Find constants with specified types • Generate source codes to stringify them

stringer: type checking • Uses go/types API • go/types.Config.Check • Obtain “Defs”, map[*ast.Ident]types.Object • Maps each identifier to its type information

stringer: find constants • Traverse AST for ast.GenDecl • Collect the constants’ names along with their (int) values

stringer: generate source • Generate source code using fmt.Sprintf • Then format by go/format.Source // Arguments to format are: // [1]: type name // [2]: size of index element (8 for uint8 etc.) // [3]: less than zero check (for signed types) const stringOneRun = `func (i %[1]s) String() string { if %[3]si >= %[1]s(len(_%[1]s_index)-1) { return fmt.Sprintf("%[1]s(%%d)", i) } return _%[1]s_name[_%[1]s_index[i]:_%[1]s_index[i+1]] } `

CLI Interface

Common interfaces • eg. gofmt [flags] [path…] • Accept files • Accept directories (process their children) • Otherwise, use stdin

Common interfaces • Output to stdout by default • Overwrite input files if -w given • Output a diff if -d given (simply executing “diff” command) • List files which are to be updated if -l given