go generate: one file to rule them all

Code generation is ultimately the fifth stage of grief over Go's lack of generics. We've come to accept that Go doesn't have generics (yet!) but we still want to avoid the tedious task of writing duplicate code. With go generate we have a powerful tool to avoid writing boilerplate code. Using the Abstract Syntax Tree (ast) package we can extract sufficient information from an API file to generate middlewares (logging, instrumentation, etc.) and even documentation.

Transcript

1. go:generate One File To Rule Them All
2. October is a social network designed for the attention economy - Post links, images and text - Interact as yourself or anonymously - Earn coins for good content - Reward other creators for their content by tipping
3. Go doesn’t have generics
4. Denial Avoidance Confusion Elation Shock Fear Anger Frustration Irritation Anxiety Bargaining Struggling to find meaning Reaching out to others Telling one’s story Depression Overwhelmed Helplessness Hostility Flight Acceptance Exploring options New plan in place Moving on Kübler-Ross Model Five Stages of Grief
5. Denial Go doesn’t lack generics. I can just use interface{}, right?
6. Denial Go doesn’t lack generics. I can just use interface{}, right?
7. Denial Go doesn’t lack generics. I can just use interface{}, right?
8. Anger That’s not fair. Why me? Why did I have to fall in love with a language that doesn’t have generics?
9. Bargaining I don't need generics. Sometimes code duplication is better than unnecessary abstractions. Why write 100 lines of code just to avoid copy & pasting 10 lines of code once, or twice, or…?
10. Depression In this state, the individual may become silent. Refuses code reviews and spends much of the time mournful and sullen at his desk.
11. Acceptance Go doesn't have generics, but I can code my own parametric polymorphism by writing the tools to generate code I need automatically.
12. Use Case REST API
13. Project Structure ├── api │ ├── api.go ├── client │ ├── client.go │ └── endpoints.go ├── main.go ├── metrics ├── model ├── server │ ├── endpoints.go │ ├── middleware.go │ ├── server.go ├── store │ └── store.go └── vendor ├── github.com ├── golang.org └── modules.txt
14. const ( LoginEndpoint = "/login" CreatePostEndpoint = "/post" GetFeedEndpoint = "/feed" ) type API interface { Login(ctx context.Context, req LoginRequest) (*LoginResponse, error) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) GetFeed(ctx context.Context, req GetFeedRequest) (*GetFeedResponse, error) } api.go
15. type CreatePostRequest struct { Body string `json:"body"` URL string `json:"url"` } type GetPriceResponse struct { ID int `json:"id"` } func (a *api) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) { // ... } api.go
17. API Middlewares API Middlewares allow to wrap endpoints for executing code before and/or after the endpoint has been processed. API
18. API Middlewares API Middlewares allow to wrap endpoints for executing code before and/or after the endpoint has been processed. API Logging Log request, response and metadata
19. API Middlewares API Middlewares allow to wrap endpoints for executing code before and/or after the endpoint has been processed. API Logging Instrumentation Log request, response and metadata Measure latency, count requests, etc.
20. API Middlewares API Middlewares allow to wrap endpoints for executing code before and/or after the endpoint has been processed. API Logging Instrumentation Tracing Log request, response and metadata Measure latency, count requests, etc. Monitor code execution across services
21. API Middlewares API Middlewares allow to wrap endpoints for executing code before and/or after the endpoint has been processed. API Logging Instrumentation Tracing Activity Recording Log request, response and metadata Measure latency, count requests, etc. Monitor code execution across services Record user activity for analytics
22. type instrumentingMiddleware struct { requestCount metrics.Counter requestLatency metrics.TimeHistogram errorCount metrics.Counter log *logrus.Logger a API } instrumenting.go
23. func NewInstrumentingMiddleware(a API, client *statsd.Client, log *logrus.Logger) API { requestCount := dogstatsd.NewCounter("request_count", 1, client, log) requestLatency := dogstatsd.NewTimeHistogram("request_latency", 1, client, log) errorCount := dogstatsd.NewCounter("error_count", 1, client, log) return &instrumentingMiddleware{ requestCount: requestCount, requestLatency: requestLatency, errorCount: errorCount, l: log, a: a, } } instrumenting.go
24. func (im *instrumentingMiddleware) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) { var err error methodTag := metrics.Tag{Key: "method", Value: "CreatePost"} defer func(begin time.Time) { if err != nil { errorTag := metrics.Tag{Key: "error", Value: err.Error()} im.errorCount.With(methodTag).With(errorTag).Add(1) } im.requestCount.With(methodTag).Add(1) im.requestLatency.With(methodTag).Observe(time.Since(begin)) }(time.Now()) response, err := im.a.CreatePost(ctx, req) return response, err } instrumenting.go
25. func main() { logger := log.NewLogger() client, err := statsd.New("127.0.0.1:8125", nil, nil) if err != nil { panic(err) } api := NewAPI() api = NewInstrumentingMiddleware(api, client, logger) server.Serve(api) } main.go
26. func main() { // ... api := NewAPI() api = NewLoggingMiddleware(api, …) api = NewInstrumentingMiddleware(api, …) api = NewTracinggMiddleware(api, …) api = NewRecordingMiddleware(api, …) server.Serve(api) } All the middlewares
27. func main() { // ... api := NewAPI() api = NewLoggingMiddleware(api, …) api = NewInstrumentingMiddleware(api, …) api = NewTracinggMiddleware(api, …) api = NewRecordingMiddleware(api, …) server.Serve(api) } All the middlewares
28. It doesn’t stop here.
29. type Client struct { conn *http.Client endpoint string } func NewClient(endpoint string) api.API { client := http.DefaultClient return &Client{ conn: client, endpoint: endpoint, } } client.go
30. func (c *Client) CreatePost(ctx context.Context, req api.CreatePostRequest) (*api.CreatePostResponse, error) { r, err := http.NewRequest(http.MethodGet, CreatePostEndpoint), nil) if err != nil { return nil, err } resp, err := c.conn.Do(r) if err != nil { return nil, err } if resp.StatusCode != http.StatusOK { return nil, errors.New(resp.Status) } // ... client.go
31. // ... var result api.CreatePostResponse decoder := json.NewDecoder(resp.Body) err = decoder.Decode(&result) if err != nil { return nil, err } return &result, err } client.go
32. func (s *Server) CreatePostHandleFunc(w http.ResponseWriter, r *http.Request) { var req api.CreatePostRequest params := mux.Vars(r) dec, err := mapstructure.NewDecoder(&mapstructure.DecoderConfig{ TagName: "json", Result: &req, }) if err != nil { Encode(nil, w, nil, err) return } err = dec.Decode(params) if err != nil { Encode(nil, w, nil, err) } // ... server.go
33. // ... err = req.Validate() if err != nil { Encode(nil, w, nil, err) return } resp, err := s.api.CreatePost(r.Context(), req) if err != nil { Encode(nil, w, nil, err) return } Encode(nil, w, resp, nil) } server.go
34. Requirements Under Fire API keeps changing: - Request parameter & response fields change - New features result in new endpoints - Deprecated features result in endpoints getting deleted What that means for our project: - Every changes requires to update all files related to api.go - 4 Middlewares + Server Encoder/Decoder + Client Library = 6 Files
37. 6 Files?
38. 6 Files?
39. There is always another layer of abstraction you can add.
40. There is always another layer of abstraction you can add.
41. Code Generation The Antidote
42. Code Generation - Work around the limitations of a language, in the case of Go: generics - Delegate task to write boilerplate code to a machine - It’s mentally more satisfying to do meta-programming - Less prone to errors Model Tool Templates Code
43. go generate The go binary comes with a subcommand called generate which scans packages for generate directives and for each directive, the specified generator is run; if files are named, they must be Go source files and generation happens only for directives in those files, i.e. you would put this line at the top of your file: //go:generate echo ‘Hello Gophercon Russia’ $ go generate api.go > Hello Gophercon Russia
44. package main import ( "fmt" "os" ) func main() { fmt.Println(os.Getenv("GOFILE")) } cmd/apigen.go
45. package main import ( "fmt" "os" ) func main() { fmt.Println(os.Getenv("GOFILE")) } cmd/apigen.go //go:generate go run ../cmd/apigen.go $ go generate api/api.go
46. package main import ( "fmt" "os" ) func main() { fmt.Println(os.Getenv("GOFILE")) } cmd/apigen.go //go:generate go run ../cmd/apigen.go $ go generate api/api.go > ../api/api.go
47. Let’s automate.
48. func (im *instrumentingMiddleware) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) { var err error methodTag := metrics.Tag{Key: "method", Value: "CreatePost"} defer func(begin time.Time) { if err != nil { errorTag := metrics.Tag{Key: "error", Value: err.Error()} im.errorCount.With(methodTag).With(errorTag).Add(1) } im.requestCount.With(methodTag).Add(1) im.requestLatency.With(methodTag).Observe(time.Since(begin)) }(time.Now()) response, err := im.a.CreatePost(ctx, req) return response, err } instrumenting.go
49. func (im *instrumentingMiddleware) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) { var err error methodTag := metrics.Tag{Key: "method", Value: "CreatePost"} defer func(begin time.Time) { if err != nil { errorTag := metrics.Tag{Key: "error", Value: err.Error()} im.errorCount.With(methodTag).With(errorTag).Add(1) } im.requestCount.With(methodTag).Add(1) im.requestLatency.With(methodTag).Observe(time.Since(begin)) }(time.Now()) response, err := im.a.CreatePost(ctx, req) return response, err } instrumenting.go
50. func (im *instrumentingMiddleware) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) { var err error methodTag := metrics.Tag{Key: "method", Value: "CreatePost"} defer func(begin time.Time) { if err != nil { errorTag := metrics.Tag{Key: "error", Value: err.Error()} im.errorCount.With(methodTag).With(errorTag).Add(1) } im.requestCount.With(methodTag).Add(1) im.requestLatency.With(methodTag).Observe(time.Since(begin)) }(time.Now()) response, err := im.a.CreatePost(ctx, req) return response, err } instrumenting.go
51. Generate Architecture go:generate Directive $ go generate ./... . Tool Invokes tool with file name as environment variable Templates Parse file(s) containing the directive Boilerplate Code Pass model to templates api/logging.go api/instrumentation.go api/recording.go api/tracing.go client/endpoints.go server/endpoints.go
52. AST Package Package declares the types used to represent syntax tree for Go packages. Abstract Syntax Tree A tree representation of the abstract syntactic structure of source code written in a programming language. Each node of the tree denotes a construct occurring in the source code. - parser.ParseFile to create the AST by parsing an actual Go source file - ast.Inspect allows to iterate every node and extract further data
53. type CreatePostRequest struct { Body string `json:"body"` URL string `json:"url"` } type GetPriceResponse struct { ID int `json:"id"` } func (a *api) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) { // ... } api.go
54. func (p *Parser) Parse() error { fset := token.NewFileSet() f, err := parser.ParseFile(fset, os.Getenv("GOFILE"), nil, 0) if err != nil { return nil, err } ast.Inspect(file, func(n ast.Node) bool { switch x := n.(type) { case *ast.TypeSpec: if err := p.parseType(x); err != nil { return false } return true }) return p.generate() }
55. func (p *Parser) parseType(st *ast.TypeSpec) error { if strings.HasSuffix(st.Name.Name, "Request") { endpoint := strings.Replace(st.Name.Name, "Request", "", -1) if p.EndpointsByName[endpoint] == nil { p.EndpointsByName[endpoint] = NewEndpoint(endpoint) } } return nil }
56. // Code generated by apigen; DO NOT EDIT. // // Source: api/api.go // Template: templates/instrumentation.go.tmpl package api // ... {{range .Endpoints}} func (im *instrumentingMiddleware) {{.Name}}(ctx context.Context, req {{.Name}}Request) (*{{.Name}}Response, error) { var err error methodTag := metrics.Tag{Key: "method", Value: "{{.Name}}"} defer func(begin time.Time) { if err != nil { errorTag := metrics.Tag{Key: "error", Value: err.Error()} im.errorCount.With(methodTag).With(errorTag).Add(1) } im.requestCount.With(methodTag).Add(1) im.requestLatency.With(methodTag).Observe(time.Since(begin)) }(time.Now()) response, err := im.a.{{.Name}}(ctx, req) return response, err } {{end}}
57. instrumentationTemplate, err := ioutil.ReadFile("../templates/instrumentation.go.tmpl") if err != nil { return nil, err } tmpl := template.Must(template.New("instrumentation").Parse(string(instrumentationTemplate))) generateCode(tmpl, "instrumentation.go") cmd/apigen.go
58. func (p *Parser) generateCode(tmpl *template.Template, fn string) error { buf := bytes.NewBuffer([]byte{}) err := tmpl.Execute(buf, p) if err != nil { return err } res, err := imports.Process(fn, buf.Bytes(), nil) if err != nil { return err } return ioutil.WriteFile(fn, res, 0666) } cmd/apigen.go
59. Demo
60. func (p *Parser) Parse() error { fset := token.NewFileSet() f, err := parser.ParseFile(fset, os.Getenv("GOFILE"), nil, 0) if err != nil { return nil, err } ast.Inspect(file, func(n ast.Node) bool { switch x := n.(type) { case *ast.TypeSpec: if err := p.parseType(x); err != nil { return false } return true }) return p.generate() }
61. func (p *Parser) Parse() error { fset := token.NewFileSet() f, err := parser.ParseFile(fset, os.Getenv("GOFILE"), nil, parser.ParseComments) if err != nil { return nil, err } ast.Inspect(file, func(n ast.Node) bool { switch x := n.(type) { case *ast.TypeSpec: if err := p.parseType(x); err != nil { return false } return true }) return p.generate() }
62. func (p *Parser) parseFunction(fd *ast.FuncDecl) { if fd.Recv == nil { return } if recv, ok := fd.Recv.List[0].Type.(*ast.StarExpr); ok { if ident, ok := recv.X.(*ast.Ident); ok { name := fd.Name.Name description := fd.Doc.Text() firstChar := string(name[0]) if ident.Name == "api" && firstChar == strings.ToUpper(firstChar) { p.AddEndpoint(name, description) } } } }
63. func (p *Parser) addParameter(endpoint string, st *ast.StructType) { for _, field := range st.Fields.List { params := Parameter{ Field: field.Names[0].Name, Description: field.Doc.Text(), Tag: parseTag(field.Tag.Value), Type: mapFieldType(field.Type), } if p.EndpointsByName[endpoint] == nil { p.EndpointsByName[endpoint] = NewEndpoint(endpoint) } p.EndpointsByName[endpoint].Parameters = append(p.EndpointsByName[endpoint].Parameters, params) p.EndpointsByName[endpoint].ParameterByName[params.Field] = params } }
64. Best Practices - Include a DO NOT EDIT line in your templates for your colleagues to see - Intended to be used by the author of the Go package, not its clients - go:generate created code should be committed to the repository - Generate your unit tests, too - Make your code generation tool robust
65. A meta tool to build your own tools - go:generate is a meta tool that you can use as a building block to build your own tools: fitted for your purpose - Identity repetitive tasks in your development cycle and try to extract the essence into tooling - Make the tool part of your existing automation as well, i.e. run it in your CI/CD pipeline - Beware of breaking changes, it’s another tool in your stack, so it may require maintenance
66. Thank you. @konradreiche github.com/konradreiche/apigen
67. Questions? @konradreiche github.com/konradreiche/apigen