go generate: one file to rule them all

go generate: one file to rule them all

Code generation is ultimately the fifth stage of grief over Go's lack of generics. We've come to accept that Go doesn't have generics (yet!) but we still want to avoid the tedious task of writing duplicate code. With go generate we have a powerful tool to avoid writing boilerplate code. Using the Abstract Syntax Tree (ast) package we can extract sufficient information from an API file to generate middlewares (logging, instrumentation, etc.) and even documentation.

661bb4dee881617e676c4d954ce97a70?s=128

Konrad Reiche

April 13, 2019
Tweet

Transcript

  1. go:generate One File To Rule Them All

  2. October is a social network designed for the attention economy

    - Post links, images and text - Interact as yourself or anonymously - Earn coins for good content - Reward other creators for their content by tipping
  3. Go doesn’t have generics

  4. Denial Avoidance Confusion Elation Shock Fear Anger Frustration Irritation Anxiety

    Bargaining Struggling to find meaning Reaching out to others Telling one’s story Depression Overwhelmed Helplessness Hostility Flight Acceptance Exploring options New plan in place Moving on Kübler-Ross Model Five Stages of Grief
  5. Denial Go doesn’t lack generics. I can just use interface{},

    right?
  6. Denial Go doesn’t lack generics. I can just use interface{},

    right?
  7. Denial Go doesn’t lack generics. I can just use interface{},

    right?
  8. Anger That’s not fair. Why me? Why did I have

    to fall in love with a language that doesn’t have generics?
  9. Bargaining I don't need generics. Sometimes code duplication is better

    than unnecessary abstractions. Why write 100 lines of code just to avoid copy & pasting 10 lines of code once, or twice, or…?
  10. Depression In this state, the individual may become silent. Refuses

    code reviews and spends much of the time mournful and sullen at his desk.
  11. Acceptance Go doesn't have generics, but I can code my

    own parametric polymorphism by writing the tools to generate code I need automatically.
  12. Use Case REST API

  13. Project Structure ├── api │ ├── api.go ├── client │

    ├── client.go │ └── endpoints.go ├── main.go ├── metrics ├── model ├── server │ ├── endpoints.go │ ├── middleware.go │ ├── server.go ├── store │ └── store.go └── vendor ├── github.com ├── golang.org └── modules.txt
  14. const ( LoginEndpoint = "/login" CreatePostEndpoint = "/post" GetFeedEndpoint =

    "/feed" ) type API interface { Login(ctx context.Context, req LoginRequest) (*LoginResponse, error) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) GetFeed(ctx context.Context, req GetFeedRequest) (*GetFeedResponse, error) } api.go
  15. type CreatePostRequest struct { Body string `json:"body"` URL string `json:"url"`

    } type GetPriceResponse struct { ID int `json:"id"` } func (a *api) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) { // ... } api.go
  16. None
  17. API Middlewares API Middlewares allow to wrap endpoints for executing

    code before and/or after the endpoint has been processed. API
  18. API Middlewares API Middlewares allow to wrap endpoints for executing

    code before and/or after the endpoint has been processed. API Logging Log request, response and metadata
  19. API Middlewares API Middlewares allow to wrap endpoints for executing

    code before and/or after the endpoint has been processed. API Logging Instrumentation Log request, response and metadata Measure latency, count requests, etc.
  20. API Middlewares API Middlewares allow to wrap endpoints for executing

    code before and/or after the endpoint has been processed. API Logging Instrumentation Tracing Log request, response and metadata Measure latency, count requests, etc. Monitor code execution across services
  21. API Middlewares API Middlewares allow to wrap endpoints for executing

    code before and/or after the endpoint has been processed. API Logging Instrumentation Tracing Activity Recording Log request, response and metadata Measure latency, count requests, etc. Monitor code execution across services Record user activity for analytics
  22. type instrumentingMiddleware struct { requestCount metrics.Counter requestLatency metrics.TimeHistogram errorCount metrics.Counter

    log *logrus.Logger a API } instrumenting.go
  23. func NewInstrumentingMiddleware(a API, client *statsd.Client, log *logrus.Logger) API { requestCount

    := dogstatsd.NewCounter("request_count", 1, client, log) requestLatency := dogstatsd.NewTimeHistogram("request_latency", 1, client, log) errorCount := dogstatsd.NewCounter("error_count", 1, client, log) return &instrumentingMiddleware{ requestCount: requestCount, requestLatency: requestLatency, errorCount: errorCount, l: log, a: a, } } instrumenting.go
  24. func (im *instrumentingMiddleware) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) {

    var err error methodTag := metrics.Tag{Key: "method", Value: "CreatePost"} defer func(begin time.Time) { if err != nil { errorTag := metrics.Tag{Key: "error", Value: err.Error()} im.errorCount.With(methodTag).With(errorTag).Add(1) } im.requestCount.With(methodTag).Add(1) im.requestLatency.With(methodTag).Observe(time.Since(begin)) }(time.Now()) response, err := im.a.CreatePost(ctx, req) return response, err } instrumenting.go
  25. func main() { logger := log.NewLogger() client, err := statsd.New("127.0.0.1:8125",

    nil, nil) if err != nil { panic(err) } api := NewAPI() api = NewInstrumentingMiddleware(api, client, logger) server.Serve(api) } main.go
  26. func main() { // ... api := NewAPI() api =

    NewLoggingMiddleware(api, …) api = NewInstrumentingMiddleware(api, …) api = NewTracinggMiddleware(api, …) api = NewRecordingMiddleware(api, …) server.Serve(api) } All the middlewares
  27. func main() { // ... api := NewAPI() api =

    NewLoggingMiddleware(api, …) api = NewInstrumentingMiddleware(api, …) api = NewTracinggMiddleware(api, …) api = NewRecordingMiddleware(api, …) server.Serve(api) } All the middlewares
  28. It doesn’t stop here.

  29. type Client struct { conn *http.Client endpoint string } func

    NewClient(endpoint string) api.API { client := http.DefaultClient return &Client{ conn: client, endpoint: endpoint, } } client.go
  30. func (c *Client) CreatePost(ctx context.Context, req api.CreatePostRequest) (*api.CreatePostResponse, error) {

    r, err := http.NewRequest(http.MethodGet, CreatePostEndpoint), nil) if err != nil { return nil, err } resp, err := c.conn.Do(r) if err != nil { return nil, err } if resp.StatusCode != http.StatusOK { return nil, errors.New(resp.Status) } // ... client.go
  31. // ... var result api.CreatePostResponse decoder := json.NewDecoder(resp.Body) err =

    decoder.Decode(&result) if err != nil { return nil, err } return &result, err } client.go
  32. func (s *Server) CreatePostHandleFunc(w http.ResponseWriter, r *http.Request) { var req

    api.CreatePostRequest params := mux.Vars(r) dec, err := mapstructure.NewDecoder(&mapstructure.DecoderConfig{ TagName: "json", Result: &req, }) if err != nil { Encode(nil, w, nil, err) return } err = dec.Decode(params) if err != nil { Encode(nil, w, nil, err) } // ... server.go
  33. // ... err = req.Validate() if err != nil {

    Encode(nil, w, nil, err) return } resp, err := s.api.CreatePost(r.Context(), req) if err != nil { Encode(nil, w, nil, err) return } Encode(nil, w, resp, nil) } server.go
  34. Requirements Under Fire API keeps changing: - Request parameter &

    response fields change - New features result in new endpoints - Deprecated features result in endpoints getting deleted What that means for our project: - Every changes requires to update all files related to api.go - 4 Middlewares + Server Encoder/Decoder + Client Library = 6 Files
  35. None
  36. 6

  37. 6 Files?

  38. 6 Files?

  39. There is always another layer of abstraction you can add.

  40. There is always another layer of abstraction you can add.

  41. Code Generation The Antidote

  42. Code Generation - Work around the limitations of a language,

    in the case of Go: generics - Delegate task to write boilerplate code to a machine - It’s mentally more satisfying to do meta-programming - Less prone to errors Model Tool Templates Code
  43. go generate The go binary comes with a subcommand called

    generate which scans packages for generate directives and for each directive, the specified generator is run; if files are named, they must be Go source files and generation happens only for directives in those files, i.e. you would put this line at the top of your file: //go:generate echo ‘Hello Gophercon Russia’ $ go generate api.go > Hello Gophercon Russia
  44. package main import ( "fmt" "os" ) func main() {

    fmt.Println(os.Getenv("GOFILE")) } cmd/apigen.go
  45. package main import ( "fmt" "os" ) func main() {

    fmt.Println(os.Getenv("GOFILE")) } cmd/apigen.go //go:generate go run ../cmd/apigen.go $ go generate api/api.go
  46. package main import ( "fmt" "os" ) func main() {

    fmt.Println(os.Getenv("GOFILE")) } cmd/apigen.go //go:generate go run ../cmd/apigen.go $ go generate api/api.go > ../api/api.go
  47. Let’s automate.

  48. func (im *instrumentingMiddleware) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) {

    var err error methodTag := metrics.Tag{Key: "method", Value: "CreatePost"} defer func(begin time.Time) { if err != nil { errorTag := metrics.Tag{Key: "error", Value: err.Error()} im.errorCount.With(methodTag).With(errorTag).Add(1) } im.requestCount.With(methodTag).Add(1) im.requestLatency.With(methodTag).Observe(time.Since(begin)) }(time.Now()) response, err := im.a.CreatePost(ctx, req) return response, err } instrumenting.go
  49. func (im *instrumentingMiddleware) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) {

    var err error methodTag := metrics.Tag{Key: "method", Value: "CreatePost"} defer func(begin time.Time) { if err != nil { errorTag := metrics.Tag{Key: "error", Value: err.Error()} im.errorCount.With(methodTag).With(errorTag).Add(1) } im.requestCount.With(methodTag).Add(1) im.requestLatency.With(methodTag).Observe(time.Since(begin)) }(time.Now()) response, err := im.a.CreatePost(ctx, req) return response, err } instrumenting.go
  50. func (im *instrumentingMiddleware) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) {

    var err error methodTag := metrics.Tag{Key: "method", Value: "CreatePost"} defer func(begin time.Time) { if err != nil { errorTag := metrics.Tag{Key: "error", Value: err.Error()} im.errorCount.With(methodTag).With(errorTag).Add(1) } im.requestCount.With(methodTag).Add(1) im.requestLatency.With(methodTag).Observe(time.Since(begin)) }(time.Now()) response, err := im.a.CreatePost(ctx, req) return response, err } instrumenting.go
  51. Generate Architecture go:generate Directive $ go generate ./... . Tool

    Invokes tool with file name as environment variable Templates Parse file(s) containing the directive Boilerplate Code Pass model to templates api/logging.go api/instrumentation.go api/recording.go api/tracing.go client/endpoints.go server/endpoints.go
  52. AST Package Package declares the types used to represent syntax

    tree for Go packages. Abstract Syntax Tree A tree representation of the abstract syntactic structure of source code written in a programming language. Each node of the tree denotes a construct occurring in the source code. - parser.ParseFile to create the AST by parsing an actual Go source file - ast.Inspect allows to iterate every node and extract further data
  53. type CreatePostRequest struct { Body string `json:"body"` URL string `json:"url"`

    } type GetPriceResponse struct { ID int `json:"id"` } func (a *api) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) { // ... } api.go
  54. func (p *Parser) Parse() error { fset := token.NewFileSet() f,

    err := parser.ParseFile(fset, os.Getenv("GOFILE"), nil, 0) if err != nil { return nil, err } ast.Inspect(file, func(n ast.Node) bool { switch x := n.(type) { case *ast.TypeSpec: if err := p.parseType(x); err != nil { return false } return true }) return p.generate() }
  55. func (p *Parser) parseType(st *ast.TypeSpec) error { if strings.HasSuffix(st.Name.Name, "Request")

    { endpoint := strings.Replace(st.Name.Name, "Request", "", -1) if p.EndpointsByName[endpoint] == nil { p.EndpointsByName[endpoint] = NewEndpoint(endpoint) } } return nil }
  56. // Code generated by apigen; DO NOT EDIT. // //

    Source: api/api.go // Template: templates/instrumentation.go.tmpl package api // ... {{range .Endpoints}} func (im *instrumentingMiddleware) {{.Name}}(ctx context.Context, req {{.Name}}Request) (*{{.Name}}Response, error) { var err error methodTag := metrics.Tag{Key: "method", Value: "{{.Name}}"} defer func(begin time.Time) { if err != nil { errorTag := metrics.Tag{Key: "error", Value: err.Error()} im.errorCount.With(methodTag).With(errorTag).Add(1) } im.requestCount.With(methodTag).Add(1) im.requestLatency.With(methodTag).Observe(time.Since(begin)) }(time.Now()) response, err := im.a.{{.Name}}(ctx, req) return response, err } {{end}}
  57. instrumentationTemplate, err := ioutil.ReadFile("../templates/instrumentation.go.tmpl") if err != nil { return

    nil, err } tmpl := template.Must(template.New("instrumentation").Parse(string(instrumentationTemplate))) generateCode(tmpl, "instrumentation.go") cmd/apigen.go
  58. func (p *Parser) generateCode(tmpl *template.Template, fn string) error { buf

    := bytes.NewBuffer([]byte{}) err := tmpl.Execute(buf, p) if err != nil { return err } res, err := imports.Process(fn, buf.Bytes(), nil) if err != nil { return err } return ioutil.WriteFile(fn, res, 0666) } cmd/apigen.go
  59. Demo

  60. func (p *Parser) Parse() error { fset := token.NewFileSet() f,

    err := parser.ParseFile(fset, os.Getenv("GOFILE"), nil, 0) if err != nil { return nil, err } ast.Inspect(file, func(n ast.Node) bool { switch x := n.(type) { case *ast.TypeSpec: if err := p.parseType(x); err != nil { return false } return true }) return p.generate() }
  61. func (p *Parser) Parse() error { fset := token.NewFileSet() f,

    err := parser.ParseFile(fset, os.Getenv("GOFILE"), nil, parser.ParseComments) if err != nil { return nil, err } ast.Inspect(file, func(n ast.Node) bool { switch x := n.(type) { case *ast.TypeSpec: if err := p.parseType(x); err != nil { return false } return true }) return p.generate() }
  62. func (p *Parser) parseFunction(fd *ast.FuncDecl) { if fd.Recv == nil

    { return } if recv, ok := fd.Recv.List[0].Type.(*ast.StarExpr); ok { if ident, ok := recv.X.(*ast.Ident); ok { name := fd.Name.Name description := fd.Doc.Text() firstChar := string(name[0]) if ident.Name == "api" && firstChar == strings.ToUpper(firstChar) { p.AddEndpoint(name, description) } } } }
  63. func (p *Parser) addParameter(endpoint string, st *ast.StructType) { for _,

    field := range st.Fields.List { params := Parameter{ Field: field.Names[0].Name, Description: field.Doc.Text(), Tag: parseTag(field.Tag.Value), Type: mapFieldType(field.Type), } if p.EndpointsByName[endpoint] == nil { p.EndpointsByName[endpoint] = NewEndpoint(endpoint) } p.EndpointsByName[endpoint].Parameters = append(p.EndpointsByName[endpoint].Parameters, params) p.EndpointsByName[endpoint].ParameterByName[params.Field] = params } }
  64. Best Practices - Include a DO NOT EDIT line in

    your templates for your colleagues to see - Intended to be used by the author of the Go package, not its clients - go:generate created code should be committed to the repository - Generate your unit tests, too - Make your code generation tool robust
  65. A meta tool to build your own tools - go:generate

    is a meta tool that you can use as a building block to build your own tools: fitted for your purpose - Identity repetitive tasks in your development cycle and try to extract the essence into tooling - Make the tool part of your existing automation as well, i.e. run it in your CI/CD pipeline - Beware of breaking changes, it’s another tool in your stack, so it may require maintenance
  66. Thank you. @konradreiche github.com/konradreiche/apigen

  67. Questions? @konradreiche github.com/konradreiche/apigen