go generate: one file to rule them all

Slide 1

Slide 1 text

go:generate One File To Rule Them All

Slide 2

Slide 2 text

October is a social network designed for the attention economy - Post links, images and text - Interact as yourself or anonymously - Earn coins for good content - Reward other creators for their content by tipping

Slide 3

Slide 3 text

Go doesn’t have generics

Slide 4

Slide 4 text

Denial Avoidance Confusion Elation Shock Fear Anger Frustration Irritation Anxiety Bargaining Struggling to find meaning Reaching out to others Telling one’s story Depression Overwhelmed Helplessness Hostility Flight Acceptance Exploring options New plan in place Moving on Kübler-Ross Model Five Stages of Grief

Slide 5

Slide 5 text

Denial Go doesn’t lack generics. I can just use interface{}, right?

Slide 6

Slide 6 text

Denial Go doesn’t lack generics. I can just use interface{}, right?

Slide 7

Slide 7 text

Denial Go doesn’t lack generics. I can just use interface{}, right?

Slide 8

Slide 8 text

Anger That’s not fair. Why me? Why did I have to fall in love with a language that doesn’t have generics?

Slide 9

Slide 9 text

Bargaining I don't need generics. Sometimes code duplication is better than unnecessary abstractions. Why write 100 lines of code just to avoid copy & pasting 10 lines of code once, or twice, or…?

Slide 10

Slide 10 text

Depression In this state, the individual may become silent. Refuses code reviews and spends much of the time mournful and sullen at his desk.

Slide 11

Slide 11 text

Acceptance Go doesn't have generics, but I can code my own parametric polymorphism by writing the tools to generate code I need automatically.

Slide 12

Slide 12 text

Use Case REST API

Slide 13

Slide 13 text

Project Structure ├── api │ ├── api.go ├── client │ ├── client.go │ └── endpoints.go ├── main.go ├── metrics ├── model ├── server │ ├── endpoints.go │ ├── middleware.go │ ├── server.go ├── store │ └── store.go └── vendor ├── github.com ├── golang.org └── modules.txt

Slide 14

Slide 14 text

const ( LoginEndpoint = "/login" CreatePostEndpoint = "/post" GetFeedEndpoint = "/feed" ) type API interface { Login(ctx context.Context, req LoginRequest) (*LoginResponse, error) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) GetFeed(ctx context.Context, req GetFeedRequest) (*GetFeedResponse, error) } api.go

Slide 15

Slide 15 text

type CreatePostRequest struct { Body string `json:"body"` URL string `json:"url"` } type GetPriceResponse struct { ID int `json:"id"` } func (a *api) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) { // ... } api.go

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

API Middlewares API Middlewares allow to wrap endpoints for executing code before and/or after the endpoint has been processed. API

Slide 18

Slide 18 text

API Middlewares API Middlewares allow to wrap endpoints for executing code before and/or after the endpoint has been processed. API Logging Log request, response and metadata

Slide 19

Slide 19 text

API Middlewares API Middlewares allow to wrap endpoints for executing code before and/or after the endpoint has been processed. API Logging Instrumentation Log request, response and metadata Measure latency, count requests, etc.

Slide 20

Slide 20 text

API Middlewares API Middlewares allow to wrap endpoints for executing code before and/or after the endpoint has been processed. API Logging Instrumentation Tracing Log request, response and metadata Measure latency, count requests, etc. Monitor code execution across services

Slide 21

Slide 21 text

API Middlewares API Middlewares allow to wrap endpoints for executing code before and/or after the endpoint has been processed. API Logging Instrumentation Tracing Activity Recording Log request, response and metadata Measure latency, count requests, etc. Monitor code execution across services Record user activity for analytics

Slide 22

Slide 22 text

type instrumentingMiddleware struct { requestCount metrics.Counter requestLatency metrics.TimeHistogram errorCount metrics.Counter log *logrus.Logger a API } instrumenting.go

Slide 23

Slide 23 text

func NewInstrumentingMiddleware(a API, client *statsd.Client, log *logrus.Logger) API { requestCount := dogstatsd.NewCounter("request_count", 1, client, log) requestLatency := dogstatsd.NewTimeHistogram("request_latency", 1, client, log) errorCount := dogstatsd.NewCounter("error_count", 1, client, log) return &instrumentingMiddleware{ requestCount: requestCount, requestLatency: requestLatency, errorCount: errorCount, l: log, a: a, } } instrumenting.go

Slide 24

Slide 24 text

func (im *instrumentingMiddleware) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) { var err error methodTag := metrics.Tag{Key: "method", Value: "CreatePost"} defer func(begin time.Time) { if err != nil { errorTag := metrics.Tag{Key: "error", Value: err.Error()} im.errorCount.With(methodTag).With(errorTag).Add(1) } im.requestCount.With(methodTag).Add(1) im.requestLatency.With(methodTag).Observe(time.Since(begin)) }(time.Now()) response, err := im.a.CreatePost(ctx, req) return response, err } instrumenting.go

Slide 25

Slide 25 text

func main() { logger := log.NewLogger() client, err := statsd.New("127.0.0.1:8125", nil, nil) if err != nil { panic(err) } api := NewAPI() api = NewInstrumentingMiddleware(api, client, logger) server.Serve(api) } main.go

Slide 26

Slide 26 text

func main() { // ... api := NewAPI() api = NewLoggingMiddleware(api, …) api = NewInstrumentingMiddleware(api, …) api = NewTracinggMiddleware(api, …) api = NewRecordingMiddleware(api, …) server.Serve(api) } All the middlewares

Slide 27

Slide 27 text

Slide 28

Slide 28 text

It doesn’t stop here.

Slide 29

Slide 29 text

type Client struct { conn *http.Client endpoint string } func NewClient(endpoint string) api.API { client := http.DefaultClient return &Client{ conn: client, endpoint: endpoint, } } client.go

Slide 30

Slide 30 text

func (c *Client) CreatePost(ctx context.Context, req api.CreatePostRequest) (*api.CreatePostResponse, error) { r, err := http.NewRequest(http.MethodGet, CreatePostEndpoint), nil) if err != nil { return nil, err } resp, err := c.conn.Do(r) if err != nil { return nil, err } if resp.StatusCode != http.StatusOK { return nil, errors.New(resp.Status) } // ... client.go

Slide 31

Slide 31 text

// ... var result api.CreatePostResponse decoder := json.NewDecoder(resp.Body) err = decoder.Decode(&result) if err != nil { return nil, err } return &result, err } client.go

Slide 32

Slide 32 text

func (s *Server) CreatePostHandleFunc(w http.ResponseWriter, r *http.Request) { var req api.CreatePostRequest params := mux.Vars(r) dec, err := mapstructure.NewDecoder(&mapstructure.DecoderConfig{ TagName: "json", Result: &req, }) if err != nil { Encode(nil, w, nil, err) return } err = dec.Decode(params) if err != nil { Encode(nil, w, nil, err) } // ... server.go

Slide 33

Slide 33 text

// ... err = req.Validate() if err != nil { Encode(nil, w, nil, err) return } resp, err := s.api.CreatePost(r.Context(), req) if err != nil { Encode(nil, w, nil, err) return } Encode(nil, w, resp, nil) } server.go

Slide 34

Slide 34 text

Requirements Under Fire API keeps changing: - Request parameter & response fields change - New features result in new endpoints - Deprecated features result in endpoints getting deleted What that means for our project: - Every changes requires to update all files related to api.go - 4 Middlewares + Server Encoder/Decoder + Client Library = 6 Files

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

Slide 37

Slide 37 text

6 Files?

Slide 38

Slide 38 text

6 Files?

Slide 39

Slide 39 text

There is always another layer of abstraction you can add.

Slide 40

Slide 40 text

There is always another layer of abstraction you can add.

Slide 41

Slide 41 text

Code Generation The Antidote

Slide 42

Slide 42 text

Code Generation - Work around the limitations of a language, in the case of Go: generics - Delegate task to write boilerplate code to a machine - It’s mentally more satisfying to do meta-programming - Less prone to errors Model Tool Templates Code

Slide 43

Slide 43 text

go generate The go binary comes with a subcommand called generate which scans packages for generate directives and for each directive, the specified generator is run; if files are named, they must be Go source files and generation happens only for directives in those files, i.e. you would put this line at the top of your file: //go:generate echo ‘Hello Gophercon Russia’ $ go generate api.go > Hello Gophercon Russia

Slide 44

Slide 44 text

package main import ( "fmt" "os" ) func main() { fmt.Println(os.Getenv("GOFILE")) } cmd/apigen.go

Slide 45

Slide 45 text

package main import ( "fmt" "os" ) func main() { fmt.Println(os.Getenv("GOFILE")) } cmd/apigen.go //go:generate go run ../cmd/apigen.go $ go generate api/api.go

Slide 46

Slide 46 text

package main import ( "fmt" "os" ) func main() { fmt.Println(os.Getenv("GOFILE")) } cmd/apigen.go //go:generate go run ../cmd/apigen.go $ go generate api/api.go > ../api/api.go

Slide 47

Slide 47 text

Let’s automate.

Slide 48

Slide 48 text

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Generate Architecture go:generate Directive $ go generate ./... . Tool Invokes tool with file name as environment variable Templates Parse file(s) containing the directive Boilerplate Code Pass model to templates api/logging.go api/instrumentation.go api/recording.go api/tracing.go client/endpoints.go server/endpoints.go

Slide 52

Slide 52 text

AST Package Package declares the types used to represent syntax tree for Go packages. Abstract Syntax Tree A tree representation of the abstract syntactic structure of source code written in a programming language. Each node of the tree denotes a construct occurring in the source code. - parser.ParseFile to create the AST by parsing an actual Go source file - ast.Inspect allows to iterate every node and extract further data

Slide 53

Slide 53 text

Slide 54

Slide 54 text

func (p *Parser) Parse() error { fset := token.NewFileSet() f, err := parser.ParseFile(fset, os.Getenv("GOFILE"), nil, 0) if err != nil { return nil, err } ast.Inspect(file, func(n ast.Node) bool { switch x := n.(type) { case *ast.TypeSpec: if err := p.parseType(x); err != nil { return false } return true }) return p.generate() }

Slide 55

Slide 55 text

func (p *Parser) parseType(st *ast.TypeSpec) error { if strings.HasSuffix(st.Name.Name, "Request") { endpoint := strings.Replace(st.Name.Name, "Request", "", -1) if p.EndpointsByName[endpoint] == nil { p.EndpointsByName[endpoint] = NewEndpoint(endpoint) } } return nil }

Slide 56

Slide 56 text

// Code generated by apigen; DO NOT EDIT. // // Source: api/api.go // Template: templates/instrumentation.go.tmpl package api // ... {{range .Endpoints}} func (im *instrumentingMiddleware) {{.Name}}(ctx context.Context, req {{.Name}}Request) (*{{.Name}}Response, error) { var err error methodTag := metrics.Tag{Key: "method", Value: "{{.Name}}"} defer func(begin time.Time) { if err != nil { errorTag := metrics.Tag{Key: "error", Value: err.Error()} im.errorCount.With(methodTag).With(errorTag).Add(1) } im.requestCount.With(methodTag).Add(1) im.requestLatency.With(methodTag).Observe(time.Since(begin)) }(time.Now()) response, err := im.a.{{.Name}}(ctx, req) return response, err } {{end}}

Slide 57

Slide 57 text

instrumentationTemplate, err := ioutil.ReadFile("../templates/instrumentation.go.tmpl") if err != nil { return nil, err } tmpl := template.Must(template.New("instrumentation").Parse(string(instrumentationTemplate))) generateCode(tmpl, "instrumentation.go") cmd/apigen.go

Slide 58

Slide 58 text

func (p *Parser) generateCode(tmpl *template.Template, fn string) error { buf := bytes.NewBuffer([]byte{}) err := tmpl.Execute(buf, p) if err != nil { return err } res, err := imports.Process(fn, buf.Bytes(), nil) if err != nil { return err } return ioutil.WriteFile(fn, res, 0666) } cmd/apigen.go

Slide 59

Slide 59 text

Demo

Slide 60

Slide 60 text

Slide 61

Slide 61 text

func (p *Parser) Parse() error { fset := token.NewFileSet() f, err := parser.ParseFile(fset, os.Getenv("GOFILE"), nil, parser.ParseComments) if err != nil { return nil, err } ast.Inspect(file, func(n ast.Node) bool { switch x := n.(type) { case *ast.TypeSpec: if err := p.parseType(x); err != nil { return false } return true }) return p.generate() }

Slide 62

Slide 62 text

func (p *Parser) parseFunction(fd *ast.FuncDecl) { if fd.Recv == nil { return } if recv, ok := fd.Recv.List[0].Type.(*ast.StarExpr); ok { if ident, ok := recv.X.(*ast.Ident); ok { name := fd.Name.Name description := fd.Doc.Text() firstChar := string(name[0]) if ident.Name == "api" && firstChar == strings.ToUpper(firstChar) { p.AddEndpoint(name, description) } } } }

Slide 63

Slide 63 text

func (p *Parser) addParameter(endpoint string, st *ast.StructType) { for _, field := range st.Fields.List { params := Parameter{ Field: field.Names[0].Name, Description: field.Doc.Text(), Tag: parseTag(field.Tag.Value), Type: mapFieldType(field.Type), } if p.EndpointsByName[endpoint] == nil { p.EndpointsByName[endpoint] = NewEndpoint(endpoint) } p.EndpointsByName[endpoint].Parameters = append(p.EndpointsByName[endpoint].Parameters, params) p.EndpointsByName[endpoint].ParameterByName[params.Field] = params } }

Slide 64

Slide 64 text

Best Practices - Include a DO NOT EDIT line in your templates for your colleagues to see - Intended to be used by the author of the Go package, not its clients - go:generate created code should be committed to the repository - Generate your unit tests, too - Make your code generation tool robust

Slide 65

Slide 65 text

A meta tool to build your own tools - go:generate is a meta tool that you can use as a building block to build your own tools: fitted for your purpose - Identity repetitive tasks in your development cycle and try to extract the essence into tooling - Make the tool part of your existing automation as well, i.e. run it in your CI/CD pipeline - Beware of breaking changes, it’s another tool in your stack, so it may require maintenance