$30 off During Our Annual Pro Sale. View Details »

go generate: one file to rule them all

go generate: one file to rule them all

Code generation is ultimately the fifth stage of grief over Go's lack of generics. We've come to accept that Go doesn't have generics (yet!) but we still want to avoid the tedious task of writing duplicate code. With go generate we have a powerful tool to avoid writing boilerplate code. Using the Abstract Syntax Tree (ast) package we can extract sufficient information from an API file to generate middlewares (logging, instrumentation, etc.) and even documentation.

Konrad Reiche

April 13, 2019
Tweet

More Decks by Konrad Reiche

Other Decks in Technology

Transcript

  1. go:generate
    One File To Rule Them All

    View Slide

  2. October is a social
    network designed
    for the attention
    economy
    - Post links, images and text
    - Interact as yourself or
    anonymously
    - Earn coins for good content
    - Reward other creators for
    their content by tipping

    View Slide

  3. Go doesn’t have generics

    View Slide

  4. Denial
    Avoidance
    Confusion
    Elation
    Shock
    Fear
    Anger
    Frustration
    Irritation
    Anxiety Bargaining
    Struggling to find meaning
    Reaching out to others
    Telling one’s story
    Depression
    Overwhelmed
    Helplessness
    Hostility
    Flight
    Acceptance
    Exploring options
    New plan in place
    Moving on
    Kübler-Ross Model
    Five Stages of Grief

    View Slide

  5. Denial
    Go doesn’t lack generics.
    I can just use interface{}, right?

    View Slide

  6. Denial
    Go doesn’t lack generics.
    I can just use interface{}, right?

    View Slide

  7. Denial
    Go doesn’t lack generics.
    I can just use interface{}, right?

    View Slide

  8. Anger
    That’s not fair. Why me?
    Why did I have to fall in love with
    a language that doesn’t have generics?

    View Slide

  9. Bargaining
    I don't need generics. Sometimes code duplication
    is better than unnecessary abstractions.
    Why write 100 lines of code just to avoid
    copy & pasting 10 lines of code once, or twice, or…?

    View Slide

  10. Depression
    In this state, the individual may become silent.
    Refuses code reviews and spends much of the time
    mournful and sullen at his desk.

    View Slide

  11. Acceptance
    Go doesn't have generics, but I can code my own
    parametric polymorphism by writing the tools to
    generate code I need automatically.

    View Slide

  12. Use Case
    REST API

    View Slide

  13. Project Structure
    ├── api
    │ ├── api.go
    ├── client
    │ ├── client.go
    │ └── endpoints.go
    ├── main.go
    ├── metrics
    ├── model
    ├── server
    │ ├── endpoints.go
    │ ├── middleware.go
    │ ├── server.go
    ├── store
    │ └── store.go
    └── vendor
    ├── github.com
    ├── golang.org
    └── modules.txt

    View Slide

  14. const (
    LoginEndpoint = "/login"
    CreatePostEndpoint = "/post"
    GetFeedEndpoint = "/feed"
    )
    type API interface {
    Login(ctx context.Context, req LoginRequest) (*LoginResponse, error)
    CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error)
    GetFeed(ctx context.Context, req GetFeedRequest) (*GetFeedResponse, error)
    }
    api.go

    View Slide

  15. type CreatePostRequest struct {
    Body string `json:"body"`
    URL string `json:"url"`
    }
    type GetPriceResponse struct {
    ID int `json:"id"`
    }
    func (a *api) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) {
    // ...
    }
    api.go

    View Slide

  16. View Slide

  17. API Middlewares
    API Middlewares allow to wrap endpoints for executing code before
    and/or after the endpoint has been processed.
    API

    View Slide

  18. API Middlewares
    API Middlewares allow to wrap endpoints for executing code before
    and/or after the endpoint has been processed.
    API
    Logging Log request, response and metadata

    View Slide

  19. API Middlewares
    API Middlewares allow to wrap endpoints for executing code before
    and/or after the endpoint has been processed.
    API
    Logging
    Instrumentation
    Log request, response and metadata
    Measure latency, count requests, etc.

    View Slide

  20. API Middlewares
    API Middlewares allow to wrap endpoints for executing code before
    and/or after the endpoint has been processed.
    API
    Logging
    Instrumentation
    Tracing
    Log request, response and metadata
    Measure latency, count requests, etc.
    Monitor code execution across services

    View Slide

  21. API Middlewares
    API Middlewares allow to wrap endpoints for executing code before
    and/or after the endpoint has been processed.
    API
    Logging
    Instrumentation
    Tracing
    Activity Recording
    Log request, response and metadata
    Measure latency, count requests, etc.
    Monitor code execution across services
    Record user activity for analytics

    View Slide

  22. type instrumentingMiddleware struct {
    requestCount metrics.Counter
    requestLatency metrics.TimeHistogram
    errorCount metrics.Counter
    log *logrus.Logger
    a API
    }
    instrumenting.go

    View Slide

  23. func NewInstrumentingMiddleware(a API, client *statsd.Client, log *logrus.Logger) API {
    requestCount := dogstatsd.NewCounter("request_count", 1, client, log)
    requestLatency := dogstatsd.NewTimeHistogram("request_latency", 1, client, log)
    errorCount := dogstatsd.NewCounter("error_count", 1, client, log)
    return &instrumentingMiddleware{
    requestCount: requestCount,
    requestLatency: requestLatency,
    errorCount: errorCount,
    l: log,
    a: a,
    }
    }
    instrumenting.go

    View Slide

  24. func (im *instrumentingMiddleware) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) {
    var err error
    methodTag := metrics.Tag{Key: "method", Value: "CreatePost"}
    defer func(begin time.Time) {
    if err != nil {
    errorTag := metrics.Tag{Key: "error", Value: err.Error()}
    im.errorCount.With(methodTag).With(errorTag).Add(1)
    }
    im.requestCount.With(methodTag).Add(1)
    im.requestLatency.With(methodTag).Observe(time.Since(begin))
    }(time.Now())
    response, err := im.a.CreatePost(ctx, req)
    return response, err
    }
    instrumenting.go

    View Slide

  25. func main() {
    logger := log.NewLogger()
    client, err := statsd.New("127.0.0.1:8125", nil, nil)
    if err != nil {
    panic(err)
    }
    api := NewAPI()
    api = NewInstrumentingMiddleware(api, client, logger)
    server.Serve(api)
    }
    main.go

    View Slide

  26. func main() {
    // ...
    api := NewAPI()
    api = NewLoggingMiddleware(api, …)
    api = NewInstrumentingMiddleware(api, …)
    api = NewTracinggMiddleware(api, …)
    api = NewRecordingMiddleware(api, …)
    server.Serve(api)
    }
    All the middlewares

    View Slide

  27. func main() {
    // ...
    api := NewAPI()
    api = NewLoggingMiddleware(api, …)
    api = NewInstrumentingMiddleware(api, …)
    api = NewTracinggMiddleware(api, …)
    api = NewRecordingMiddleware(api, …)
    server.Serve(api)
    }
    All the middlewares

    View Slide

  28. It doesn’t stop here.

    View Slide

  29. type Client struct {
    conn *http.Client
    endpoint string
    }
    func NewClient(endpoint string) api.API {
    client := http.DefaultClient
    return &Client{
    conn: client,
    endpoint: endpoint,
    }
    }
    client.go

    View Slide

  30. func (c *Client) CreatePost(ctx context.Context, req api.CreatePostRequest) (*api.CreatePostResponse, error) {
    r, err := http.NewRequest(http.MethodGet, CreatePostEndpoint), nil)
    if err != nil {
    return nil, err
    }
    resp, err := c.conn.Do(r)
    if err != nil {
    return nil, err
    }
    if resp.StatusCode != http.StatusOK {
    return nil, errors.New(resp.Status)
    }
    // ...
    client.go

    View Slide

  31. // ...
    var result api.CreatePostResponse
    decoder := json.NewDecoder(resp.Body)
    err = decoder.Decode(&result)
    if err != nil {
    return nil, err
    }
    return &result, err
    }
    client.go

    View Slide

  32. func (s *Server) CreatePostHandleFunc(w http.ResponseWriter, r *http.Request) {
    var req api.CreatePostRequest
    params := mux.Vars(r)
    dec, err := mapstructure.NewDecoder(&mapstructure.DecoderConfig{
    TagName: "json",
    Result: &req,
    })
    if err != nil {
    Encode(nil, w, nil, err)
    return
    }
    err = dec.Decode(params)
    if err != nil {
    Encode(nil, w, nil, err)
    }
    // ...
    server.go

    View Slide

  33. // ...
    err = req.Validate()
    if err != nil {
    Encode(nil, w, nil, err)
    return
    }
    resp, err := s.api.CreatePost(r.Context(), req)
    if err != nil {
    Encode(nil, w, nil, err)
    return
    }
    Encode(nil, w, resp, nil)
    }
    server.go

    View Slide

  34. Requirements Under Fire
    API keeps changing:
    - Request parameter & response fields change
    - New features result in new endpoints
    - Deprecated features result in endpoints getting deleted
    What that means for our project:
    - Every changes requires to update all files related to api.go
    - 4 Middlewares + Server Encoder/Decoder + Client Library = 6 Files

    View Slide

  35. View Slide

  36. 6

    View Slide

  37. 6 Files?

    View Slide

  38. 6 Files?

    View Slide

  39. There is always another
    layer of abstraction you
    can add.

    View Slide

  40. There is always another
    layer of abstraction you
    can add.

    View Slide

  41. Code Generation
    The Antidote

    View Slide

  42. Code Generation
    - Work around the limitations of a language, in the case of Go: generics
    - Delegate task to write boilerplate code to a machine
    - It’s mentally more satisfying to do meta-programming
    - Less prone to errors
    Model Tool
    Templates
    Code

    View Slide

  43. go generate
    The go binary comes with a subcommand called generate which scans
    packages for generate directives and for each directive, the specified
    generator is run; if files are named, they must be Go source files and
    generation happens only for directives in those files, i.e. you would put this
    line at the top of your file:
    //go:generate echo ‘Hello Gophercon Russia’
    $ go generate api.go
    > Hello Gophercon Russia

    View Slide

  44. package main
    import (
    "fmt"
    "os"
    )
    func main() {
    fmt.Println(os.Getenv("GOFILE"))
    }
    cmd/apigen.go

    View Slide

  45. package main
    import (
    "fmt"
    "os"
    )
    func main() {
    fmt.Println(os.Getenv("GOFILE"))
    }
    cmd/apigen.go
    //go:generate go run ../cmd/apigen.go
    $ go generate api/api.go

    View Slide

  46. package main
    import (
    "fmt"
    "os"
    )
    func main() {
    fmt.Println(os.Getenv("GOFILE"))
    }
    cmd/apigen.go
    //go:generate go run ../cmd/apigen.go
    $ go generate api/api.go
    > ../api/api.go

    View Slide

  47. Let’s automate.

    View Slide

  48. func (im *instrumentingMiddleware) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) {
    var err error
    methodTag := metrics.Tag{Key: "method", Value: "CreatePost"}
    defer func(begin time.Time) {
    if err != nil {
    errorTag := metrics.Tag{Key: "error", Value: err.Error()}
    im.errorCount.With(methodTag).With(errorTag).Add(1)
    }
    im.requestCount.With(methodTag).Add(1)
    im.requestLatency.With(methodTag).Observe(time.Since(begin))
    }(time.Now())
    response, err := im.a.CreatePost(ctx, req)
    return response, err
    }
    instrumenting.go

    View Slide

  49. func (im *instrumentingMiddleware) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) {
    var err error
    methodTag := metrics.Tag{Key: "method", Value: "CreatePost"}
    defer func(begin time.Time) {
    if err != nil {
    errorTag := metrics.Tag{Key: "error", Value: err.Error()}
    im.errorCount.With(methodTag).With(errorTag).Add(1)
    }
    im.requestCount.With(methodTag).Add(1)
    im.requestLatency.With(methodTag).Observe(time.Since(begin))
    }(time.Now())
    response, err := im.a.CreatePost(ctx, req)
    return response, err
    }
    instrumenting.go

    View Slide

  50. func (im *instrumentingMiddleware) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) {
    var err error
    methodTag := metrics.Tag{Key: "method", Value: "CreatePost"}
    defer func(begin time.Time) {
    if err != nil {
    errorTag := metrics.Tag{Key: "error", Value: err.Error()}
    im.errorCount.With(methodTag).With(errorTag).Add(1)
    }
    im.requestCount.With(methodTag).Add(1)
    im.requestLatency.With(methodTag).Observe(time.Since(begin))
    }(time.Now())
    response, err := im.a.CreatePost(ctx, req)
    return response, err
    }
    instrumenting.go

    View Slide

  51. Generate Architecture
    go:generate
    Directive
    $ go generate ./...
    .
    Tool
    Invokes tool with file
    name as environment
    variable
    Templates
    Parse file(s)
    containing the
    directive
    Boilerplate Code
    Pass model
    to templates
    api/logging.go
    api/instrumentation.go
    api/recording.go
    api/tracing.go
    client/endpoints.go
    server/endpoints.go

    View Slide

  52. AST Package
    Package declares the types used to represent syntax tree for Go packages.
    Abstract Syntax Tree
    A tree representation of the abstract syntactic structure of source code
    written in a programming language. Each node of the tree denotes a construct
    occurring in the source code.
    - parser.ParseFile to create the AST by parsing an actual Go source file
    - ast.Inspect allows to iterate every node and extract further data

    View Slide

  53. type CreatePostRequest struct {
    Body string `json:"body"`
    URL string `json:"url"`
    }
    type GetPriceResponse struct {
    ID int `json:"id"`
    }
    func (a *api) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) {
    // ...
    }
    api.go

    View Slide

  54. func (p *Parser) Parse() error {
    fset := token.NewFileSet()
    f, err := parser.ParseFile(fset, os.Getenv("GOFILE"), nil, 0)
    if err != nil {
    return nil, err
    }
    ast.Inspect(file, func(n ast.Node) bool {
    switch x := n.(type) {
    case *ast.TypeSpec:
    if err := p.parseType(x); err != nil {
    return false
    }
    return true
    })
    return p.generate()
    }

    View Slide

  55. func (p *Parser) parseType(st *ast.TypeSpec) error {
    if strings.HasSuffix(st.Name.Name, "Request") {
    endpoint := strings.Replace(st.Name.Name, "Request", "", -1)
    if p.EndpointsByName[endpoint] == nil {
    p.EndpointsByName[endpoint] = NewEndpoint(endpoint)
    }
    }
    return nil
    }

    View Slide

  56. // Code generated by apigen; DO NOT EDIT.
    //
    // Source: api/api.go
    // Template: templates/instrumentation.go.tmpl
    package api
    // ...
    {{range .Endpoints}}
    func (im *instrumentingMiddleware) {{.Name}}(ctx context.Context, req {{.Name}}Request) (*{{.Name}}Response, error) {
    var err error
    methodTag := metrics.Tag{Key: "method", Value: "{{.Name}}"}
    defer func(begin time.Time) {
    if err != nil {
    errorTag := metrics.Tag{Key: "error", Value: err.Error()}
    im.errorCount.With(methodTag).With(errorTag).Add(1)
    }
    im.requestCount.With(methodTag).Add(1)
    im.requestLatency.With(methodTag).Observe(time.Since(begin))
    }(time.Now())
    response, err := im.a.{{.Name}}(ctx, req)
    return response, err
    }
    {{end}}

    View Slide

  57. instrumentationTemplate, err := ioutil.ReadFile("../templates/instrumentation.go.tmpl")
    if err != nil {
    return nil, err
    }
    tmpl := template.Must(template.New("instrumentation").Parse(string(instrumentationTemplate)))
    generateCode(tmpl, "instrumentation.go")
    cmd/apigen.go

    View Slide

  58. func (p *Parser) generateCode(tmpl *template.Template, fn string) error {
    buf := bytes.NewBuffer([]byte{})
    err := tmpl.Execute(buf, p)
    if err != nil {
    return err
    }
    res, err := imports.Process(fn, buf.Bytes(), nil)
    if err != nil {
    return err
    }
    return ioutil.WriteFile(fn, res, 0666)
    }
    cmd/apigen.go

    View Slide

  59. Demo

    View Slide

  60. func (p *Parser) Parse() error {
    fset := token.NewFileSet()
    f, err := parser.ParseFile(fset, os.Getenv("GOFILE"), nil, 0)
    if err != nil {
    return nil, err
    }
    ast.Inspect(file, func(n ast.Node) bool {
    switch x := n.(type) {
    case *ast.TypeSpec:
    if err := p.parseType(x); err != nil {
    return false
    }
    return true
    })
    return p.generate()
    }

    View Slide

  61. func (p *Parser) Parse() error {
    fset := token.NewFileSet()
    f, err := parser.ParseFile(fset, os.Getenv("GOFILE"), nil, parser.ParseComments)
    if err != nil {
    return nil, err
    }
    ast.Inspect(file, func(n ast.Node) bool {
    switch x := n.(type) {
    case *ast.TypeSpec:
    if err := p.parseType(x); err != nil {
    return false
    }
    return true
    })
    return p.generate()
    }

    View Slide

  62. func (p *Parser) parseFunction(fd *ast.FuncDecl) {
    if fd.Recv == nil {
    return
    }
    if recv, ok := fd.Recv.List[0].Type.(*ast.StarExpr); ok {
    if ident, ok := recv.X.(*ast.Ident); ok {
    name := fd.Name.Name
    description := fd.Doc.Text()
    firstChar := string(name[0])
    if ident.Name == "api" && firstChar == strings.ToUpper(firstChar) {
    p.AddEndpoint(name, description)
    }
    }
    }
    }

    View Slide

  63. func (p *Parser) addParameter(endpoint string, st *ast.StructType) {
    for _, field := range st.Fields.List {
    params := Parameter{
    Field: field.Names[0].Name,
    Description: field.Doc.Text(),
    Tag: parseTag(field.Tag.Value),
    Type: mapFieldType(field.Type),
    }
    if p.EndpointsByName[endpoint] == nil {
    p.EndpointsByName[endpoint] = NewEndpoint(endpoint)
    }
    p.EndpointsByName[endpoint].Parameters = append(p.EndpointsByName[endpoint].Parameters, params)
    p.EndpointsByName[endpoint].ParameterByName[params.Field] = params
    }
    }

    View Slide

  64. Best Practices
    - Include a DO NOT EDIT line in your templates for your colleagues to see
    - Intended to be used by the author of the Go package, not its clients
    - go:generate created code should be committed to the repository
    - Generate your unit tests, too
    - Make your code generation tool robust

    View Slide

  65. A meta tool to build your own tools
    - go:generate is a meta tool that you can use as a building block to build
    your own tools: fitted for your purpose
    - Identity repetitive tasks in your development cycle and try to extract the
    essence into tooling
    - Make the tool part of your existing automation as well, i.e. run it in your
    CI/CD pipeline
    - Beware of breaking changes, it’s another tool in your stack, so it may
    require maintenance

    View Slide

  66. Thank you.
    @konradreiche
    github.com/konradreiche/apigen

    View Slide

  67. Questions?
    @konradreiche
    github.com/konradreiche/apigen

    View Slide