go generate: one file to rule them all

Code generation is ultimately the fifth stage of grief over Go's lack of generics. We've come to accept that Go doesn't have generics (yet!) but we still want to avoid the tedious task of writing duplicate code. With go generate we have a powerful tool to avoid writing boilerplate code. Using the Abstract Syntax Tree (ast) package we can extract sufficient information from an API file to generate middlewares (logging, instrumentation, etc.) and even documentation.

Transcript

1. go:generate
   One File To Rule Them All
   

   View Slide

2. October is a social
   network designed
   for the attention
   economy
   - Post links, images and text
   - Interact as yourself or
   anonymously
   - Earn coins for good content
   - Reward other creators for
   their content by tipping
   

   View Slide

3. Go doesn’t have generics
   

   View Slide

4. Denial
   Avoidance
   Confusion
   Elation
   Shock
   Fear
   Anger
   Frustration
   Irritation
   Anxiety Bargaining
   Struggling to find meaning
   Reaching out to others
   Telling one’s story
   Depression
   Overwhelmed
   Helplessness
   Hostility
   Flight
   Acceptance
   Exploring options
   New plan in place
   Moving on
   Kübler-Ross Model
   Five Stages of Grief
   

   View Slide

5. Denial
   Go doesn’t lack generics.
   I can just use interface{}, right?
   

   View Slide

6. Denial
   Go doesn’t lack generics.
   I can just use interface{}, right?
   

   View Slide

7. Denial
   Go doesn’t lack generics.
   I can just use interface{}, right?
   

   View Slide

8. Anger
   That’s not fair. Why me?
   Why did I have to fall in love with
   a language that doesn’t have generics?
   

   View Slide

9. Bargaining
   I don't need generics. Sometimes code duplication
   is better than unnecessary abstractions.
   Why write 100 lines of code just to avoid
   copy & pasting 10 lines of code once, or twice, or…?
   

   View Slide

10. Depression
    In this state, the individual may become silent.
    Refuses code reviews and spends much of the time
    mournful and sullen at his desk.
    

    View Slide

11. Acceptance
    Go doesn't have generics, but I can code my own
    parametric polymorphism by writing the tools to
    generate code I need automatically.
    

    View Slide

12. Use Case
    REST API
    

    View Slide

13. Project Structure
    ├── api
    │ ├── api.go
    ├── client
    │ ├── client.go
    │ └── endpoints.go
    ├── main.go
    ├── metrics
    ├── model
    ├── server
    │ ├── endpoints.go
    │ ├── middleware.go
    │ ├── server.go
    ├── store
    │ └── store.go
    └── vendor
    ├── github.com
    ├── golang.org
    └── modules.txt
    

    View Slide

14. const (
    LoginEndpoint = "/login"
    CreatePostEndpoint = "/post"
    GetFeedEndpoint = "/feed"
    )
    type API interface {
    Login(ctx context.Context, req LoginRequest) (*LoginResponse, error)
    CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error)
    GetFeed(ctx context.Context, req GetFeedRequest) (*GetFeedResponse, error)
    }
    api.go
    

    View Slide

15. type CreatePostRequest struct {
    Body string `json:"body"`
    URL string `json:"url"`
    }
    type GetPriceResponse struct {
    ID int `json:"id"`
    }
    func (a *api) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) {
    // ...
    }
    api.go
    

    View Slide

16. View Slide

17. API Middlewares
    API Middlewares allow to wrap endpoints for executing code before
    and/or after the endpoint has been processed.
    API
    

    View Slide

18. API Middlewares
    API Middlewares allow to wrap endpoints for executing code before
    and/or after the endpoint has been processed.
    API
    Logging Log request, response and metadata
    

    View Slide

19. API Middlewares
    API Middlewares allow to wrap endpoints for executing code before
    and/or after the endpoint has been processed.
    API
    Logging
    Instrumentation
    Log request, response and metadata
    Measure latency, count requests, etc.
    

    View Slide

20. API Middlewares
    API Middlewares allow to wrap endpoints for executing code before
    and/or after the endpoint has been processed.
    API
    Logging
    Instrumentation
    Tracing
    Log request, response and metadata
    Measure latency, count requests, etc.
    Monitor code execution across services
    

    View Slide

21. API Middlewares
    API Middlewares allow to wrap endpoints for executing code before
    and/or after the endpoint has been processed.
    API
    Logging
    Instrumentation
    Tracing
    Activity Recording
    Log request, response and metadata
    Measure latency, count requests, etc.
    Monitor code execution across services
    Record user activity for analytics
    

    View Slide

22. type instrumentingMiddleware struct {
    requestCount metrics.Counter
    requestLatency metrics.TimeHistogram
    errorCount metrics.Counter
    log *logrus.Logger
    a API
    }
    instrumenting.go
    

    View Slide

23. func NewInstrumentingMiddleware(a API, client *statsd.Client, log *logrus.Logger) API {
    requestCount := dogstatsd.NewCounter("request_count", 1, client, log)
    requestLatency := dogstatsd.NewTimeHistogram("request_latency", 1, client, log)
    errorCount := dogstatsd.NewCounter("error_count", 1, client, log)
    return &instrumentingMiddleware{
    requestCount: requestCount,
    requestLatency: requestLatency,
    errorCount: errorCount,
    l: log,
    a: a,
    }
    }
    instrumenting.go
    

    View Slide

24. func (im *instrumentingMiddleware) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) {
    var err error
    methodTag := metrics.Tag{Key: "method", Value: "CreatePost"}
    defer func(begin time.Time) {
    if err != nil {
    errorTag := metrics.Tag{Key: "error", Value: err.Error()}
    im.errorCount.With(methodTag).With(errorTag).Add(1)
    }
    im.requestCount.With(methodTag).Add(1)
    im.requestLatency.With(methodTag).Observe(time.Since(begin))
    }(time.Now())
    response, err := im.a.CreatePost(ctx, req)
    return response, err
    }
    instrumenting.go
    

    View Slide

25. func main() {
    logger := log.NewLogger()
    client, err := statsd.New("127.0.0.1:8125", nil, nil)
    if err != nil {
    panic(err)
    }
    api := NewAPI()
    api = NewInstrumentingMiddleware(api, client, logger)
    server.Serve(api)
    }
    main.go
    

    View Slide

26. func main() {
    // ...
    api := NewAPI()
    api = NewLoggingMiddleware(api, …)
    api = NewInstrumentingMiddleware(api, …)
    api = NewTracinggMiddleware(api, …)
    api = NewRecordingMiddleware(api, …)
    server.Serve(api)
    }
    All the middlewares
    

    View Slide

27. func main() {
    // ...
    api := NewAPI()
    api = NewLoggingMiddleware(api, …)
    api = NewInstrumentingMiddleware(api, …)
    api = NewTracinggMiddleware(api, …)
    api = NewRecordingMiddleware(api, …)
    server.Serve(api)
    }
    All the middlewares
    

    View Slide

28. It doesn’t stop here.
    

    View Slide

29. type Client struct {
    conn *http.Client
    endpoint string
    }
    func NewClient(endpoint string) api.API {
    client := http.DefaultClient
    return &Client{
    conn: client,
    endpoint: endpoint,
    }
    }
    client.go
    

    View Slide

30. func (c *Client) CreatePost(ctx context.Context, req api.CreatePostRequest) (*api.CreatePostResponse, error) {
    r, err := http.NewRequest(http.MethodGet, CreatePostEndpoint), nil)
    if err != nil {
    return nil, err
    }
    resp, err := c.conn.Do(r)
    if err != nil {
    return nil, err
    }
    if resp.StatusCode != http.StatusOK {
    return nil, errors.New(resp.Status)
    }
    // ...
    client.go
    

    View Slide

31. // ...
    var result api.CreatePostResponse
    decoder := json.NewDecoder(resp.Body)
    err = decoder.Decode(&result)
    if err != nil {
    return nil, err
    }
    return &result, err
    }
    client.go
    

    View Slide

32. func (s *Server) CreatePostHandleFunc(w http.ResponseWriter, r *http.Request) {
    var req api.CreatePostRequest
    params := mux.Vars(r)
    dec, err := mapstructure.NewDecoder(&mapstructure.DecoderConfig{
    TagName: "json",
    Result: &req,
    })
    if err != nil {
    Encode(nil, w, nil, err)
    return
    }
    err = dec.Decode(params)
    if err != nil {
    Encode(nil, w, nil, err)
    }
    // ...
    server.go
    

    View Slide

33. // ...
    err = req.Validate()
    if err != nil {
    Encode(nil, w, nil, err)
    return
    }
    resp, err := s.api.CreatePost(r.Context(), req)
    if err != nil {
    Encode(nil, w, nil, err)
    return
    }
    Encode(nil, w, resp, nil)
    }
    server.go
    

    View Slide

34. Requirements Under Fire
    API keeps changing:
    - Request parameter & response fields change
    - New features result in new endpoints
    - Deprecated features result in endpoints getting deleted
    What that means for our project:
    - Every changes requires to update all files related to api.go
    - 4 Middlewares + Server Encoder/Decoder + Client Library = 6 Files
    

    View Slide

35. View Slide

36. View Slide

37. 6 Files?
    

    View Slide

38. 6 Files?
    

    View Slide

39. There is always another
    layer of abstraction you
    can add.
    

    View Slide

40. There is always another
    layer of abstraction you
    can add.
    

    View Slide

41. Code Generation
    The Antidote
    

    View Slide

42. Code Generation
    - Work around the limitations of a language, in the case of Go: generics
    - Delegate task to write boilerplate code to a machine
    - It’s mentally more satisfying to do meta-programming
    - Less prone to errors
    Model Tool
    Templates
    Code
    

    View Slide

43. go generate
    The go binary comes with a subcommand called generate which scans
    packages for generate directives and for each directive, the specified
    generator is run; if files are named, they must be Go source files and
    generation happens only for directives in those files, i.e. you would put this
    line at the top of your file:
    //go:generate echo ‘Hello Gophercon Russia’
    $ go generate api.go
    > Hello Gophercon Russia
    

    View Slide

44. package main
    import (
    "fmt"
    "os"
    )
    func main() {
    fmt.Println(os.Getenv("GOFILE"))
    }
    cmd/apigen.go
    

    View Slide

45. package main
    import (
    "fmt"
    "os"
    )
    func main() {
    fmt.Println(os.Getenv("GOFILE"))
    }
    cmd/apigen.go
    //go:generate go run ../cmd/apigen.go
    $ go generate api/api.go
    

    View Slide

46. package main
    import (
    "fmt"
    "os"
    )
    func main() {
    fmt.Println(os.Getenv("GOFILE"))
    }
    cmd/apigen.go
    //go:generate go run ../cmd/apigen.go
    $ go generate api/api.go
    > ../api/api.go
    

    View Slide

47. Let’s automate.
    

    View Slide

48. func (im *instrumentingMiddleware) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) {
    var err error
    methodTag := metrics.Tag{Key: "method", Value: "CreatePost"}
    defer func(begin time.Time) {
    if err != nil {
    errorTag := metrics.Tag{Key: "error", Value: err.Error()}
    im.errorCount.With(methodTag).With(errorTag).Add(1)
    }
    im.requestCount.With(methodTag).Add(1)
    im.requestLatency.With(methodTag).Observe(time.Since(begin))
    }(time.Now())
    response, err := im.a.CreatePost(ctx, req)
    return response, err
    }
    instrumenting.go
    

    View Slide

49. func (im *instrumentingMiddleware) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) {
    var err error
    methodTag := metrics.Tag{Key: "method", Value: "CreatePost"}
    defer func(begin time.Time) {
    if err != nil {
    errorTag := metrics.Tag{Key: "error", Value: err.Error()}
    im.errorCount.With(methodTag).With(errorTag).Add(1)
    }
    im.requestCount.With(methodTag).Add(1)
    im.requestLatency.With(methodTag).Observe(time.Since(begin))
    }(time.Now())
    response, err := im.a.CreatePost(ctx, req)
    return response, err
    }
    instrumenting.go
    

    View Slide

50. func (im *instrumentingMiddleware) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) {
    var err error
    methodTag := metrics.Tag{Key: "method", Value: "CreatePost"}
    defer func(begin time.Time) {
    if err != nil {
    errorTag := metrics.Tag{Key: "error", Value: err.Error()}
    im.errorCount.With(methodTag).With(errorTag).Add(1)
    }
    im.requestCount.With(methodTag).Add(1)
    im.requestLatency.With(methodTag).Observe(time.Since(begin))
    }(time.Now())
    response, err := im.a.CreatePost(ctx, req)
    return response, err
    }
    instrumenting.go
    

    View Slide

51. Generate Architecture
    go:generate
    Directive
    $ go generate ./...
    .
    Tool
    Invokes tool with file
    name as environment
    variable
    Templates
    Parse file(s)
    containing the
    directive
    Boilerplate Code
    Pass model
    to templates
    api/logging.go
    api/instrumentation.go
    api/recording.go
    api/tracing.go
    client/endpoints.go
    server/endpoints.go
    

    View Slide

52. AST Package
    Package declares the types used to represent syntax tree for Go packages.
    Abstract Syntax Tree
    A tree representation of the abstract syntactic structure of source code
    written in a programming language. Each node of the tree denotes a construct
    occurring in the source code.
    - parser.ParseFile to create the AST by parsing an actual Go source file
    - ast.Inspect allows to iterate every node and extract further data
    

    View Slide

53. type CreatePostRequest struct {
    Body string `json:"body"`
    URL string `json:"url"`
    }
    type GetPriceResponse struct {
    ID int `json:"id"`
    }
    func (a *api) CreatePost(ctx context.Context, req CreatePostRequest) (*CreatePostResponse, error) {
    // ...
    }
    api.go
    

    View Slide

54. func (p *Parser) Parse() error {
    fset := token.NewFileSet()
    f, err := parser.ParseFile(fset, os.Getenv("GOFILE"), nil, 0)
    if err != nil {
    return nil, err
    }
    ast.Inspect(file, func(n ast.Node) bool {
    switch x := n.(type) {
    case *ast.TypeSpec:
    if err := p.parseType(x); err != nil {
    return false
    }
    return true
    })
    return p.generate()
    }
    

    View Slide

55. func (p *Parser) parseType(st *ast.TypeSpec) error {
    if strings.HasSuffix(st.Name.Name, "Request") {
    endpoint := strings.Replace(st.Name.Name, "Request", "", -1)
    if p.EndpointsByName[endpoint] == nil {
    p.EndpointsByName[endpoint] = NewEndpoint(endpoint)
    }
    }
    return nil
    }
    

    View Slide

56. // Code generated by apigen; DO NOT EDIT.
    //
    // Source: api/api.go
    // Template: templates/instrumentation.go.tmpl
    package api
    // ...
    {{range .Endpoints}}
    func (im *instrumentingMiddleware) {{.Name}}(ctx context.Context, req {{.Name}}Request) (*{{.Name}}Response, error) {
    var err error
    methodTag := metrics.Tag{Key: "method", Value: "{{.Name}}"}
    defer func(begin time.Time) {
    if err != nil {
    errorTag := metrics.Tag{Key: "error", Value: err.Error()}
    im.errorCount.With(methodTag).With(errorTag).Add(1)
    }
    im.requestCount.With(methodTag).Add(1)
    im.requestLatency.With(methodTag).Observe(time.Since(begin))
    }(time.Now())
    response, err := im.a.{{.Name}}(ctx, req)
    return response, err
    }
    {{end}}
    

    View Slide

57. instrumentationTemplate, err := ioutil.ReadFile("../templates/instrumentation.go.tmpl")
    if err != nil {
    return nil, err
    }
    tmpl := template.Must(template.New("instrumentation").Parse(string(instrumentationTemplate)))
    generateCode(tmpl, "instrumentation.go")
    cmd/apigen.go
    

    View Slide

58. func (p *Parser) generateCode(tmpl *template.Template, fn string) error {
    buf := bytes.NewBuffer([]byte{})
    err := tmpl.Execute(buf, p)
    if err != nil {
    return err
    }
    res, err := imports.Process(fn, buf.Bytes(), nil)
    if err != nil {
    return err
    }
    return ioutil.WriteFile(fn, res, 0666)
    }
    cmd/apigen.go
    

    View Slide

59. Demo
    

    View Slide

60. func (p *Parser) Parse() error {
    fset := token.NewFileSet()
    f, err := parser.ParseFile(fset, os.Getenv("GOFILE"), nil, 0)
    if err != nil {
    return nil, err
    }
    ast.Inspect(file, func(n ast.Node) bool {
    switch x := n.(type) {
    case *ast.TypeSpec:
    if err := p.parseType(x); err != nil {
    return false
    }
    return true
    })
    return p.generate()
    }
    

    View Slide

61. func (p *Parser) Parse() error {
    fset := token.NewFileSet()
    f, err := parser.ParseFile(fset, os.Getenv("GOFILE"), nil, parser.ParseComments)
    if err != nil {
    return nil, err
    }
    ast.Inspect(file, func(n ast.Node) bool {
    switch x := n.(type) {
    case *ast.TypeSpec:
    if err := p.parseType(x); err != nil {
    return false
    }
    return true
    })
    return p.generate()
    }
    

    View Slide

62. func (p *Parser) parseFunction(fd *ast.FuncDecl) {
    if fd.Recv == nil {
    return
    }
    if recv, ok := fd.Recv.List[0].Type.(*ast.StarExpr); ok {
    if ident, ok := recv.X.(*ast.Ident); ok {
    name := fd.Name.Name
    description := fd.Doc.Text()
    firstChar := string(name[0])
    if ident.Name == "api" && firstChar == strings.ToUpper(firstChar) {
    p.AddEndpoint(name, description)
    }
    }
    }
    }
    

    View Slide

63. func (p *Parser) addParameter(endpoint string, st *ast.StructType) {
    for _, field := range st.Fields.List {
    params := Parameter{
    Field: field.Names[0].Name,
    Description: field.Doc.Text(),
    Tag: parseTag(field.Tag.Value),
    Type: mapFieldType(field.Type),
    }
    if p.EndpointsByName[endpoint] == nil {
    p.EndpointsByName[endpoint] = NewEndpoint(endpoint)
    }
    p.EndpointsByName[endpoint].Parameters = append(p.EndpointsByName[endpoint].Parameters, params)
    p.EndpointsByName[endpoint].ParameterByName[params.Field] = params
    }
    }
    

    View Slide

64. Best Practices
    - Include a DO NOT EDIT line in your templates for your colleagues to see
    - Intended to be used by the author of the Go package, not its clients
    - go:generate created code should be committed to the repository
    - Generate your unit tests, too
    - Make your code generation tool robust
    

    View Slide

65. A meta tool to build your own tools
    - go:generate is a meta tool that you can use as a building block to build
    your own tools: fitted for your purpose
    - Identity repetitive tasks in your development cycle and try to extract the
    essence into tooling
    - Make the tool part of your existing automation as well, i.e. run it in your
    CI/CD pipeline
    - Beware of breaking changes, it’s another tool in your stack, so it may
    require maintenance
    

    View Slide

66. Thank you.
    @konradreiche
    github.com/konradreiche/apigen
    

    View Slide

67. Questions?
    @konradreiche
    github.com/konradreiche/apigen
    

    View Slide