Inception. Go programs that generate Go code

A review of tools helping reduce boilerplate code in Go using code generation and tips on how they were implemented.

Transcript

1. Inception Go programs that generate Go code
2. I really like Go…
3. it is very simple
4. and very powerful
5. it has great tooling
6. and the stdlib is amazing
7. but…
8. Who likes boilerplate?
9. this talk
10. My experience
13. reducing boilerplate
14. Go programs that: 1. Parse data to generate Go code 2. Modify Go code 3. Parse Go code to generate Go code
15. 1. Parsing data to generate Go code
16. structured data => repetitive code
23. $ schematic platform-api.json > heroku/heroku.go
24. import "text/template"
27. 2. Modifying Go code
28. a big part of Go's brilliant tooling
29. $ go fmt
30. package main import "fmt" func main( ) { fmt. Println ( "Hello world")}
31. package main import "fmt" func main() { fmt.Println("Hello world") }
32. golang.org/x/tools/cmd/goimports
33. package main func main() { fmt.Println("Hello world") }
34. package main import "fmt" func main() { fmt.Println("Hello world") }
35. golang.org/x/tools/cmd/gorename github.com/sqs/goreturns ...
36. Read code > Modify code > Replace code
37. powered by the stdlib
38. Read code > Modify code > Replace code
39. import "go/build" // gathers information about Go packages
40. import "go/scanner" import "go/parser" // read and parse Go source code
41. Read code > Modify code > Replace code
42. import "go/ast" // declares types used to represent the syntax tree
43. import "go/format" // "go fmt" go code, including AST
44. Read code > Modify code > Replace code
45. import "go/printer" // generating code from AST nodes
46. Read code > Generate AST > Modify AST > Replace code
47. 3. Parse Go code to generate Go code
48. your code => boilerplate code
50. Read code > Generate AST > Analyze AST > Generate Code
51. generating mocks
53. package main import ( "testing" "github.com/stretchr/testify/mock" ) type downcaser interface { Downcase(string) (string, error) } func TestMock(t *testing.T) { m := &mockDowncaser{} m.On("Downcase", "FOO").Return("foo", nil) m.Downcase("FOO") m.AssertNumberOfCalls(t, "Downcase", 1) }
54. type mockDowncaser struct { mock.Mock } func (m *mockDowncaser) Downcase(a0 string) (string, error) { ret := m.Called(a0) return ret.Get(0).(string), ret.Error(1) }
55. type downcaser interface { Downcase(string) (string, error) } func (m *mockDowncaser) Downcase(a0 string) (string, error) { ret := m.Called(a0) return ret.Get(0).(string), ret.Error(1) }
57. $ mockery -inpkg -testonly -name=downcaser Generating mock for: downcaser
58. mock_downcaser_test.go package main import "github.com/stretchr/testify/mock" type mockDowncaser struct { mock.Mock } func (_m *mockDowncaser) Downcase(_a0 string) (string, error) { ret := _m.Called(_a0) var r0 string if rf, ok := ret.Get(0).(func(string) string); ok { r0 = rf(_a0) } else { r0 = ret.Get(0).(string) } var r1 error if rf, ok := ret.Get(1).(func(string) error); ok { r1 = rf(_a0) } else { r1 = ret.Error(1) } return r0, r1 }
59. how do you document the steps to generate that code?
60. $ go generate
61. package main import ( "testing" ) type downcaser interface { Downcase(string) (string, error) } //go:generate mockery -inpkg -testonly -name=downcaser func TestMock(t *testing.T) { m := &mockDowncaser{} m.On("Downcase", "FOO").Return("foo", nil) m.Downcase("FOO") m.AssertNumberOfCalls(t, "Downcase", 1) }
62. $ go test # github.com/ernesto-jimenez/test ./main_test.go:14: undefined: mockDowncaser FAIL github.com/ernesto-jimenez/test [build failed] $ go generate Generating mock for: downcaser $ go test PASS ok github.com/ernesto-jimenez/test 0.011s
63. Read code > Generate AST > Analyze AST > Generate Code
65. Yeah, this is a known issue. I'm not sure how deal with it because it requires being able to parse entire packages to find the other interfaces.
66. go/ast provides very raw data
67. go/ast go/types
68. import "go/types" // types and algorithms for type-checking of Go packages
69. Read code > Generate AST > Analyze AST > Generate Code
70. generating any mock
71. github.com/ernesto-jimenez/gogen/cmd/goautomock
72. less code and less issues
73. $ goautomock -o=- io.Reader /* * CODE GENERATED AUTOMATICALLY WITH github.com/ernesto-jimenez/gogen/automock * THIS FILE SHOULD NOT BE EDITED BY HAND */ package app import ( "fmt" mock "github.com/stretchr/testify/mock" ) // ReaderMock mock type ReaderMock struct { mock.Mock } // Read mocked method func (m *ReaderMock) Read(p0 []byte) (int, error) { // […]
75. tens of assertion helpers
76. two functions per assertion func Panics(t TestingT, fn func(), msgAndArgs ...interface{}) bool { if funcDidPanic, _ := didPanic(f); !funcDidPanic { return Fail(t, fmt.Sprintf("func %#v should panic", f), msgAndArgs...) } return true } func (a *Assertions) Panics(fn func(), msgAndArgs ...interface{}) bool { return Panics(a.t, expected, actual, msgAndArgs…) }
78. two extra functions per assertion func Panics(t TestingT, fn func(), msgAndArgs ...interface{}) bool { if !assert.Panics(t, fn, msgAndArgs...) { t.FailNow() } } func (a *Assertions) Panics(fn func(), msgAndArgs ...interface{}) bool { Panics(a.t, fn, msgAndArgs...) }
79. 3 boilerplate functions per each assertion
81. //go:generate go run ../_codegen/main.go -output-package=assert -template=assertion_forward.go.tmpl //go:generate go run ../_codegen/main.go -output-package=require -template=require_forward.go.tmpl //go:generate go run ../_codegen/main.go -output-package=require -template=require.go.tmpl
82. {{.Comment}} func (a *Assertions) {{.DocInfo.Name}}({{.Params}}) bool { return {{.DocInfo.Name}}(a.t, {{.ForwardedParams}}) }
83. go/types go/doc
84. codegen: 239 lines of code removed: 839 lines of code
85. net/http func Get(url string) (resp *Response, err error) { return DefaultClient.Get(url) }
86. var DefaultClient = New() func New() *Client { return &Client{} } func (c *Client) Do(interface{}) error { // […] } // Do calls `DefaultClient.Do` func Do(v interface{}) error { return DefaultClient.Do(v) }
88. var DefaultClient = New() //go:generate goexportdefault DefaultClient func New() *Client { return &Client{} } func (c *Client) Do(interface{}) error { // […] }
89. $ go generate Generating funcs for DefaultClient
90. /* * CODE GENERATED AUTOMATICALLY WITH goexportdefault * THIS FILE MUST NOT BE EDITED BY HAND * * Install goexportdefault with: * go get github.com/ernesto-jimenez/gogen/cmd/goexportdefault */ package example import () // Do is a wrapper around DefaultClient.Do func Do(p0 interface{}) error { return DefaultClient.Do(p0) }
91. avoid goimports on generated code
92. Generating decoders
93. WebSocket > map[string]interface{} > Check m[“type”] > Create struct
95. import “reflect”
96. BenchmarkMapstructure 100.000 17.493 ns/op 2.825 B/op 61 allocs/op
97. if v, ok := m["name"].(string); ok { s.Name = v } else if v, exists := m["name"]; exists && v != nil { return fmt.Errorf("expected field name to be string but got %T", m["name"]) }
99. //go:generate gounmarshalmap -o user_unmarshalmap.go User
100. $ cat user_unmarshalmap.go| wc -l 218
101. user.UnmarshalMap(m)
102. BenchmarkMapstructure 100.000 17.493 ns/op 2.825 B/op 61 allocs/op BenchmarkUnmarshalmap 5.000.000 320 ns/op 0 B/op 0 allocs/op
103. What if we add a new field to the struct?
104. Read code > Analyze code > Generate code and tests
105. $ go test ./unmarshalmap/testpkg --- FAIL: TestNestedUnmarshalMap (0.00s) test.go:35: UnmarshalMap() method from *testpkg.Nested out of date. regenerate the code FAIL FAIL github.com/ernesto-jimenez/gogen/unmarshalmap/testpkg 0.010s $ go generate ./unmarshalmap/testpkg Generating func (*Nested) UnmarshalMap(map[string]interface{}) error $ go test ./unmarshalmap/testpkg ok github.com/ernesto-jimenez/gogen/unmarshalmap/testpkg 0.009s
106. one more thing…
108. 4. Go DSLs (Domain-Specific Languages)
109. http://goa.design
110. $ tree goa-adder . └── design └── design.go 1 directory, 1 file
111. package design import ( . "github.com/goadesign/goa/design" . "github.com/goadesign/goa/design/apidsl" ) var _ = API("adder", func() { Title("The adder API") Description("A teaser for goa") Host("localhost:8080") Scheme("http") }) var _ = Resource("operands", func() { Action("add", func() { Routing(GET("add/:left/:right")) Description("add returns the sum of the left and right parameters in the response body") Params(func() { Param("left", Integer, "Left operand") Param("right", Integer, "Right operand") }) Response(OK, "plain/text") }) })
112. $ goagen bootstrap -d goa-adder/design
113. $ tree goa-adder . ├── app │ ├── contexts.go │ ├── controllers.go │ ├── hrefs.go │ ├── media_types.go │ ├── test │ │ └── operands_testing.go │ └── user_types.go ├── client │ ├── client.go │ ├── media_types.go │ ├── operands.go │ └── user_types.go ├── design │ └── design.go ├── main.go ├── operands.go ├── swagger │ ├── swagger.json │ └── swagger.yaml └── tool ├── adder-cli │ └── main.go └── cli └── commands.go 8 directories, 17 files
114. func main() { // Create service service := goa.New("adder") // Mount middleware service.Use(middleware.RequestID()) service.Use(middleware.LogRequest(true)) service.Use(middleware.ErrorHandler(service, true)) service.Use(middleware.Recover()) // Mount "operands" controller c := NewOperandsController(service) app.MountOperandsController(service, c) // Start service if err := service.ListenAndServe(":8080"); err != nil { service.LogError("startup", "err", err) } }
115. // OperandsController implements the operands resource. type OperandsController struct { *goa.Controller } // Add runs the add action. func (c *OperandsController) Add(ctx *app.AddOperandsContext) error { // OperandsController_Add: start_implement // Put your logic here // OperandsController_Add: end_implement return nil }
116. // Add runs the add action. func (c *OperandsController) Add(ctx *app.AddOperandsContext) error { sum := ctx.Left + ctx.Right return ctx.OK([]byte(strconv.Itoa(sum))) }
117. $ goa-adder [INFO] mount ctrl=Operands action=Add route=GET /add/:left/:right [INFO] listen transport=http addr=:8080 [INFO] started req_id=iaRusv3Joj-1 GET=/add/10/23 from=::1 ctrl=OperandsController action=Add [INFO] params req_id=iaRusv3Joj-1 right=23 left=10 [INFO] completed req_id=iaRusv3Joj-1 status=200 bytes=2 time=316.398µs ------------------------------------------------ $ adder-cli add operands --left 10 --right 23 [INFO] started id=ACowX1Kj GET=http://localhost:8080/add/10/23 [INFO] completed id=ACowX1Kj status=200 time=5.981623ms 33
118. How does goa do it?
119. package design import ( . "github.com/goadesign/goa/design" . "github.com/goadesign/goa/design/apidsl" ) var _ = API("adder", func() { Title("The adder API") Description("A teaser for goa") Host("localhost:8080") Scheme("http") }) var _ = Resource("operands", func() { Action("add", func() { Routing(GET("add/:left/:right")) Description("add returns the sum of the left and right parameters in the response body") Params(func() { Param("left", Integer, "Left operand") Param("right", Integer, "Right operand") }) Response(OK, "plain/text") }) })
120. Magic
122. design code > temp program > generated code
123. package main import ( "fmt" "strings" "github.com/goadesign/goa/dslengine" "github.com/goadesign/goa/goagen/gen_app" _ "github.com/ernesto-jimenez/golang-uk/goa-adder/design" ) func main() { // Check if there were errors while running the first DSL pass dslengine.FailOnError(dslengine.Errors) // Now run the secondary DSLs dslengine.FailOnError(dslengine.Run()) files, err := genapp.Generate() dslengine.FailOnError(err) // We're done fmt.Println(strings.Join(files, "\n")) }
124. Wrapping up
125. you can reduce boilerplate
126. tools already available to reduce boilerplate
127. stdlib has great packages to build your own tools
128. many open source projects to learn from
129. Just remember…
131. Questions? @ernesto_jimenez ernesto@slackline.io