Building a go tool to modify struct tags

Struct field tags are an important part of encode/decode types, especially when using packages such as encoding/json. However, modifying tags is repetitive, cumbersome and open to human errors. We can make it easy to modify tags with an automated tool that is written for this sole purpose.

Transcript

1. Building a go tool to modify struct tags Fatih Arslan Software Engineer @DigitalOcean
2. Why do we need tooling?
3. type Example struct { Foo string }
4. type Example struct { Foo string `json:"foo"` }
5. What about structs with many fields?
6. type Example struct { StatusID int64 Foo string Bar bool Server struct { Address string TLS bool } DiskSize int64 Volumes []string }
7. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo" xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` }
8. Working with struct tags is not fun
9. So, what's a struct tag? type Example struct { Foo string `json:"foo"` }
10. There is no official spec of the struct tag
11. Struct tag definition
12. But fortunately, we have an inofficial one in stdlib!
13. Spec is defined in the reflect package
14. reflect.StructTag
15. Let's decompose the spec
16. json:"foo"
17. json:"foo" key
18. json:"foo" value (quoted)
19. json:"foo" colon (not specified clearly)
20. json:"foo" xml:"foo" space separation
21. type Example struct { Foo string `json:"foo"` } backticks
22. type Example struct { Foo string `json:"foo"` }
23. type Example struct { Foo string "json:\"foo\"" } quotes instead of backticks (works, but not fun to deal with)
24. json:"foo,omitempty" value
25. json:"foo,omitempty" option (not part of the spec)
26. Recap
27. `json:"foo,omitempty" option (not part of the spec) space separation key value xml:"bar"` backtick
28. Back to our initial slide!
29. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo" xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` }
30. It has some errors
31. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo" xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` }
32. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo" xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` } `json:"tls" xml:"tls"`
33. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo" xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` }
34. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo" xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` } `json:"server"`
35. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo" xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` }
36. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo" xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` } `json:"disk_size"`
37. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo" xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` }
38. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo" xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` } `json:"volumes"`
39. First attempt (to automate it)
40. Old implementation • From the cursor search backwards for the first 'struct {' literal • Once found, do a forward search for the right hand brace } • Get the line numbers between the two braces • For each line, get the first identifier, convert it to camel_case word and then append to the same line • A combination of Vim's search() function and regex is being used
41. type Example struct { Foo string Bar |bool } cursor
42. type Example struct { Foo string Bar |bool }
43. type Example struct { Foo string Bar |bool }
44. type Example struct { Foo string Bar |bool }
45. type Example struct { Foo string Bar |bool }
46. type Example struct { Foo string `json:"foo"` Bar |bool `json:"bar"` }
47. Problems
48. No Formatting type Example struct { Bar string `json:"bar"` Foo bool `json:"foo"` } type Example struct { Bar string `json:"bar"` Foo bool `json:"foo"` } want have
49. In-line comments type Example struct { Bar string `json:"bar"` // comment for bar Foo bool `json:"foo"` } type Example struct { Bar string // comment for bar `json:"bar"` Foo bool `json:"foo"` } want have
50. Nested structs type Example struct { Bar bool `json:"bar"` Server struct { `json:"server"` Address string `json:"address"` } } type Example struct { Bar bool `json:"bar"` Server struct { Address string `json:"address"` } `json:"server"` } have want
51. Duplicate tags type Example struct { Bar string `json:"bar"` `xml:"bar"` Foo bool `xml:"foo"` } have type Example struct { Bar string `json:"bar" xml:"bar"` Foo bool `xml:"foo"` } want
52. more quirks • not able to remove tags • not able to add or remove options • field with interface{} types don't work • comment with braces ({ and }) don't work • ...
53. How do we fix it?
54. Second attempt (and also the final one )
55. Two issues that need to be fixed Parse struct Parse struct tag
56. 1. Parse Struct
57. The Go Parser Family
58. go/token go/scanner go/parser go/ast go/printer
59. First, a quick tutorial about go/ast
60. 3 + 2 *ast.BinaryExpr
61. 3 + 2 *ast.BinaryExpr *ast.BasicLit 3
62. 3 + 2 *ast.BinaryExpr *ast.BasicLit 2
63. 3 + 2 *ast.BinaryExpr *ast.BasicLit *ast.BasicLit 3 2 +
64. 3 + (7 - 5) *ast.BinaryExpr *ast.BasicLit 3 + *ast.BinaryExpr *ast.BasicLit *ast.BasicLit 5 - 7
65. 3 + (7 - 5) *ast.BinaryExpr *ast.BasicLit 3 + *ast.BinaryExpr *ast.BasicLit *ast.BasicLit 5 - 7
66. type Example struct { Foo string `json:"foo"` } *ast.TypeSpec
67. type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType
68. type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType *ast.FieldList
69. type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field
70. type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident Foo
71. type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field *ast.Ident string
72. type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field *ast.BasicLit `json:"foo"` (a.k.a: struct tag)
73. type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident *ast.BasicLit
74. ast.Node
75. All nodes define the starting and ending positions
76. How do we parse a Struct?
77. src := `package main type Example struct { Foo string` + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { fmt.Printf("Field: %s\n", field.Names[0].Name) fmt.Printf("Tag: %s\n", field.Tag.Value) } return false })
78. src := `package main type Example struct { Foo string` + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { fmt.Printf("Field: %s\n", field.Names[0].Name) fmt.Printf("Tag: %s\n", field.Tag.Value) } return false })
79. src := `package main type Example struct { Foo string` + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { fmt.Printf("Field: %s\n", field.Names[0].Name) fmt.Printf("Tag: %s\n", field.Tag.Value) } return false })
80. src := `package main type Example struct { Foo string` + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { fmt.Printf("Field: %s\n", field.Names[0].Name) fmt.Printf("Tag: %s\n", field.Tag.Value) } return false })
81. src := `package main type Example struct { Foo string` + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { fmt.Printf("Field: %s\n", field.Names[0].Name) fmt.Printf("Tag: %s\n", field.Tag.Value) } return false })
82. src := `package main type Example struct { Foo string` + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { fmt.Printf("Field: %s\n", field.Names[0].Name) fmt.Printf("Tag: %s\n", field.Tag.Value) } return false }) Field: Foo Tag: `json:"foo"`
83. 2. Parse Struct Tag
84. How to parse a Struct Tag?
85. Remember this?
86. package main import ( "fmt" "reflect" ) func main() { tag := reflect.StructTag(`json:"foo"`) value := tag.Get("json") fmt.Printf("value: %q\n", value) }
87. package main import ( "fmt" "reflect" ) func main() { tag := reflect.StructTag(`json:"foo"`) value := tag.Get("json") fmt.Printf("value: %q\n", value) } $ go run main.go value: "foo"
88. reflect.StructTag is not perfect ...
89. Issues with reflect.StructTag • can't detect if the tag is malformed (only go vet knows that) • doesn't know the semantics of options (i.e: omitempty) • doesn't return all existing tags • modifying existing tags is not possible
90. reflect cmd/vet
91. Let's improve it with a custom package
92. import "github.com/fatih/structtag"
93. structtag
94. Parse and list all tags tags, err := structtag.Parse(`json:"foo,omitempty" xml:"foo"`) if err != nil { panic(err) } // iterate over all key-value pairs for _, t := range tags.Tags() { fmt.Printf("tag: %+v\n", t) } $ go run main.go tag: json:"foo,omitempty" tag: xml:"foo"
95. Get a single Tag tags, err := structtag.Parse(`json:"foo,omitempty" xml:"foo"`) if err != nil { panic(err) } jsonTag, err := tags.Get("json") if err != nil { panic(err) } fmt.Println(jsonTag) // Output: json:"foo,omitempty" fmt.Println(jsonTag.Key) // Output: json fmt.Println(jsonTag.Name) // Output: foo fmt.Println(jsonTag.Options) // Output: [omitempty]
96. Change existing tag jsonTag, err := tags.Get(`json:"foo,omitempty"`) if err != nil { panic(err) } jsonTag.Name = "bar" jsonTag.Options = nil tags.Set(jsonTag) fmt.Println(tags) json:"bar"
97. Add new tag tags, err := structtag.Parse(`json:"foo,omitempty" xml:"foo"`) hclTag := &structtag.Tag{ Key: "hcl", Name: "gopher", Options: []string{"squash"}, } // add new tag tags.Set(hclTag) fmt.Println(tags) json:"foo,omitempty" xml:"foo" hcl:"gopher,squash"
98. Add/remove options tags, err := structtag.Parse(`json:"foo" xml:"bar,comment"`) if err != nil { panic(err) } tags.AddOptions("json", "omitempty") tags.DeleteOptions("xml", "comment") fmt.Println(tags) // json:"foo,omitempty" xml:"bar"
99. Both issues are fixed now Parse struct (using go/parser) Parse struct tag (using fatih/structtag)
100. Write a CLI tool
101. gomodifytags
102. go get github.com/fatih/gomodifytags
103. func main() { var cfg config node = cfg.parse() start, end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch configuration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result
104. func main() { var cfg config node = cfg.parse() start, end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch configuration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result
105. Fetch configuration settings
106. cfg := &config{ file: *flagFile, line: *flagLine, structName: *flagStruct, offset: *flagOffset, output: *flagOutput, write: *flagWrite, clear: *flagClearTags, clearOption: *flagClearOptions, transform: *flagTransform, sort: *flagSort, override: *flagOverride, } $ gomodifytags --file example.go ... Use flags to set configuration
107. Things we need: 1. What content to process 2. Where and what to output 3. Which struct to modify 4. Which tags to modify
108. Things we need: type config struct { file string modified io.Reader output string write bool offset int structName string line string start, end int remove []string add []string override bool transform string sort bool clear bool addOpts []string removeOpts []string clearOpt bool } 1. What content to process 2. Where and what to output 3. Which struct to modify 4. Which tags to modify
109. Things we need: type config struct { file string modified io.Reader output string write bool offset int structName string line string start, end int remove []string add []string override bool transform string sort bool clear bool addOpts []string removeOpts []string clearOpt bool } 1. What content to process 2. Where and what to output 3. Which struct to modify 4. Which tags to modify
110. Things we need: type config struct { file string modified io.Reader output string write bool offset int structName string line string start, end int remove []string add []string override bool transform string sort bool clear bool addOpts []string removeOpts []string clearOpt bool } 1. What content to process 2. Where and what to output 3. Which struct to modify 4. Which tags to modify
111. Things we need: type config struct { file string modified io.Reader output string write bool offset int structName string line string start, end int remove []string add []string override bool transform string sort bool clear bool addOpts []string removeOpts []string clearOpt bool } 1. What content to process 2. Where and what to output 3. Which struct to modify 4. Which tags to modify
112. package main type Example struct { Foo string } $ gomodifytags -file example.go -struct Example -add-tags json
113. package main type Example struct { Foo string } parse gomodifytags
114. Parse content
115. func main() { var cfg config node = cfg.parse() start, end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch configuration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result
116. parse gomodifytags go/parser package main type Example struct { Foo string }
117. parse gomodifytags go/parser go/ast.Node package main type Example struct { Foo string }
118. gomodifytags parse parse go/parser package main type Example struct { Foo string } *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident tag
119. func (c *config) parse() (ast.Node, error) { c.fset = token.NewFileSet() var contents interface{} if c.modified != nil { archive, err := buildutil.ParseOverlayArchive(c.modified) if err != nil { return nil, fmt.Errorf("failed to parse -modified archive: %v", err) } fc, ok := archive[c.file] if !ok { return nil, fmt.Errorf("couldn't find %s in archive", c.file) } contents = fc } return parser.ParseFile(c.fset, c.file, contents, parser.ParseComments) } Parse content
120. func (c *config) parse() (ast.Node, error) { c.fset = token.NewFileSet() var contents interface{} if c.modified != nil { archive, err := buildutil.ParseOverlayArchive(c.modified) if err != nil { return nil, fmt.Errorf("failed to parse -modified archive: %v", err) } fc, ok := archive[c.file] if !ok { return nil, fmt.Errorf("couldn't find %s in archive", c.file) } contents = fc } return parser.ParseFile(c.fset, c.file, contents, parser.ParseComments) } Parse content (cont.)
121. package main type Example struct { Foo string } parse gomodifytags parse
122. package main type Example struct { Foo string } parse find gomodifytags
123. Find start and end positions
124. func main() { var cfg config node = cfg.parse() start, end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch configuration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result
125. gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident tag this is our file ast.Node
126. parse find gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit which is an AST tree
127. parse find gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit select struct that matches our criteria ...
128. There are many ways to find a struct
129. package main type Example struct { Foo string } type Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
130. package main type Example struct { Foo string } type Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
131. package main type Example struct { Foo string } type Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 struct name: Server
132. package main type Example struct { Foo string } type Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 struct name: Server start line: 8 end line: 10
133. package main type Example struct { Foo string } type Server struct { Name string Port| int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 offset byte: 96 (of 163)
134. package main type Example struct { Foo string } type Server struct { Name string Port| int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 offset byte: 96 (of 163) start line: 8 end line: 10
135. package main type Example struct { Foo string } type Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 start line: 8 end line: 10 .. or specify explicit lines
136. package main type Example struct { Foo string } type Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 start line: 9 end line: 9 .. or specify explicit lines
137. parse find gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit select struct that matches our criteria ...
138. gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit ... and get the start and end lines start line: 8 end line: 10
139. func (c *config) findSelection(node ast.Node) (int, int, error) { if c.line != "" { return c.lineSelection(node) } else if c.offset != 0 { return c.offsetSelection(node) } else if c.structName != "" { return c.structSelection(node) } return 0, 0, errors.New("-line, -offset or -struct is not passed") } Find selection
140. func (c *config) findSelection(node ast.Node) (int, int, error) { if c.line != "" { return c.lineSelection(node) } else if c.offset != 0 { return c.offsetSelection(node) } else if c.structName != "" { return c.structSelection(node) } return 0, 0, errors.New("-line, -offset or -struct is not passed") } Find selection
141. Collecting structs
142. // collectStructs collects and maps structType nodes to their positions func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType, 0) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs }
143. // collectStructs collects and maps structType nodes to their positions func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs } type structType struct { name string node *ast.StructType }
144. // collectStructs collects and maps structType nodes to their positions func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs }
145. // collectStructs collects and maps structType nodes to their positions func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs } type structType struct { name string node *ast.StructType }
146. // collectStructs collects and maps structType nodes to their positions func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs }
147. var encStruct *ast.StructType for _, st := range collectStructs() { if st.name == c.structName { encStruct = st.node } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line } Struct selection
148. var encStruct *ast.StructType for _, st := range collectStructs() { if st.name == c.structName { encStruct = st.node } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line } Struct selection
149. var encStruct *ast.StructType for _, st := range collectStructs() { if st.name == c.structName { encStruct = st.node } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line } Struct selection
150. var encStruct *ast.StructType for _, st := range collectStructs() { if st.name == c.structName { encStruct = st.node } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line } Struct selection
151. var encStruct *ast.StructType for _, st := range collectStructs() { structBegin := c.fset.Position(st.node.Pos()).Offset structEnd := c.fset.Position(st.node.End()).Offset if structBegin <= c.offset && c.offset <= structEnd { encStruct = st.node break } } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line Offset selection
152. var encStruct *ast.StructType for _, st := range collectStructs() { structBegin := c.fset.Position(st.node.Pos()).Offset structEnd := c.fset.Position(st.node.End()).Offset if structBegin <= c.offset && c.offset <= structEnd { encStruct = st.node break } } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line Offset selection
153. var encStruct *ast.StructType for _, st := range collectStructs() { structBegin := c.fset.Position(st.node.Pos()).Offset structEnd := c.fset.Position(st.node.End()).Offset if structBegin <= c.offset && c.offset <= structEnd { encStruct = st.node break } } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line Offset selection
154. var encStruct *ast.StructType for _, st := range collectStructs() { structBegin := c.fset.Position(st.node.Pos()).Offset structEnd := c.fset.Position(st.node.End()).Offset if structBegin <= c.offset && c.offset <= structEnd { encStruct = st.node break } } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line Offset selection
155. var encStruct *ast.StructType for _, st := range collectStructs() { structBegin := c.fset.Position(st.node.Pos()).Offset structEnd := c.fset.Position(st.node.End()).Offset if structBegin <= c.offset && c.offset <= structEnd { encStruct = st.node break } } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line Offset selection
156. package main type Example struct { Foo string } parse find gomodifytags find
157. package main type Example struct { Foo string } parse find modify gomodifytags
158. Rewrite the struct tag
159. func main() { var cfg config node = cfg.parse() start, end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch configuration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result
160. gomodifytags start and end lines we're going to modify start line: 8 end line: 10
161. gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit our parsed AST start line: 8 end line: 10
162. gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit struct we're going to modify start line: 8 end line: 10 *ast.StructType
163. gomodifytags *ast.StructType *ast.FieldList
164. gomodifytags *ast.StructType 6 7 8 9 10 11 12 *ast.Field *ast.Field *ast.Field *ast.Field *ast.Field
165. gomodifytags *ast.StructType *ast.Field []*ast.Ident *ast.Ident *ast.BasicLit
166. type Example struct { Foo string `json:"foo"` } gomodifytags *ast.StructType *ast.Field []*ast.Ident *ast.Ident *ast.BasicLit
167. type Example struct { Foo string `json:"foo" } gomodifytags *ast.StructType *ast.Field Foo string `json:"foo"`
168. type Example struct { Foo string `json:"foo"` } gomodifytags *ast.StructType *ast.Field Foo string `json:"foo"`
169. type Example struct { Foo string `json:"bar" } gomodifytags *ast.StructType *ast.Field Foo string `json:"foo" `json:"bar"` type Example struct { Foo string `json:"foo"` }
170. How do we rewrite a struct tag?
171. src := `package main type Example struct { Foo string` + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { // found field! field.Tag.Value = `json:"bar"` } return false }) After finding a field, replace the value
172. src := `package main type Example struct { Foo string` + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { // found field! field.Tag.Value = `json:"bar"` } return false }) Tag: `json:"foo"` Tag: `json:"bar"` After finding a field, replace the value
173. type Example struct { DiskSize string } gomodifytags *ast.StructType *ast.Field DiskSize string
174. type Example struct { DiskSize string } gomodifytags *ast.StructType *ast.Field DiskSize string
175. gomodifytags *ast.StructType *ast.Field DiskSize string json:"disk_size" process() type Example struct { DiskSize string } type Example struct { DiskSize string `json:"disk_size"` }
176. src := `package main type Example struct { Foo string` + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { // found field! field.Tag.Value = process(field) } return false }) tags = c.removeTags(tags) tags, err = c.removeTagOptions(tags) if err != nil { return "", err } tags = c.clearTags(tags) tags = c.clearOptions(tags) tags, err = c.addTags(fieldName, tags) if err != nil { return "", err } tags, err = c.addTagOptions(tags) if err != nil { return "", err } if c.sort { sort.Sort(tags) }
177. gomodifytags *ast.StructType 8 9 10 *ast.Field *ast.Field *ast.Field *ast.Field *ast.Field select field between start and end lines ... Modify overview
178. gomodifytags *ast.StructType 8 9 10 *ast.Field *ast.Field *ast.Field *ast.Field *ast.Field ... and rewrite selected fields tags 8 9 10 *ast.Field *ast.Field *ast.Field *ast.Field *ast.Field rewrite Modify overview
179. package main type Example struct { Foo string } parse find modify gomodifytags modify
180. func (c *config) rewriteFields(node ast.Node) (ast.Node, error) { var rewriteErr error rewriteFunc := func(n ast.Node) bool { x, ok := n.(*ast.StructType) if !ok { return true } for _, f := range x.Fields.List { // process each field // ... } return true } ast.Inspect(node, rewriteFunc) return node, rewriteErr }
181. func (c *config) rewriteFields(node ast.Node) (ast.Node, error) { var rewriteErr error rewriteFunc := func(n ast.Node) bool { x, ok := n.(*ast.StructType) if !ok { return true } for _, f := range x.Fields.List { // process each field // ... } return true } ast.Inspect(node, rewriteFunc) return node, rewriteErr }
182. func (c *config) rewriteFields(node ast.Node) (ast.Node, error) { var rewriteErr error rewriteFunc := func(n ast.Node) bool { x, ok := n.(*ast.StructType) if !ok { return true } for _, f := range x.Fields.List { // process each field // ... } return true } ast.Inspect(node, rewriteFunc) return node, rewriteErr }
183. for _, f := range x.Fields.List { line := c.fset.Position(f.Pos()).Line if !(c.start <= line && line <= c.end) { continue } if f.Tag == nil { f.Tag = &ast.BasicLit{} } // ... }
184. for _, f := range x.Fields.List { line := c.fset.Position(f.Pos()).Line if !(c.start <= line && line <= c.end) { continue } if f.Tag == nil { f.Tag = &ast.BasicLit{} } // ... }
185. for _, f := range x.Fields.List { line := c.fset.Position(f.Pos()).Line if !(c.start <= line && line <= c.end) { continue } if f.Tag == nil { f.Tag = &ast.BasicLit{} } // ... }
186. for _, f := range x.Fields.List { fieldName := "" if len(f.Names) != 0 { fieldName = f.Names[0].Name } if f.Names == nil { ident, ok := f.Type.(*ast.Ident) if !ok { continue // anonymous field } fieldName = ident.Name } res, err := c.process(fieldName, f.Tag.Value) if err != nil { rewriteErr = err return true } f.Tag.Value = res }
187. for _, f := range x.Fields.List { fieldName := "" if len(f.Names) != 0 { fieldName = f.Names[0].Name } if f.Names == nil { ident, ok := f.Type.(*ast.Ident) if !ok { continue // anonymous field } fieldName = ident.Name } res, err := c.process(fieldName, f.Tag.Value) if err != nil { rewriteErr = err return true } f.Tag.Value = res }
188. for _, f := range x.Fields.List { fieldName := "" if len(f.Names) != 0 { fieldName = f.Names[0].Name } if f.Names == nil { ident, ok := f.Type.(*ast.Ident) if !ok { continue // anonymous field } fieldName = ident.Name } res, err := c.process(fieldName, f.Tag.Value) if err != nil { rewriteErr = err return true } f.Tag.Value = res }
189. Processing the tags
190. func (c *config) process(fieldName, tagVal string) (string, error) { var tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil }
191. func (c *config) process(fieldName, tagVal string) (string, error) { var tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil }
192. func (c *config) process(fieldName, tagVal string) (string, error) { var tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil }
193. func (c *config) process(fieldName, tagVal string) (string, error) { var tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil } tags = c.removeTags(tags) tags, err = c.removeTagOptions(tags) if err != nil { return "", err } tags = c.clearTags(tags) tags = c.clearOptions(tags) tags, err = c.addTags(fieldName, tags) if err != nil { return "", err } tags, err = c.addTagOptions(tags) if err != nil { return "", err } if c.sort { sort.Sort(tags) }
194. func (c *config) process(fieldName, tagVal string) (string, error) { var tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil } tags = c.removeTags(tags) tags, err = c.removeTagOptions(tags) if err != nil { return "", err } tags = c.clearTags(tags) tags = c.clearOptions(tags) tags, err = c.addTags(fieldName, tags) if err != nil { return "", err } tags, err = c.addTagOptions(tags) if err != nil { return "", err } if c.sort { sort.Sort(tags) }
195. func (c *config) process(fieldName, tagVal string) (string, error) { var tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil }
196. func (c *config) rewriteFields(node ast.Node) (ast.Node, error) { var rewriteErr error rewriteFunc := func(n ast.Node) bool { x, ok := n.(*ast.StructType) if !ok { return true } for _, f := range x.Fields.List { // process each field // ... } return true } ast.Inspect(node, rewriteFunc) return node, rewriteErr }
197. package main type Example struct { Foo string } parse find modify format gomodifytags
198. Output the result
199. func main() { var cfg config node = cfg.parse() start, end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch configuration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result
200. gomodifytags ast.Node (modified)
201. gomodifytags ast.Node (modified) go/format Pass the modified node to go/format
202. gomodifytags ast.Node (modified) go/format ... and output the result to stdout package main type Example struct { Foo string `json:"foo"` }
203. package main type Server struct { Name string Port int EnableLogs bool BaseDomain string Credentials struct { Username string Password string } } package main type Server struct { Name string `json:"name"` Port int `json:"port"` EnableLogs bool `json:"enable_logs"` BaseDomain string `json:"base_domain"` Credentials struct { Username string `json:"username"` Password string `json:"password"` } `json:"credentials"` } Input Output $ gomodifytags -file example.go -struct Server -add-tags json
204. gomodifytags ast.Node (modified) go/format Also has support for custom JSON output { "start": 3, "end": 5, "lines": [ "type Example struct {", " Foo string `json:\"foo\"`", "}" ] }
205. package main type Server struct { Name string Port int EnableLogs bool BaseDomain string Credentials struct { Username string Password string } } { "start": 3, "end": 12, "lines": [ "type Server struct {", " Name string `xml:\"name\"`", " Port int `xml:\"port\"`", " EnableLogs bool `xml:\"enable_logs\"`", " BaseDomain string `xml:\"base_domain\"`", " Credentials struct {", " Username string `xml:\"username\"`", " Password string `xml:\"password\"`", " } `xml:\"credentials\"`", "}" ] } Input JSON Output $ gomodifytags -file example.go -struct Server -add-tags json -format json
206. Why stdout? • by default, it means "dry-run" • immediate feedback • tools can redirect the output • composable (gomodifytags myfile.go > newfile.go)
207. func (c *config) format(file ast.Node) (string, error) { switch c.output { case "source": // return formatted source case "json": // return only changes in json default: return "", fmt.Errorf("unknown output mode: %s", c.output) } } Output the result
208. func (c *config) format(file ast.Node) (string, error) { switch c.output { case "source": // return formatted source case "json": // return only changes in json default: return "", fmt.Errorf("unknown output mode: %s", c.output) } } Output the result
209. var buf bytes.Buffer err := format.Node(&buf, c.fset, file) if err != nil { return "", err } if c.write { err = ioutil.WriteFile(c.file, buf.Bytes(), 0) if err != nil { return "", err } } return buf.String(), nil Source file
210. var buf bytes.Buffer err := format.Node(&buf, c.fset, file) if err != nil { return "", err } if c.write { err = ioutil.WriteFile(c.file, buf.Bytes(), 0) if err != nil { return "", err } } return buf.String(), nil Source file
211. var buf bytes.Buffer err := format.Node(&buf, c.fset, file) if err != nil { return "", err } if c.write { err = ioutil.WriteFile(c.file, buf.Bytes(), 0) if err != nil { return "", err } } return buf.String(), nil Source file
212. var buf bytes.Buffer err := format.Node(&buf, c.fset, file) if err != nil { return "", err } var lines []string scanner := bufio.NewScanner(bytes.NewBufferString(buf.String())) for scanner.Scan() { lines = append(lines, scanner.Text()) } if c.start > len(lines) { return "", errors.New("line selection is invalid") } ... JSON output
213. var buf bytes.Buffer err := format.Node(&buf, c.fset, file) if err != nil { return "", err } var lines []string scanner := bufio.NewScanner(bytes.NewBufferString(buf.String())) for scanner.Scan() { lines = append(lines, scanner.Text()) } if c.start > len(lines) { return "", errors.New("line selection is invalid") } ... JSON output
214. out := &output{ Start: c.start, End: c.end, Lines: lines[c.start-1 : c.end], } o, err := json.MarshalIndent(out, "", " ") if err != nil { return "", err } return string(o), nil JSON output (cont.)
215. out := &output{ Start: c.start, End: c.end, Lines: lines[c.start-1 : c.end], } o, err := json.MarshalIndent(out, "", " ") if err != nil { return "", err } return string(o), nil JSON output (cont.)
216. package main type Example struct { Foo string } parse find modify gomodifytags format format
217. package main type Example struct { Foo string } parse find modify gomodifytags format Overview
218. Overview
219. gomodifytags
220. gomodifytags package main type Example struct { Foo string }
221. package main type Example struct { Foo string } -file example.go -struct Example -add-tags json gomodifytags
222. package main type Example struct { Foo string } parse gomodifytags
223. package main type Example struct { Foo string } parse find gomodifytags
224. package main type Example struct { Foo string } parse find modify gomodifytags
225. package main type Example struct { Foo string } parse find modify format gomodifytags
226. package main type Example struct { Foo string } package main type Example struct { Foo string `json:"foo"` } parse find modify format gomodifytags
227. Demo time
228. Built from the ground up for editors • selecting structs based on offset or range of lines • write result to stdout or file • json output for easy parsing • support for unsaved buffers • individual cli flags for all features
229. Supported editors • vim with vim-go • atom with go-plus (thanks Zac Bergquist) • vscode with vscode-go (thanks Ramya Rao) • emacs - WIP (https://github.com/syohex/emacs-go-add-tags/issues/ 6)
230. Fixed bugs
231. Recap • A single tool to rule all editors • Easy maintenance • Free of bugs due stdlib tooling (go/parser family) • Happy and productive users
232. Thanks! Fatih Arslan @fatih @fatih fatih@arslan.io