Building a go tool to modify struct tags

Building a go tool to modify struct tags

Struct field tags are an important part of encode/decode types, especially when using packages such as encoding/json. However, modifying tags is repetitive, cumbersome and open to human errors. We can make it easy to modify tags with an automated tool that is written for this sole purpose.

B1019ca5714cf8e9951868d6bc517827?s=128

Fatih Arslan

July 13, 2017
Tweet

Transcript

  1. Building a go tool to modify struct tags Fatih Arslan

    Software Engineer @DigitalOcean
  2. Why do we need tooling?

  3. type Example struct { Foo string }

  4. type Example struct { Foo string `json:"foo"` }

  5. What about structs with many fields?

  6. type Example struct { StatusID int64 Foo string Bar bool

    Server struct { Address string TLS bool } DiskSize int64 Volumes []string }
  7. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"

    xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` }
  8. Working with struct tags is not fun

  9. So, what's a struct tag? type Example struct { Foo

    string `json:"foo"` }
  10. There is no official spec of the struct tag

  11. Struct tag definition

  12. But fortunately, we have an inofficial one in stdlib!

  13. Spec is defined in the reflect package

  14. reflect.StructTag

  15. Let's decompose the spec

  16. json:"foo"

  17. json:"foo" key

  18. json:"foo" value (quoted)

  19. json:"foo" colon (not specified clearly)

  20. json:"foo" xml:"foo" space separation

  21. type Example struct { Foo string `json:"foo"` } backticks

  22. type Example struct { Foo string `json:"foo"` }

  23. type Example struct { Foo string "json:\"foo\"" } quotes instead

    of backticks (works, but not fun to deal with)
  24. json:"foo,omitempty" value

  25. json:"foo,omitempty" option (not part of the spec)

  26. Recap

  27. `json:"foo,omitempty" option (not part of the spec) space separation key

    value xml:"bar"` backtick
  28. Back to our initial slide!

  29. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"

    xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` }
  30. It has some errors

  31. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"

    xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` }
  32. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"

    xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` } `json:"tls" xml:"tls"`
  33. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"

    xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` }
  34. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"

    xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` } `json:"server"`
  35. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"

    xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` }
  36. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"

    xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` } `json:"disk_size"`
  37. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"

    xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` }
  38. type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"

    xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` } `json:"volumes"`
  39. First attempt (to automate it)

  40. Old implementation • From the cursor search backwards for the

    first 'struct {' literal • Once found, do a forward search for the right hand brace } • Get the line numbers between the two braces • For each line, get the first identifier, convert it to camel_case word and then append to the same line • A combination of Vim's search() function and regex is being used
  41. type Example struct { Foo string Bar |bool } cursor

  42. type Example struct { Foo string Bar |bool }

  43. type Example struct { Foo string Bar |bool }

  44. type Example struct { Foo string Bar |bool }

  45. type Example struct { Foo string Bar |bool }

  46. type Example struct { Foo string `json:"foo"` Bar |bool `json:"bar"`

    }
  47. Problems

  48. No Formatting type Example struct { Bar string `json:"bar"` Foo

    bool `json:"foo"` } type Example struct { Bar string `json:"bar"` Foo bool `json:"foo"` } want have
  49. In-line comments type Example struct { Bar string `json:"bar"` //

    comment for bar Foo bool `json:"foo"` } type Example struct { Bar string // comment for bar `json:"bar"` Foo bool `json:"foo"` } want have
  50. Nested structs type Example struct { Bar bool `json:"bar"` Server

    struct { `json:"server"` Address string `json:"address"` } } type Example struct { Bar bool `json:"bar"` Server struct { Address string `json:"address"` } `json:"server"` } have want
  51. Duplicate tags type Example struct { Bar string `json:"bar"` `xml:"bar"`

    Foo bool `xml:"foo"` } have type Example struct { Bar string `json:"bar" xml:"bar"` Foo bool `xml:"foo"` } want
  52. more quirks • not able to remove tags • not

    able to add or remove options • field with interface{} types don't work • comment with braces ({ and }) don't work • ...
  53. How do we fix it?

  54. Second attempt (and also the final one )

  55. Two issues that need to be fixed Parse struct Parse

    struct tag
  56. 1. Parse Struct

  57. The Go Parser Family

  58. go/token go/scanner go/parser go/ast go/printer

  59. First, a quick tutorial about go/ast

  60. 3 + 2 *ast.BinaryExpr

  61. 3 + 2 *ast.BinaryExpr *ast.BasicLit 3

  62. 3 + 2 *ast.BinaryExpr *ast.BasicLit 2

  63. 3 + 2 *ast.BinaryExpr *ast.BasicLit *ast.BasicLit 3 2 +

  64. 3 + (7 - 5) *ast.BinaryExpr *ast.BasicLit 3 + *ast.BinaryExpr

    *ast.BasicLit *ast.BasicLit 5 - 7
  65. 3 + (7 - 5) *ast.BinaryExpr *ast.BasicLit 3 + *ast.BinaryExpr

    *ast.BasicLit *ast.BasicLit 5 - 7
  66. type Example struct { Foo string `json:"foo"` } *ast.TypeSpec

  67. type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType

  68. type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType

    *ast.FieldList
  69. type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType

    *ast.FieldList *ast.Field
  70. type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType

    *ast.FieldList *ast.Field []*ast.Ident Foo
  71. type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType

    *ast.FieldList *ast.Field *ast.Ident string
  72. type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType

    *ast.FieldList *ast.Field *ast.BasicLit `json:"foo"` (a.k.a: struct tag)
  73. type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType

    *ast.FieldList *ast.Field []*ast.Ident *ast.Ident *ast.BasicLit
  74. ast.Node

  75. All nodes define the starting and ending positions

  76. How do we parse a Struct?

  77. src := `package main type Example struct { Foo string`

    + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { fmt.Printf("Field: %s\n", field.Names[0].Name) fmt.Printf("Tag: %s\n", field.Tag.Value) } return false })
  78. src := `package main type Example struct { Foo string`

    + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { fmt.Printf("Field: %s\n", field.Names[0].Name) fmt.Printf("Tag: %s\n", field.Tag.Value) } return false })
  79. src := `package main type Example struct { Foo string`

    + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { fmt.Printf("Field: %s\n", field.Names[0].Name) fmt.Printf("Tag: %s\n", field.Tag.Value) } return false })
  80. src := `package main type Example struct { Foo string`

    + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { fmt.Printf("Field: %s\n", field.Names[0].Name) fmt.Printf("Tag: %s\n", field.Tag.Value) } return false })
  81. src := `package main type Example struct { Foo string`

    + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { fmt.Printf("Field: %s\n", field.Names[0].Name) fmt.Printf("Tag: %s\n", field.Tag.Value) } return false })
  82. src := `package main type Example struct { Foo string`

    + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { fmt.Printf("Field: %s\n", field.Names[0].Name) fmt.Printf("Tag: %s\n", field.Tag.Value) } return false }) Field: Foo Tag: `json:"foo"`
  83. 2. Parse Struct Tag

  84. How to parse a Struct Tag?

  85. Remember this?

  86. package main import ( "fmt" "reflect" ) func main() {

    tag := reflect.StructTag(`json:"foo"`) value := tag.Get("json") fmt.Printf("value: %q\n", value) }
  87. package main import ( "fmt" "reflect" ) func main() {

    tag := reflect.StructTag(`json:"foo"`) value := tag.Get("json") fmt.Printf("value: %q\n", value) } $ go run main.go value: "foo"
  88. reflect.StructTag is not perfect ...

  89. Issues with reflect.StructTag • can't detect if the tag is

    malformed (only go vet knows that) • doesn't know the semantics of options (i.e: omitempty) • doesn't return all existing tags • modifying existing tags is not possible
  90. reflect cmd/vet

  91. Let's improve it with a custom package

  92. import "github.com/fatih/structtag"

  93. structtag

  94. Parse and list all tags tags, err := structtag.Parse(`json:"foo,omitempty" xml:"foo"`)

    if err != nil { panic(err) } // iterate over all key-value pairs for _, t := range tags.Tags() { fmt.Printf("tag: %+v\n", t) } $ go run main.go tag: json:"foo,omitempty" tag: xml:"foo"
  95. Get a single Tag tags, err := structtag.Parse(`json:"foo,omitempty" xml:"foo"`) if

    err != nil { panic(err) } jsonTag, err := tags.Get("json") if err != nil { panic(err) } fmt.Println(jsonTag) // Output: json:"foo,omitempty" fmt.Println(jsonTag.Key) // Output: json fmt.Println(jsonTag.Name) // Output: foo fmt.Println(jsonTag.Options) // Output: [omitempty]
  96. Change existing tag jsonTag, err := tags.Get(`json:"foo,omitempty"`) if err !=

    nil { panic(err) } jsonTag.Name = "bar" jsonTag.Options = nil tags.Set(jsonTag) fmt.Println(tags) json:"bar"
  97. Add new tag tags, err := structtag.Parse(`json:"foo,omitempty" xml:"foo"`) hclTag :=

    &structtag.Tag{ Key: "hcl", Name: "gopher", Options: []string{"squash"}, } // add new tag tags.Set(hclTag) fmt.Println(tags) json:"foo,omitempty" xml:"foo" hcl:"gopher,squash"
  98. Add/remove options tags, err := structtag.Parse(`json:"foo" xml:"bar,comment"`) if err !=

    nil { panic(err) } tags.AddOptions("json", "omitempty") tags.DeleteOptions("xml", "comment") fmt.Println(tags) // json:"foo,omitempty" xml:"bar"
  99. Both issues are fixed now Parse struct (using go/parser) Parse

    struct tag (using fatih/structtag)
  100. Write a CLI tool

  101. gomodifytags

  102. go get github.com/fatih/gomodifytags

  103. func main() { var cfg config node = cfg.parse() start,

    end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch configuration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result
  104. func main() { var cfg config node = cfg.parse() start,

    end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch configuration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result
  105. Fetch configuration settings

  106. cfg := &config{ file: *flagFile, line: *flagLine, structName: *flagStruct, offset:

    *flagOffset, output: *flagOutput, write: *flagWrite, clear: *flagClearTags, clearOption: *flagClearOptions, transform: *flagTransform, sort: *flagSort, override: *flagOverride, } $ gomodifytags --file example.go ... Use flags to set configuration
  107. Things we need: 1. What content to process 2. Where

    and what to output 3. Which struct to modify 4. Which tags to modify
  108. Things we need: type config struct { file string modified

    io.Reader output string write bool offset int structName string line string start, end int remove []string add []string override bool transform string sort bool clear bool addOpts []string removeOpts []string clearOpt bool } 1. What content to process 2. Where and what to output 3. Which struct to modify 4. Which tags to modify
  109. Things we need: type config struct { file string modified

    io.Reader output string write bool offset int structName string line string start, end int remove []string add []string override bool transform string sort bool clear bool addOpts []string removeOpts []string clearOpt bool } 1. What content to process 2. Where and what to output 3. Which struct to modify 4. Which tags to modify
  110. Things we need: type config struct { file string modified

    io.Reader output string write bool offset int structName string line string start, end int remove []string add []string override bool transform string sort bool clear bool addOpts []string removeOpts []string clearOpt bool } 1. What content to process 2. Where and what to output 3. Which struct to modify 4. Which tags to modify
  111. Things we need: type config struct { file string modified

    io.Reader output string write bool offset int structName string line string start, end int remove []string add []string override bool transform string sort bool clear bool addOpts []string removeOpts []string clearOpt bool } 1. What content to process 2. Where and what to output 3. Which struct to modify 4. Which tags to modify
  112. package main type Example struct { Foo string } $

    gomodifytags -file example.go -struct Example -add-tags json
  113. package main type Example struct { Foo string } parse

    gomodifytags
  114. Parse content

  115. func main() { var cfg config node = cfg.parse() start,

    end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch configuration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result
  116. parse gomodifytags go/parser package main type Example struct { Foo

    string }
  117. parse gomodifytags go/parser go/ast.Node package main type Example struct {

    Foo string }
  118. gomodifytags parse parse go/parser package main type Example struct {

    Foo string } *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident tag
  119. func (c *config) parse() (ast.Node, error) { c.fset = token.NewFileSet()

    var contents interface{} if c.modified != nil { archive, err := buildutil.ParseOverlayArchive(c.modified) if err != nil { return nil, fmt.Errorf("failed to parse -modified archive: %v", err) } fc, ok := archive[c.file] if !ok { return nil, fmt.Errorf("couldn't find %s in archive", c.file) } contents = fc } return parser.ParseFile(c.fset, c.file, contents, parser.ParseComments) } Parse content
  120. func (c *config) parse() (ast.Node, error) { c.fset = token.NewFileSet()

    var contents interface{} if c.modified != nil { archive, err := buildutil.ParseOverlayArchive(c.modified) if err != nil { return nil, fmt.Errorf("failed to parse -modified archive: %v", err) } fc, ok := archive[c.file] if !ok { return nil, fmt.Errorf("couldn't find %s in archive", c.file) } contents = fc } return parser.ParseFile(c.fset, c.file, contents, parser.ParseComments) } Parse content (cont.)
  121. package main type Example struct { Foo string } parse

    gomodifytags parse
  122. package main type Example struct { Foo string } parse

    find gomodifytags
  123. Find start and end positions

  124. func main() { var cfg config node = cfg.parse() start,

    end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch configuration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result
  125. gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident tag this is

    our file ast.Node
  126. parse find gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit

    which is an AST tree
  127. parse find gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit

    select struct that matches our criteria ...
  128. There are many ways to find a struct

  129. package main type Example struct { Foo string } type

    Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
  130. package main type Example struct { Foo string } type

    Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
  131. package main type Example struct { Foo string } type

    Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 struct name: Server
  132. package main type Example struct { Foo string } type

    Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 struct name: Server start line: 8 end line: 10
  133. package main type Example struct { Foo string } type

    Server struct { Name string Port| int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 offset byte: 96 (of 163)
  134. package main type Example struct { Foo string } type

    Server struct { Name string Port| int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 offset byte: 96 (of 163) start line: 8 end line: 10
  135. package main type Example struct { Foo string } type

    Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 start line: 8 end line: 10 .. or specify explicit lines
  136. package main type Example struct { Foo string } type

    Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 start line: 9 end line: 9 .. or specify explicit lines
  137. parse find gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit

    select struct that matches our criteria ...
  138. gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit ... and

    get the start and end lines start line: 8 end line: 10
  139. func (c *config) findSelection(node ast.Node) (int, int, error) { if

    c.line != "" { return c.lineSelection(node) } else if c.offset != 0 { return c.offsetSelection(node) } else if c.structName != "" { return c.structSelection(node) } return 0, 0, errors.New("-line, -offset or -struct is not passed") } Find selection
  140. func (c *config) findSelection(node ast.Node) (int, int, error) { if

    c.line != "" { return c.lineSelection(node) } else if c.offset != 0 { return c.offsetSelection(node) } else if c.structName != "" { return c.structSelection(node) } return 0, 0, errors.New("-line, -offset or -struct is not passed") } Find selection
  141. Collecting structs

  142. // collectStructs collects and maps structType nodes to their positions

    func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType, 0) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs }
  143. // collectStructs collects and maps structType nodes to their positions

    func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs } type structType struct { name string node *ast.StructType }
  144. // collectStructs collects and maps structType nodes to their positions

    func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs }
  145. // collectStructs collects and maps structType nodes to their positions

    func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs } type structType struct { name string node *ast.StructType }
  146. // collectStructs collects and maps structType nodes to their positions

    func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs }
  147. var encStruct *ast.StructType for _, st := range collectStructs() {

    if st.name == c.structName { encStruct = st.node } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line } Struct selection
  148. var encStruct *ast.StructType for _, st := range collectStructs() {

    if st.name == c.structName { encStruct = st.node } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line } Struct selection
  149. var encStruct *ast.StructType for _, st := range collectStructs() {

    if st.name == c.structName { encStruct = st.node } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line } Struct selection
  150. var encStruct *ast.StructType for _, st := range collectStructs() {

    if st.name == c.structName { encStruct = st.node } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line } Struct selection
  151. var encStruct *ast.StructType for _, st := range collectStructs() {

    structBegin := c.fset.Position(st.node.Pos()).Offset structEnd := c.fset.Position(st.node.End()).Offset if structBegin <= c.offset && c.offset <= structEnd { encStruct = st.node break } } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line Offset selection
  152. var encStruct *ast.StructType for _, st := range collectStructs() {

    structBegin := c.fset.Position(st.node.Pos()).Offset structEnd := c.fset.Position(st.node.End()).Offset if structBegin <= c.offset && c.offset <= structEnd { encStruct = st.node break } } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line Offset selection
  153. var encStruct *ast.StructType for _, st := range collectStructs() {

    structBegin := c.fset.Position(st.node.Pos()).Offset structEnd := c.fset.Position(st.node.End()).Offset if structBegin <= c.offset && c.offset <= structEnd { encStruct = st.node break } } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line Offset selection
  154. var encStruct *ast.StructType for _, st := range collectStructs() {

    structBegin := c.fset.Position(st.node.Pos()).Offset structEnd := c.fset.Position(st.node.End()).Offset if structBegin <= c.offset && c.offset <= structEnd { encStruct = st.node break } } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line Offset selection
  155. var encStruct *ast.StructType for _, st := range collectStructs() {

    structBegin := c.fset.Position(st.node.Pos()).Offset structEnd := c.fset.Position(st.node.End()).Offset if structBegin <= c.offset && c.offset <= structEnd { encStruct = st.node break } } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line Offset selection
  156. package main type Example struct { Foo string } parse

    find gomodifytags find
  157. package main type Example struct { Foo string } parse

    find modify gomodifytags
  158. Rewrite the struct tag

  159. func main() { var cfg config node = cfg.parse() start,

    end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch configuration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result
  160. gomodifytags start and end lines we're going to modify start

    line: 8 end line: 10
  161. gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit our parsed

    AST start line: 8 end line: 10
  162. gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit struct we're

    going to modify start line: 8 end line: 10 *ast.StructType
  163. gomodifytags *ast.StructType *ast.FieldList

  164. gomodifytags *ast.StructType 6 7 8 9 10 11 12 *ast.Field

    *ast.Field *ast.Field *ast.Field *ast.Field
  165. gomodifytags *ast.StructType *ast.Field []*ast.Ident *ast.Ident *ast.BasicLit

  166. type Example struct { Foo string `json:"foo"` } gomodifytags *ast.StructType

    *ast.Field []*ast.Ident *ast.Ident *ast.BasicLit
  167. type Example struct { Foo string `json:"foo" } gomodifytags *ast.StructType

    *ast.Field Foo string `json:"foo"`
  168. type Example struct { Foo string `json:"foo"` } gomodifytags *ast.StructType

    *ast.Field Foo string `json:"foo"`
  169. type Example struct { Foo string `json:"bar" } gomodifytags *ast.StructType

    *ast.Field Foo string `json:"foo" `json:"bar"` type Example struct { Foo string `json:"foo"` }
  170. How do we rewrite a struct tag?

  171. src := `package main type Example struct { Foo string`

    + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { // found field! field.Tag.Value = `json:"bar"` } return false }) After finding a field, replace the value
  172. src := `package main type Example struct { Foo string`

    + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { // found field! field.Tag.Value = `json:"bar"` } return false }) Tag: `json:"foo"` Tag: `json:"bar"` After finding a field, replace the value
  173. type Example struct { DiskSize string } gomodifytags *ast.StructType *ast.Field

    DiskSize string
  174. type Example struct { DiskSize string } gomodifytags *ast.StructType *ast.Field

    DiskSize string
  175. gomodifytags *ast.StructType *ast.Field DiskSize string json:"disk_size" process() type Example struct

    { DiskSize string } type Example struct { DiskSize string `json:"disk_size"` }
  176. src := `package main type Example struct { Foo string`

    + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { // found field! field.Tag.Value = process(field) } return false }) tags = c.removeTags(tags) tags, err = c.removeTagOptions(tags) if err != nil { return "", err } tags = c.clearTags(tags) tags = c.clearOptions(tags) tags, err = c.addTags(fieldName, tags) if err != nil { return "", err } tags, err = c.addTagOptions(tags) if err != nil { return "", err } if c.sort { sort.Sort(tags) }
  177. gomodifytags *ast.StructType 8 9 10 *ast.Field *ast.Field *ast.Field *ast.Field *ast.Field

    select field between start and end lines ... Modify overview
  178. gomodifytags *ast.StructType 8 9 10 *ast.Field *ast.Field *ast.Field *ast.Field *ast.Field

    ... and rewrite selected fields tags 8 9 10 *ast.Field *ast.Field *ast.Field *ast.Field *ast.Field rewrite Modify overview
  179. package main type Example struct { Foo string } parse

    find modify gomodifytags modify
  180. func (c *config) rewriteFields(node ast.Node) (ast.Node, error) { var rewriteErr

    error rewriteFunc := func(n ast.Node) bool { x, ok := n.(*ast.StructType) if !ok { return true } for _, f := range x.Fields.List { // process each field // ... } return true } ast.Inspect(node, rewriteFunc) return node, rewriteErr }
  181. func (c *config) rewriteFields(node ast.Node) (ast.Node, error) { var rewriteErr

    error rewriteFunc := func(n ast.Node) bool { x, ok := n.(*ast.StructType) if !ok { return true } for _, f := range x.Fields.List { // process each field // ... } return true } ast.Inspect(node, rewriteFunc) return node, rewriteErr }
  182. func (c *config) rewriteFields(node ast.Node) (ast.Node, error) { var rewriteErr

    error rewriteFunc := func(n ast.Node) bool { x, ok := n.(*ast.StructType) if !ok { return true } for _, f := range x.Fields.List { // process each field // ... } return true } ast.Inspect(node, rewriteFunc) return node, rewriteErr }
  183. for _, f := range x.Fields.List { line := c.fset.Position(f.Pos()).Line

    if !(c.start <= line && line <= c.end) { continue } if f.Tag == nil { f.Tag = &ast.BasicLit{} } // ... }
  184. for _, f := range x.Fields.List { line := c.fset.Position(f.Pos()).Line

    if !(c.start <= line && line <= c.end) { continue } if f.Tag == nil { f.Tag = &ast.BasicLit{} } // ... }
  185. for _, f := range x.Fields.List { line := c.fset.Position(f.Pos()).Line

    if !(c.start <= line && line <= c.end) { continue } if f.Tag == nil { f.Tag = &ast.BasicLit{} } // ... }
  186. for _, f := range x.Fields.List { fieldName := ""

    if len(f.Names) != 0 { fieldName = f.Names[0].Name } if f.Names == nil { ident, ok := f.Type.(*ast.Ident) if !ok { continue // anonymous field } fieldName = ident.Name } res, err := c.process(fieldName, f.Tag.Value) if err != nil { rewriteErr = err return true } f.Tag.Value = res }
  187. for _, f := range x.Fields.List { fieldName := ""

    if len(f.Names) != 0 { fieldName = f.Names[0].Name } if f.Names == nil { ident, ok := f.Type.(*ast.Ident) if !ok { continue // anonymous field } fieldName = ident.Name } res, err := c.process(fieldName, f.Tag.Value) if err != nil { rewriteErr = err return true } f.Tag.Value = res }
  188. for _, f := range x.Fields.List { fieldName := ""

    if len(f.Names) != 0 { fieldName = f.Names[0].Name } if f.Names == nil { ident, ok := f.Type.(*ast.Ident) if !ok { continue // anonymous field } fieldName = ident.Name } res, err := c.process(fieldName, f.Tag.Value) if err != nil { rewriteErr = err return true } f.Tag.Value = res }
  189. Processing the tags

  190. func (c *config) process(fieldName, tagVal string) (string, error) { var

    tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil }
  191. func (c *config) process(fieldName, tagVal string) (string, error) { var

    tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil }
  192. func (c *config) process(fieldName, tagVal string) (string, error) { var

    tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil }
  193. func (c *config) process(fieldName, tagVal string) (string, error) { var

    tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil } tags = c.removeTags(tags) tags, err = c.removeTagOptions(tags) if err != nil { return "", err } tags = c.clearTags(tags) tags = c.clearOptions(tags) tags, err = c.addTags(fieldName, tags) if err != nil { return "", err } tags, err = c.addTagOptions(tags) if err != nil { return "", err } if c.sort { sort.Sort(tags) }
  194. func (c *config) process(fieldName, tagVal string) (string, error) { var

    tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil } tags = c.removeTags(tags) tags, err = c.removeTagOptions(tags) if err != nil { return "", err } tags = c.clearTags(tags) tags = c.clearOptions(tags) tags, err = c.addTags(fieldName, tags) if err != nil { return "", err } tags, err = c.addTagOptions(tags) if err != nil { return "", err } if c.sort { sort.Sort(tags) }
  195. func (c *config) process(fieldName, tagVal string) (string, error) { var

    tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil }
  196. func (c *config) rewriteFields(node ast.Node) (ast.Node, error) { var rewriteErr

    error rewriteFunc := func(n ast.Node) bool { x, ok := n.(*ast.StructType) if !ok { return true } for _, f := range x.Fields.List { // process each field // ... } return true } ast.Inspect(node, rewriteFunc) return node, rewriteErr }
  197. package main type Example struct { Foo string } parse

    find modify format gomodifytags
  198. Output the result

  199. func main() { var cfg config node = cfg.parse() start,

    end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch configuration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result
  200. gomodifytags ast.Node (modified)

  201. gomodifytags ast.Node (modified) go/format Pass the modified node to go/format

  202. gomodifytags ast.Node (modified) go/format ... and output the result to

    stdout package main type Example struct { Foo string `json:"foo"` }
  203. package main type Server struct { Name string Port int

    EnableLogs bool BaseDomain string Credentials struct { Username string Password string } } package main type Server struct { Name string `json:"name"` Port int `json:"port"` EnableLogs bool `json:"enable_logs"` BaseDomain string `json:"base_domain"` Credentials struct { Username string `json:"username"` Password string `json:"password"` } `json:"credentials"` } Input Output $ gomodifytags -file example.go -struct Server -add-tags json
  204. gomodifytags ast.Node (modified) go/format Also has support for custom JSON

    output { "start": 3, "end": 5, "lines": [ "type Example struct {", " Foo string `json:\"foo\"`", "}" ] }
  205. package main type Server struct { Name string Port int

    EnableLogs bool BaseDomain string Credentials struct { Username string Password string } } { "start": 3, "end": 12, "lines": [ "type Server struct {", " Name string `xml:\"name\"`", " Port int `xml:\"port\"`", " EnableLogs bool `xml:\"enable_logs\"`", " BaseDomain string `xml:\"base_domain\"`", " Credentials struct {", " Username string `xml:\"username\"`", " Password string `xml:\"password\"`", " } `xml:\"credentials\"`", "}" ] } Input JSON Output $ gomodifytags -file example.go -struct Server -add-tags json -format json
  206. Why stdout? • by default, it means "dry-run" • immediate

    feedback • tools can redirect the output • composable (gomodifytags myfile.go > newfile.go)
  207. func (c *config) format(file ast.Node) (string, error) { switch c.output

    { case "source": // return formatted source case "json": // return only changes in json default: return "", fmt.Errorf("unknown output mode: %s", c.output) } } Output the result
  208. func (c *config) format(file ast.Node) (string, error) { switch c.output

    { case "source": // return formatted source case "json": // return only changes in json default: return "", fmt.Errorf("unknown output mode: %s", c.output) } } Output the result
  209. var buf bytes.Buffer err := format.Node(&buf, c.fset, file) if err

    != nil { return "", err } if c.write { err = ioutil.WriteFile(c.file, buf.Bytes(), 0) if err != nil { return "", err } } return buf.String(), nil Source file
  210. var buf bytes.Buffer err := format.Node(&buf, c.fset, file) if err

    != nil { return "", err } if c.write { err = ioutil.WriteFile(c.file, buf.Bytes(), 0) if err != nil { return "", err } } return buf.String(), nil Source file
  211. var buf bytes.Buffer err := format.Node(&buf, c.fset, file) if err

    != nil { return "", err } if c.write { err = ioutil.WriteFile(c.file, buf.Bytes(), 0) if err != nil { return "", err } } return buf.String(), nil Source file
  212. var buf bytes.Buffer err := format.Node(&buf, c.fset, file) if err

    != nil { return "", err } var lines []string scanner := bufio.NewScanner(bytes.NewBufferString(buf.String())) for scanner.Scan() { lines = append(lines, scanner.Text()) } if c.start > len(lines) { return "", errors.New("line selection is invalid") } ... JSON output
  213. var buf bytes.Buffer err := format.Node(&buf, c.fset, file) if err

    != nil { return "", err } var lines []string scanner := bufio.NewScanner(bytes.NewBufferString(buf.String())) for scanner.Scan() { lines = append(lines, scanner.Text()) } if c.start > len(lines) { return "", errors.New("line selection is invalid") } ... JSON output
  214. out := &output{ Start: c.start, End: c.end, Lines: lines[c.start-1 :

    c.end], } o, err := json.MarshalIndent(out, "", " ") if err != nil { return "", err } return string(o), nil JSON output (cont.)
  215. out := &output{ Start: c.start, End: c.end, Lines: lines[c.start-1 :

    c.end], } o, err := json.MarshalIndent(out, "", " ") if err != nil { return "", err } return string(o), nil JSON output (cont.)
  216. package main type Example struct { Foo string } parse

    find modify gomodifytags format format
  217. package main type Example struct { Foo string } parse

    find modify gomodifytags format Overview
  218. Overview

  219. gomodifytags

  220. gomodifytags package main type Example struct { Foo string }

  221. package main type Example struct { Foo string } -file

    example.go -struct Example -add-tags json gomodifytags
  222. package main type Example struct { Foo string } parse

    gomodifytags
  223. package main type Example struct { Foo string } parse

    find gomodifytags
  224. package main type Example struct { Foo string } parse

    find modify gomodifytags
  225. package main type Example struct { Foo string } parse

    find modify format gomodifytags
  226. package main type Example struct { Foo string } package

    main type Example struct { Foo string `json:"foo"` } parse find modify format gomodifytags
  227. Demo time

  228. Built from the ground up for editors • selecting structs

    based on offset or range of lines • write result to stdout or file • json output for easy parsing • support for unsaved buffers • individual cli flags for all features
  229. Supported editors • vim with vim-go • atom with go-plus

    (thanks Zac Bergquist) • vscode with vscode-go (thanks Ramya Rao) • emacs - WIP (https://github.com/syohex/emacs-go-add-tags/issues/ 6)
  230. Fixed bugs

  231. Recap • A single tool to rule all editors •

    Easy maintenance • Free of bugs due stdlib tooling (go/parser family) • Happy and productive users
  232. Thanks! Fatih Arslan @fatih @fatih fatih@arslan.io