Building a go tool to modify struct tags

Slide 1

Slide 1 text

Building a go tool to modify struct tags Fatih Arslan Software Engineer @DigitalOcean

Slide 2

Slide 2 text

Why do we need tooling?

Slide 3

Slide 3 text

type Example struct { Foo string }

Slide 4

Slide 4 text

type Example struct { Foo string `json:"foo"` }

Slide 5

Slide 5 text

What about structs with many ﬁelds?

Slide 6

Slide 6 text

type Example struct { StatusID int64 Foo string Bar bool Server struct { Address string TLS bool } DiskSize int64 Volumes []string }

Slide 7

Slide 7 text

type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo" xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` }

Slide 8

Slide 8 text

Working with struct tags is not fun

Slide 9

Slide 9 text

So, what's a struct tag? type Example struct { Foo string `json:"foo"` }

Slide 10

Slide 10 text

There is no ofﬁcial spec of the struct tag

Slide 11

Slide 11 text

Struct tag deﬁnition

Slide 12

Slide 12 text

But fortunately, we have an inofﬁcial one in stdlib!

Slide 13

Slide 13 text

Spec is deﬁned in the reﬂect package

Slide 14

Slide 14 text

reﬂect.StructTag

Slide 15

Slide 15 text

Let's decompose the spec

Slide 16

Slide 16 text

json:"foo"

Slide 17

Slide 17 text

json:"foo" key

Slide 18

Slide 18 text

json:"foo" value (quoted)

Slide 19

Slide 19 text

json:"foo" colon (not speciﬁed clearly)

Slide 20

Slide 20 text

json:"foo" xml:"foo" space separation

Slide 21

Slide 21 text

type Example struct { Foo string `json:"foo"` } backticks

Slide 22

Slide 22 text

type Example struct { Foo string `json:"foo"` }

Slide 23

Slide 23 text

type Example struct { Foo string "json:\"foo\"" } quotes instead of backticks (works, but not fun to deal with)

Slide 24

Slide 24 text

json:"foo,omitempty" value

Slide 25

Slide 25 text

json:"foo,omitempty" option (not part of the spec)

Slide 26

Slide 26 text

Recap

Slide 27

Slide 27 text

`json:"foo,omitempty" option (not part of the spec) space separation key value xml:"bar"` backtick

Slide 28

Slide 28 text

Back to our initial slide!

Slide 29

Slide 29 text

Slide 30

Slide 30 text

It has some errors

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Slide 39

Slide 39 text

First attempt (to automate it)

Slide 40

Slide 40 text

Old implementation • From the cursor search backwards for the first 'struct {' literal • Once found, do a forward search for the right hand brace } • Get the line numbers between the two braces • For each line, get the first identifier, convert it to camel_case word and then append to the same line • A combination of Vim's search() function and regex is being used

Slide 41

Slide 41 text

type Example struct { Foo string Bar |bool } cursor

Slide 42

Slide 42 text

type Example struct { Foo string Bar |bool }

Slide 43

Slide 43 text

type Example struct { Foo string Bar |bool }

Slide 44

Slide 44 text

type Example struct { Foo string Bar |bool }

Slide 45

Slide 45 text

type Example struct { Foo string Bar |bool }

Slide 46

Slide 46 text

type Example struct { Foo string `json:"foo"` Bar |bool `json:"bar"` }

Slide 47

Slide 47 text

Problems

Slide 48

Slide 48 text

No Formatting type Example struct { Bar string `json:"bar"` Foo bool `json:"foo"` } type Example struct { Bar string `json:"bar"` Foo bool `json:"foo"` } want have

Slide 49

Slide 49 text

In-line comments type Example struct { Bar string `json:"bar"` // comment for bar Foo bool `json:"foo"` } type Example struct { Bar string // comment for bar `json:"bar"` Foo bool `json:"foo"` } want have

Slide 50

Slide 50 text

Nested structs type Example struct { Bar bool `json:"bar"` Server struct { `json:"server"` Address string `json:"address"` } } type Example struct { Bar bool `json:"bar"` Server struct { Address string `json:"address"` } `json:"server"` } have want

Slide 51

Slide 51 text

Duplicate tags type Example struct { Bar string `json:"bar"` `xml:"bar"` Foo bool `xml:"foo"` } have type Example struct { Bar string `json:"bar" xml:"bar"` Foo bool `xml:"foo"` } want

Slide 52

Slide 52 text

more quirks • not able to remove tags • not able to add or remove options • ﬁeld with interface{} types don't work • comment with braces ({ and }) don't work • ...

Slide 53

Slide 53 text

How do we ﬁx it?

Slide 54

Slide 54 text

Second attempt (and also the ﬁnal one )

Slide 55

Slide 55 text

Two issues that need to be ﬁxed Parse struct Parse struct tag

Slide 56

Slide 56 text

1. Parse Struct

Slide 57

Slide 57 text

The Go Parser Family

Slide 58

Slide 58 text

go/token go/scanner go/parser go/ast go/printer

Slide 59

Slide 59 text

First, a quick tutorial about go/ast

Slide 60

Slide 60 text

3 + 2 *ast.BinaryExpr

Slide 61

Slide 61 text

3 + 2 *ast.BinaryExpr *ast.BasicLit 3

Slide 62

Slide 62 text

3 + 2 *ast.BinaryExpr *ast.BasicLit 2

Slide 63

Slide 63 text

3 + 2 *ast.BinaryExpr *ast.BasicLit *ast.BasicLit 3 2 +

Slide 64

Slide 64 text

3 + (7 - 5) *ast.BinaryExpr *ast.BasicLit 3 + *ast.BinaryExpr *ast.BasicLit *ast.BasicLit 5 - 7

Slide 65

Slide 65 text

3 + (7 - 5) *ast.BinaryExpr *ast.BasicLit 3 + *ast.BinaryExpr *ast.BasicLit *ast.BasicLit 5 - 7

Slide 66

Slide 66 text

type Example struct { Foo string `json:"foo"` } *ast.TypeSpec

Slide 67

Slide 67 text

type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType

Slide 68

Slide 68 text

type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType *ast.FieldList

Slide 69

Slide 69 text

type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field

Slide 70

Slide 70 text

type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident Foo

Slide 71

Slide 71 text

type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field *ast.Ident string

Slide 72

Slide 72 text

type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field *ast.BasicLit `json:"foo"` (a.k.a: struct tag)

Slide 73

Slide 73 text

type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident *ast.BasicLit

Slide 74

Slide 74 text

ast.Node

Slide 75

Slide 75 text

All nodes deﬁne the starting and ending positions

Slide 76

Slide 76 text

How do we parse a Struct?

Slide 77

Slide 77 text

Slide 78

Slide 78 text

Slide 79

Slide 79 text

Slide 80

Slide 80 text

Slide 81

Slide 81 text

Slide 82

Slide 82 text

Slide 83

Slide 83 text

2. Parse Struct Tag

Slide 84

Slide 84 text

How to parse a Struct Tag?

Slide 85

Slide 85 text

Remember this?

Slide 86

Slide 86 text

package main import ( "fmt" "reflect" ) func main() { tag := reflect.StructTag(`json:"foo"`) value := tag.Get("json") fmt.Printf("value: %q\n", value) }

Slide 87

Slide 87 text

package main import ( "fmt" "reflect" ) func main() { tag := reflect.StructTag(`json:"foo"`) value := tag.Get("json") fmt.Printf("value: %q\n", value) } $ go run main.go value: "foo"

Slide 88

Slide 88 text

reﬂect.StructTag is not perfect ...

Slide 89

Slide 89 text

Issues with reﬂect.StructTag • can't detect if the tag is malformed (only go vet knows that) • doesn't know the semantics of options (i.e: omitempty) • doesn't return all existing tags • modifying existing tags is not possible

Slide 90

Slide 90 text

reﬂect cmd/vet

Slide 91

Slide 91 text

Let's improve it with a custom package

Slide 92

Slide 92 text

import "github.com/fatih/structtag"

Slide 93

Slide 93 text

structtag

Slide 94

Slide 94 text

Parse and list all tags tags, err := structtag.Parse(`json:"foo,omitempty" xml:"foo"`) if err != nil { panic(err) } // iterate over all key-value pairs for _, t := range tags.Tags() { fmt.Printf("tag: %+v\n", t) } $ go run main.go tag: json:"foo,omitempty" tag: xml:"foo"

Slide 95

Slide 95 text

Get a single Tag tags, err := structtag.Parse(`json:"foo,omitempty" xml:"foo"`) if err != nil { panic(err) } jsonTag, err := tags.Get("json") if err != nil { panic(err) } fmt.Println(jsonTag) // Output: json:"foo,omitempty" fmt.Println(jsonTag.Key) // Output: json fmt.Println(jsonTag.Name) // Output: foo fmt.Println(jsonTag.Options) // Output: [omitempty]

Slide 96

Slide 96 text

Change existing tag jsonTag, err := tags.Get(`json:"foo,omitempty"`) if err != nil { panic(err) } jsonTag.Name = "bar" jsonTag.Options = nil tags.Set(jsonTag) fmt.Println(tags) json:"bar"

Slide 97

Slide 97 text

Add new tag tags, err := structtag.Parse(`json:"foo,omitempty" xml:"foo"`) hclTag := &structtag.Tag{ Key: "hcl", Name: "gopher", Options: []string{"squash"}, } // add new tag tags.Set(hclTag) fmt.Println(tags) json:"foo,omitempty" xml:"foo" hcl:"gopher,squash"

Slide 98

Slide 98 text

Add/remove options tags, err := structtag.Parse(`json:"foo" xml:"bar,comment"`) if err != nil { panic(err) } tags.AddOptions("json", "omitempty") tags.DeleteOptions("xml", "comment") fmt.Println(tags) // json:"foo,omitempty" xml:"bar"

Slide 99

Slide 99 text

Both issues are ﬁxed now Parse struct (using go/parser) Parse struct tag (using fatih/structtag)

Slide 100

Slide 100 text

Write a CLI tool

Slide 101

Slide 101 text

gomodifytags

Slide 102

Slide 102 text

go get github.com/fatih/gomodifytags

Slide 103

Slide 103 text

func main() { var cfg config node = cfg.parse() start, end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch conﬁguration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result

Slide 104

Slide 104 text

Slide 105

Slide 105 text

Fetch conﬁguration settings

Slide 106

Slide 106 text

cfg := &config{ file: *flagFile, line: *flagLine, structName: *flagStruct, offset: *flagOffset, output: *flagOutput, write: *flagWrite, clear: *flagClearTags, clearOption: *flagClearOptions, transform: *flagTransform, sort: *flagSort, override: *flagOverride, } $ gomodifytags --file example.go ... Use ﬂags to set conﬁguration

Slide 107

Slide 107 text

Things we need: 1. What content to process 2. Where and what to output 3. Which struct to modify 4. Which tags to modify

Slide 108

Slide 108 text

Things we need: type config struct { file string modified io.Reader output string write bool offset int structName string line string start, end int remove []string add []string override bool transform string sort bool clear bool addOpts []string removeOpts []string clearOpt bool } 1. What content to process 2. Where and what to output 3. Which struct to modify 4. Which tags to modify

Slide 109

Slide 109 text

Slide 110

Slide 110 text

Slide 111

Slide 111 text

Slide 112

Slide 112 text

package main type Example struct { Foo string } $ gomodifytags -ﬁle example.go -struct Example -add-tags json

Slide 113

Slide 113 text

package main type Example struct { Foo string } parse gomodifytags

Slide 114

Slide 114 text

Parse content

Slide 115

Slide 115 text

Slide 116

Slide 116 text

parse gomodifytags go/parser package main type Example struct { Foo string }

Slide 117

Slide 117 text

parse gomodifytags go/parser go/ast.Node package main type Example struct { Foo string }

Slide 118

Slide 118 text

gomodifytags parse parse go/parser package main type Example struct { Foo string } *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident tag

Slide 119

Slide 119 text

func (c *config) parse() (ast.Node, error) { c.fset = token.NewFileSet() var contents interface{} if c.modified != nil { archive, err := buildutil.ParseOverlayArchive(c.modified) if err != nil { return nil, fmt.Errorf("failed to parse -modified archive: %v", err) } fc, ok := archive[c.file] if !ok { return nil, fmt.Errorf("couldn't find %s in archive", c.file) } contents = fc } return parser.ParseFile(c.fset, c.file, contents, parser.ParseComments) } Parse content

Slide 120

Slide 120 text

Slide 121

Slide 121 text

package main type Example struct { Foo string } parse gomodifytags parse

Slide 122

Slide 122 text

package main type Example struct { Foo string } parse ﬁnd gomodifytags

Slide 123

Slide 123 text

Find start and end positions

Slide 124

Slide 124 text

Slide 125

Slide 125 text

gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident tag this is our ﬁle ast.Node

Slide 126

Slide 126 text

parse ﬁnd gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit which is an AST tree

Slide 127

Slide 127 text

parse ﬁnd gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit select struct that matches our criteria ...

Slide 128

Slide 128 text

There are many ways to ﬁnd a struct

Slide 129

Slide 129 text

package main type Example struct { Foo string } type Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Slide 130

Slide 130 text

package main type Example struct { Foo string } type Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Slide 131

Slide 131 text

package main type Example struct { Foo string } type Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 struct name: Server

Slide 132

Slide 132 text

package main type Example struct { Foo string } type Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 struct name: Server start line: 8 end line: 10

Slide 133

Slide 133 text

package main type Example struct { Foo string } type Server struct { Name string Port| int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 offset byte: 96 (of 163)

Slide 134

Slide 134 text

Slide 135

Slide 135 text

package main type Example struct { Foo string } type Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 start line: 8 end line: 10 .. or specify explicit lines

Slide 136

Slide 136 text

package main type Example struct { Foo string } type Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 start line: 9 end line: 9 .. or specify explicit lines

Slide 137

Slide 137 text

parse ﬁnd gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit select struct that matches our criteria ...

Slide 138

Slide 138 text

gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit ... and get the start and end lines start line: 8 end line: 10

Slide 139

Slide 139 text

func (c *config) findSelection(node ast.Node) (int, int, error) { if c.line != "" { return c.lineSelection(node) } else if c.offset != 0 { return c.offsetSelection(node) } else if c.structName != "" { return c.structSelection(node) } return 0, 0, errors.New("-line, -offset or -struct is not passed") } Find selection

Slide 140

Slide 140 text

Slide 141

Slide 141 text

Collecting structs

Slide 142

Slide 142 text

// collectStructs collects and maps structType nodes to their positions func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType, 0) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs }

Slide 143

Slide 143 text

// collectStructs collects and maps structType nodes to their positions func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs } type structType struct { name string node *ast.StructType }

Slide 144

Slide 144 text

// collectStructs collects and maps structType nodes to their positions func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs }

Slide 145

Slide 145 text

// collectStructs collects and maps structType nodes to their positions func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs } type structType struct { name string node *ast.StructType }

Slide 146

Slide 146 text

// collectStructs collects and maps structType nodes to their positions func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs }

Slide 147

Slide 147 text

var encStruct *ast.StructType for _, st := range collectStructs() { if st.name == c.structName { encStruct = st.node } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line } Struct selection

Slide 148

Slide 148 text

Slide 149

Slide 149 text

Slide 150

Slide 150 text

Slide 151

Slide 151 text

var encStruct *ast.StructType for _, st := range collectStructs() { structBegin := c.fset.Position(st.node.Pos()).Offset structEnd := c.fset.Position(st.node.End()).Offset if structBegin <= c.offset && c.offset <= structEnd { encStruct = st.node break } } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line Offset selection

Slide 152

Slide 152 text

Slide 153

Slide 153 text

Slide 154

Slide 154 text

Slide 155

Slide 155 text

Slide 156

Slide 156 text

package main type Example struct { Foo string } parse ﬁnd gomodifytags ﬁnd

Slide 157

Slide 157 text

package main type Example struct { Foo string } parse ﬁnd modify gomodifytags

Slide 158

Slide 158 text

Rewrite the struct tag

Slide 159

Slide 159 text

Slide 160

Slide 160 text

gomodifytags start and end lines we're going to modify start line: 8 end line: 10

Slide 161

Slide 161 text

gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit our parsed AST start line: 8 end line: 10

Slide 162

Slide 162 text

gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit struct we're going to modify start line: 8 end line: 10 *ast.StructType

Slide 163

Slide 163 text

gomodifytags *ast.StructType *ast.FieldList

Slide 164

Slide 164 text

gomodifytags *ast.StructType 6 7 8 9 10 11 12 *ast.Field *ast.Field *ast.Field *ast.Field *ast.Field

Slide 165

Slide 165 text

gomodifytags *ast.StructType *ast.Field []*ast.Ident *ast.Ident *ast.BasicLit

Slide 166

Slide 166 text

type Example struct { Foo string `json:"foo"` } gomodifytags *ast.StructType *ast.Field []*ast.Ident *ast.Ident *ast.BasicLit

Slide 167

Slide 167 text

type Example struct { Foo string `json:"foo" } gomodifytags *ast.StructType *ast.Field Foo string `json:"foo"`

Slide 168

Slide 168 text

type Example struct { Foo string `json:"foo"` } gomodifytags *ast.StructType *ast.Field Foo string `json:"foo"`

Slide 169

Slide 169 text

type Example struct { Foo string `json:"bar" } gomodifytags *ast.StructType *ast.Field Foo string `json:"foo" `json:"bar"` type Example struct { Foo string `json:"foo"` }

Slide 170

Slide 170 text

How do we rewrite a struct tag?

Slide 171

Slide 171 text

Slide 172

Slide 172 text

Slide 173

Slide 173 text

type Example struct { DiskSize string } gomodifytags *ast.StructType *ast.Field DiskSize string

Slide 174

Slide 174 text

type Example struct { DiskSize string } gomodifytags *ast.StructType *ast.Field DiskSize string

Slide 175

Slide 175 text

gomodifytags *ast.StructType *ast.Field DiskSize string json:"disk_size" process() type Example struct { DiskSize string } type Example struct { DiskSize string `json:"disk_size"` }

Slide 176

Slide 176 text

src := `package main type Example struct { Foo string` + " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { // found field! field.Tag.Value = process(field) } return false }) tags = c.removeTags(tags) tags, err = c.removeTagOptions(tags) if err != nil { return "", err } tags = c.clearTags(tags) tags = c.clearOptions(tags) tags, err = c.addTags(fieldName, tags) if err != nil { return "", err } tags, err = c.addTagOptions(tags) if err != nil { return "", err } if c.sort { sort.Sort(tags) }

Slide 177

Slide 177 text

gomodifytags *ast.StructType 8 9 10 *ast.Field *ast.Field *ast.Field *ast.Field *ast.Field select ﬁeld between start and end lines ... Modify overview

Slide 178

Slide 178 text

gomodifytags *ast.StructType 8 9 10 *ast.Field *ast.Field *ast.Field *ast.Field *ast.Field ... and rewrite selected ﬁelds tags 8 9 10 *ast.Field *ast.Field *ast.Field *ast.Field *ast.Field rewrite Modify overview

Slide 179

Slide 179 text

package main type Example struct { Foo string } parse ﬁnd modify gomodifytags modify

Slide 180

Slide 180 text

func (c *config) rewriteFields(node ast.Node) (ast.Node, error) { var rewriteErr error rewriteFunc := func(n ast.Node) bool { x, ok := n.(*ast.StructType) if !ok { return true } for _, f := range x.Fields.List { // process each field // ... } return true } ast.Inspect(node, rewriteFunc) return node, rewriteErr }

Slide 181

Slide 181 text

Slide 182

Slide 182 text

Slide 183

Slide 183 text

for _, f := range x.Fields.List { line := c.fset.Position(f.Pos()).Line if !(c.start <= line && line <= c.end) { continue } if f.Tag == nil { f.Tag = &ast.BasicLit{} } // ... }

Slide 184

Slide 184 text

for _, f := range x.Fields.List { line := c.fset.Position(f.Pos()).Line if !(c.start <= line && line <= c.end) { continue } if f.Tag == nil { f.Tag = &ast.BasicLit{} } // ... }

Slide 185

Slide 185 text

for _, f := range x.Fields.List { line := c.fset.Position(f.Pos()).Line if !(c.start <= line && line <= c.end) { continue } if f.Tag == nil { f.Tag = &ast.BasicLit{} } // ... }

Slide 186

Slide 186 text

for _, f := range x.Fields.List { fieldName := "" if len(f.Names) != 0 { fieldName = f.Names[0].Name } if f.Names == nil { ident, ok := f.Type.(*ast.Ident) if !ok { continue // anonymous field } fieldName = ident.Name } res, err := c.process(fieldName, f.Tag.Value) if err != nil { rewriteErr = err return true } f.Tag.Value = res }

Slide 187

Slide 187 text

Slide 188

Slide 188 text

Slide 189

Slide 189 text

Processing the tags

Slide 190

Slide 190 text

Slide 191

Slide 191 text

Slide 192

Slide 192 text

Slide 193

Slide 193 text

func (c *config) process(fieldName, tagVal string) (string, error) { var tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil } tags = c.removeTags(tags) tags, err = c.removeTagOptions(tags) if err != nil { return "", err } tags = c.clearTags(tags) tags = c.clearOptions(tags) tags, err = c.addTags(fieldName, tags) if err != nil { return "", err } tags, err = c.addTagOptions(tags) if err != nil { return "", err } if c.sort { sort.Sort(tags) }

Slide 194

Slide 194 text

Slide 195

Slide 195 text

Slide 196

Slide 196 text

Slide 197

Slide 197 text

package main type Example struct { Foo string } parse ﬁnd modify format gomodifytags

Slide 198

Slide 198 text

Output the result

Slide 199

Slide 199 text

Slide 200

Slide 200 text

gomodifytags ast.Node (modiﬁed)

Slide 201

Slide 201 text

gomodifytags ast.Node (modiﬁed) go/format Pass the modiﬁed node to go/format

Slide 202

Slide 202 text

gomodifytags ast.Node (modiﬁed) go/format ... and output the result to stdout package main type Example struct { Foo string `json:"foo"` }

Slide 203

Slide 203 text

package main type Server struct { Name string Port int EnableLogs bool BaseDomain string Credentials struct { Username string Password string } } package main type Server struct { Name string `json:"name"` Port int `json:"port"` EnableLogs bool `json:"enable_logs"` BaseDomain string `json:"base_domain"` Credentials struct { Username string `json:"username"` Password string `json:"password"` } `json:"credentials"` } Input Output $ gomodifytags -ﬁle example.go -struct Server -add-tags json

Slide 204

Slide 204 text

gomodifytags ast.Node (modiﬁed) go/format Also has support for custom JSON output { "start": 3, "end": 5, "lines": [ "type Example struct {", " Foo string `json:\"foo\"`", "}" ] }

Slide 205

Slide 205 text

package main type Server struct { Name string Port int EnableLogs bool BaseDomain string Credentials struct { Username string Password string } } { "start": 3, "end": 12, "lines": [ "type Server struct {", " Name string `xml:\"name\"`", " Port int `xml:\"port\"`", " EnableLogs bool `xml:\"enable_logs\"`", " BaseDomain string `xml:\"base_domain\"`", " Credentials struct {", " Username string `xml:\"username\"`", " Password string `xml:\"password\"`", " } `xml:\"credentials\"`", "}" ] } Input JSON Output $ gomodifytags -ﬁle example.go -struct Server -add-tags json -format json

Slide 206

Slide 206 text

Why stdout? • by default, it means "dry-run" • immediate feedback • tools can redirect the output • composable (gomodifytags myﬁle.go > newﬁle.go)

Slide 207

Slide 207 text

func (c *config) format(file ast.Node) (string, error) { switch c.output { case "source": // return formatted source case "json": // return only changes in json default: return "", fmt.Errorf("unknown output mode: %s", c.output) } } Output the result

Slide 208

Slide 208 text

Slide 209

Slide 209 text

var buf bytes.Buffer err := format.Node(&buf, c.fset, file) if err != nil { return "", err } if c.write { err = ioutil.WriteFile(c.file, buf.Bytes(), 0) if err != nil { return "", err } } return buf.String(), nil Source ﬁle

Slide 210

Slide 210 text

Slide 211

Slide 211 text

Slide 212

Slide 212 text

var buf bytes.Buffer err := format.Node(&buf, c.fset, file) if err != nil { return "", err } var lines []string scanner := bufio.NewScanner(bytes.NewBufferString(buf.String())) for scanner.Scan() { lines = append(lines, scanner.Text()) } if c.start > len(lines) { return "", errors.New("line selection is invalid") } ... JSON output

Slide 213

Slide 213 text

Slide 214

Slide 214 text

out := &output{ Start: c.start, End: c.end, Lines: lines[c.start-1 : c.end], } o, err := json.MarshalIndent(out, "", " ") if err != nil { return "", err } return string(o), nil JSON output (cont.)

Slide 215

Slide 215 text

out := &output{ Start: c.start, End: c.end, Lines: lines[c.start-1 : c.end], } o, err := json.MarshalIndent(out, "", " ") if err != nil { return "", err } return string(o), nil JSON output (cont.)

Slide 216

Slide 216 text

package main type Example struct { Foo string } parse ﬁnd modify gomodifytags format format

Slide 217

Slide 217 text

package main type Example struct { Foo string } parse ﬁnd modify gomodifytags format Overview

Slide 218

Slide 218 text

Overview

Slide 219

Slide 219 text

gomodifytags

Slide 220

Slide 220 text

gomodifytags package main type Example struct { Foo string }

Slide 221

Slide 221 text

package main type Example struct { Foo string } -ﬁle example.go -struct Example -add-tags json gomodifytags

Slide 222

Slide 222 text

package main type Example struct { Foo string } parse gomodifytags

Slide 223

Slide 223 text

package main type Example struct { Foo string } parse ﬁnd gomodifytags

Slide 224

Slide 224 text

package main type Example struct { Foo string } parse ﬁnd modify gomodifytags

Slide 225

Slide 225 text

package main type Example struct { Foo string } parse ﬁnd modify format gomodifytags

Slide 226

Slide 226 text

package main type Example struct { Foo string } package main type Example struct { Foo string `json:"foo"` } parse ﬁnd modify format gomodifytags

Slide 227

Slide 227 text

Demo time

Slide 228

Slide 228 text

Built from the ground up for editors • selecting structs based on offset or range of lines • write result to stdout or ﬁle • json output for easy parsing • support for unsaved buffers • individual cli ﬂags for all features

Slide 229

Slide 229 text

Supported editors • vim with vim-go • atom with go-plus (thanks Zac Bergquist) • vscode with vscode-go (thanks Ramya Rao) • emacs - WIP (https://github.com/syohex/emacs-go-add-tags/issues/ 6)

Slide 230

Slide 230 text

Fixed bugs

Slide 231

Slide 231 text

Recap • A single tool to rule all editors • Easy maintenance • Free of bugs due stdlib tooling (go/parser family) • Happy and productive users

Slide 232

Slide 232 text

Thanks! Fatih Arslan @fatih @fatih fatih@arslan.io