Writing a code generator to make your code more secure - Anderson Queiroz - Blacklane

We all have at some point wished some tool could write that piece of code we repeat over and over again. Some have written a library or small framework for that, but they can only go so far. A few have written a code generator. Besides most of us have written code dealing with sensitive information, such as password or OAuth and JWT tokens.
In this talk, I'll walk through how to write a code generator, using the tooling available in Go. Our code generator will aim to generate code to help us to keep sensitive information safe.

Transcript

1. Writing a code generator to make your code more secure 22nd January 2020 Anderson Queiroz Lead Backend Engineer Blacklane
2. whoami Brazilian Living in Berlin for 2 years 15 years coding experience ~2.5 years writing Go Gopher by gopherize.me (https://gopherize.me) 2
3. Working at We're hiring! (https://www.blacklane.com/en/career/) 3
4. Motivation GopherCon UK 2019 + Go Time #101: Security for Gophers Go package main import "fmt" func main() { fmt.Println("Hello"); } Chapter 1 Gopher by Egon Elbre (https://github.com/egonelbre/gophers) 4
5. Agenda Code generator definition Why they are useful Our scenario Manual approach Generating code Running it 5
6. Code Generator? I haven't found a proper formal definition, but let's check a few: A code generator is a tool or resource that generates a particular sort of code or computer programming language. (techopedia.com) In computing, code generation is the process by which a compiler's code generator converts some intermediate representation of source code into a form that can be readily executed by a machine. (Wikipedia) A program that writes a program Anything capable of writing (generating) code Are we code generators then? Well, we could say so For us today, let's define a code generator as: a program which writes code 6
7. Examples: go test: it scans the packages to be tested, writes out a Go program containing a test harness customized for the package, and then compiles and runs it; Yacc: reads in a description of a grammar and writes out a program to parse that grammar. protoc: Protocol Buffers'compiler sources: (https://blog.golang.org/generate) (https://github.com/protocolbuffers/protobuf) 7
8. Agenda Code generator definition and examples Why they are useful Our scenario Manual approach Generating code Running it 8
9. Why they are useful They automate writing code which would be too expensive to us humans to write. Either because we'd need to be quite knowledgeable in some area, e.g. Protocol Buffers, or too repetitive e.g. stringer. further reading: https://blog.golang.org/generate nice article exploring how to use code generators with Go. 9
10. Agenda Code generator definition and examples Why they are useful Our scenario Manual approach Generating code Running it 10
11. Our scenario We deal with sensitive data all the time, even more now with GDPR. Username and passwords, access tokens, client secrets, secret keys, data which personally identify people and so on. When writing a program, it's normal to face some obstacles, we've followed all the steps in the docs, all looks fine, but still, it does not work. So, debug time! Even better let's print debug it all. It's just on my machine, at most on some testing/QA environment. The code will be code reviewed, no way the debug can make to production. Never put your trust on it. I've seen such print debug almost making to production after have been code reviewed. 11
12. Agenda Code generator definition and examples Why they are useful Our scenario Proposed solution Generating code Running it 12
13. Proposed solution For simplicity, let's say we have a struct holding our user's credentials type Credentials struct { username string password string } In go we can easily decide how a type will be printed, and therefore, avoid such leaks. If we want to prevent it to be printed, we can implement the Stringer interface and print something else instead. 13
14. Proposed solution func (c Credentials) String() string { return fmt.Sprintf("username: %s, password: %s", strings.Repeat("*", len(c.username)), strings.Repeat("*", len(c.password))) } Let's give it a try: log.Printf("[DEBUG] crecentials: \n%#v", credentials) Run 14
15. What happened? Let's have a look at the docs: // Stringer is implemented by any value that has a String method, // which defines the ``native'' format for that value. / // / T Th he e S St tr ri in ng g m me et th ho od d i is s u us se ed d t to o p pr ri in nt t v va al lu ue es s p pa as ss se ed d a as s a an n o op pe er ra an nd d / // / t to o a an ny y f fo or rm ma at t t th ha at t a ac cc ce ep pt ts s a a s st tr ri in ng g o or r t to o a an n u un nf fo or rm ma at tt te ed d p pr ri in nt te er r // such as Print. type Stringer interface { String() string } looks fine... but if you keep scrolling down the code... 15
16. What happened? Just below, we find: // GoStringer is implemented by any value that has a GoString method, // which defines the Go syntax for that value. / // / T Th he e G Go oS St tr ri in ng g m me et th ho od d i is s u us se ed d t to o p pr ri in nt t v va al lu ue es s p pa as ss se ed d a as s a an n o op pe er ra an nd d / // / t to o a a % %# #v v f fo or rm ma at t. . type GoStringer interface { G Go oS St tr ri in ng g( () ) s st tr ri in ng g } 16
17. Let's try now func (c Credentials) String() string { return fmt.Sprintf("username: %s, password: %s", strings.Repeat("*", len(c.username)), strings.Repeat("*", len(c.password))) } func (c Credentials) GoString() string { return c.String() } log.Printf("crecentials: \n%#v", credentials) Run 17
18. and now, keeping repeating it? 19
19. Let's generate this code!
20. Agenda Code generator definition and examples Why they are useful Our scenario Manual approach Generating code Running it 20
21. Our code generation tool obfuscate, a code generation tool to obfuscate sensitive information when printing Go types Requirements: Given a type, implement the Stringer and GoStringer interfaces to obfuscate the data when printing it. Input: - type name - go package Output: - a go file with the generated code 21
22. Generating code - Agenda load the go package lookup for the type implement the Stringer and GoStringer interfaces save to gen_[TYPE_NAME]_obfuscated.go run goimports on gen_[TYPE_NAME]_obfuscated.go 22
23. Loading a Go package We'll use the Go tool packages Package packages loads Go packages for inspection and analysis. It returns a data struct representing a package. It's Go code as data! Not just a huge text hard to infer meaning from. 23
24. Loading a Go package (code) loadCfg := &packages.Config{ Mode: packages.NeedTypes | packages.NeedTypesInfo | packages.NeedSyntax | packages.NeedName, Context: ctx, } Mode controls which information will be returned. The docs fully explain which each of them means. Tl;dr: we are loading, types and information about them, syntax and names. 24
25. Loading a Go package (code) For simplicity we'll only deal with one package and one type at a time. loadCfg := &packages.Config{ Mode: packages.NeedTypes | packages.NeedTypesInfo | packages.NeedSyntax | packages.NeedName, Context: ctx, } p pk kg gs s, , e er rr r : := = p pa ac ck ka ag ge es s. .L Lo oa ad d( (l lo oa ad dC Cf fg g) ) if err != nil { logger.Fatal().Err(err).Msgf("could not load packages") } if len(pkgs) != 1 { logger.Fatal().Err(err).Msgf( "expecting only one package, received %d: %v", len(pkgs), pkgs) } pkg := pkgs[0] 25
26. Generating code - Agenda load the go package lookup for the type to obfuscate implement the Stringer and GoStringer interfaces save to gen_[TYPE_NAME]_obfuscated.go run goimports on gen_[TYPE_NAME]_obfuscated.go 26
27. Searching the type Types: type information for the loaded package Scope(): the objects declared, type names, variables, constants and functions Lookup(): the object with the given name if all succeed, we'll be holding our type! // Lookup the type to obfuscate o ob bj j : := = p pk kg g. .T Ty yp pe es s. .S Sc co op pe e( () ). .L Lo oo ok ku up p( (t ta ar rg ge et tT Ty yp pe eN Na am me e) ) if obj == nil { // Nothing found logger.Fatal().Msgf("could not find type %q in package %q", targetTypeName, pkg.Name) } 27
28. Generating code - Agenda load the go package lookup for the type implement the Stringer and GoStringer interfaces save to gen_[TYPE_NAME]_obfuscated.go run goimports on gen_[TYPE_NAME]_obfuscated.go 28
29. What we'll write Now we found out type in the code, we can implement the Stringer and GoStringer interfaces. We want something like: func (t typeName) String() string { return "*****" } func (t typeName) GoString() string { return "*****" } 29
30. Writing the code We can use the package text/template to write our code codeTmpl = ` // Code generated by obfuscate. DO NOT EDIT. package {{.PkgName}} import ( "fmt" "strings" ) func ({{.ReceiverName}} {{.TypeName}}) String() string { return %s } func ({{.ReceiverName}} {{.TypeName}}) GoString() string { return %s } ` 30
31. Let's play a bit For the propose of exploring the information we have about our type we'll divide it in two cases: The underlying type is a string: type Password string or it's something else, such as: type Secret struct { Key1 string Key2 int } if its underlying type is a string, we'll print n *, where n is the length of the string, 10 * otherwise 31
32. Checking the underlying type It couldn't be easier: func underlyingString(obj types.Object) bool { r re et tu ur rn n o ob bj j. .T Ty yp pe e( () ). .U Un nd de er rl ly yi in ng g( () ). .S St tr ri in ng g( () ) = == = " "s st tr ri in ng g" " } and then we choose one of the two as our implementation tmplStr = `fmt.Sprint(strings.Repeat("{{.ReplaceBy}}", len(string(s))))` tmplOther = `fmt.Sprint(strings.Repeat("{{.ReplaceBy}}", 10))` func chooseTmpl(isString bool) string { if isString { return fmt.Sprintf(codeTmpl, tmplStr, tmplStr) } return fmt.Sprintf(codeTmpl, tmplOther, tmplOther) } 32
33. Generating the code i is sS St tr ri in ng g : := = u un nd de er rl ly yi in ng gS St tr ri in ng g( (o ob bj j) ) vars := tmplVars{} vars.PkgName = pkg.Name vars.TypeName = targetTypeName vars.ReplaceBy = "*" vars.ReceiverName = receiverName(targetTypeName) t, err := template.New("tmpl").Parse(chooseTmpl(isString)) if err != nil { logger.Fatal().Err(err).Msgf("could not parse template") } err = t.Execute(w, vars) if err != nil { logger.Fatal().Msgf("could not execute template for %s", targetTypeName) } 33
34. Generating the code Here we gather the data to fill in our template isString := underlyingString(obj) v va ar rs s : := = t tm mp pl lV Va ar rs s{ {} } v va ar rs s. .P Pk kg gN Na am me e = = p pk kg g. .N Na am me e v va ar rs s. .T Ty yp pe eN Na am me e = = t ta ar rg ge et tT Ty yp pe eN Na am me e v va ar rs s. .R Re ep pl la ac ce eB By y = = " "* *" " v va ar rs s. .R Re ec ce ei iv ve er rN Na am me e = = r re ec ce ei iv ve er rN Na am me e( (t ta ar rg ge et tT Ty yp pe eN Na am me e) ) t, err := template.New("tmpl").Parse(chooseTmpl(isString)) if err != nil { logger.Fatal().Err(err).Msgf("could not parse template") } err = t.Execute(w, vars) if err != nil { logger.Fatal().Msgf("could not execute template for %s", targetTypeName) } 34
35. Generating the code We parse our template choosing it based on the underlying type of our type isString := underlyingString(obj) vars := tmplVars{} vars.PkgName = pkg.Name vars.TypeName = targetTypeName vars.ReplaceBy = "*" vars.ReceiverName = receiverName(targetTypeName) t t, , e er rr r : := = t te em mp pl la at te e. .N Ne ew w( (" "t tm mp pl l" ") ). .P Pa ar rs se e( (c ch ho oo os se eT Tm mp pl l( (i is sS St tr ri in ng g) )) ) if err != nil { logger.Fatal().Err(err).Msgf("could not parse template") } err = t.Execute(w, vars) if err != nil { logger.Fatal().Msgf("could not execute template for %s", targetTypeName) } 35
36. Generating the code Finally we execute it. w is the file where we'll save our code isString := underlyingString(obj) vars := tmplVars{} vars.PkgName = pkg.Name vars.TypeName = targetTypeName vars.ReplaceBy = "*" vars.ReceiverName = receiverName(targetTypeName) t, err := template.New("tmpl").Parse(chooseTmpl(isString)) if err != nil { logger.Fatal().Err(err).Msgf("could not parse template") } e er rr r = = t t. .E Ex xe ec cu ut te e( (w w, , v va ar rs s) ) if err != nil { logger.Fatal().Msgf("could not execute template for %s", targetTypeName) } 36
37. Generating code - Agenda load the go package lookup for the type implement the Stringer and GoStringer interfaces save to gen_[TYPE_NAME]_obfuscated.go run goimports on gen_[TYPE_NAME]_obfuscated.go 37
38. Save the generated code We just need a file to write our template to, after having written it, we close the file fileNameTmpl = "gen_%s_obfuscated.go" func createFile(targetTypeName string, logger zerolog.Logger) *os.File { fileName := fmt.Sprintf(fileNameTmpl, strings.ToLower(targetTypeName)) f, err := os.Create(fileName) if err != nil { logger.Fatal().Msgf("could not create file %s", fileName) } return f } if err := file.Close(); err != nil { logger.Fatal().Err(err).Msgf("could not close file %s", file.Name()) } 38
39. Generating code - Agenda load the go package lookup for the type implement the Stringer and GoStringer interfaces save to gen_[TYPE_NAME]_obfuscated.go run goimports on gen_[TYPE_NAME]_obfuscated.go 39
40. goimport the code Now we have our code written to a file, let's goimport it just to make sure all is fine We just call goimports from the cli func runGoimport(file *os.File, logger zerolog.Logger) { c cm md dG Go oI Im mp po or rt ts s : := = e ex xe ec c. .C Co om mm ma an nd d( (" "g go oi im mp po or rt ts s" ", , " "- -w w" ", , f fi il le e. .N Na am me e( () )) ) i if f o ou ut t, , e er rr r : := = c cm md dG Go oI Im mp po or rt ts s. .C Co om mb bi in ne ed dO Ou ut tp pu ut t( () ); ; e er rr r ! != = n ni il l { { logger.Fatal().Err(err).Msgf("could not run goimports: %v: %v\n%s", strings.Join(cmdGoImports.Args, " "), err, out) } } 40
41. Generating code - Agenda load the go package lookup for the type implement the Stringer and GoStringer interfaces save to gen_[TYPE_NAME]_obfuscated.go run goimports on gen_[TYPE_NAME]_obfuscated.go all together 41
42. Putting it all together targetTypeName := os.Args[1] pkg := loadPkg(ctx, logger) logger.Debug().Msgf("looking into package %q for type %q", pkg.Name, targetTypeName) // Lookup the type to obfuscate obj := pkg.Types.Scope().Lookup(targetTypeName) if obj == nil { // Nothing found logger.Fatal().Msgf("could not find type %q in package %q", targetTypeName, pkg.Name) } file := createFile(targetTypeName, logger) generateCode(obj, pkg, targetTypeName, file, logger) if err := file.Close(); err != nil { logger.Fatal().Err(err).Msgf("could not close file %s", file.Name()) } // format the file runGoimport(file, logger) 42
43. Agenda Code generator definition and examples Why they are useful Our scenario Manual approach Generating code Running it 43
44. Running it obfuscate, our code generation tool, is a cli program receiving a type name to obfuscate. It loads the go package in the current directory to look for the given type usage: obfuscate typeName 44
45. Running it Go 1.4 added generate, a tool to help to run code generation tools It will look for comments in the flowing form: //go:generate tool tool_arguments Where - tool is any binary - tool_arguments are the arguments required by tool go generate is not part of the building process, therefore we need to call it before go build $ go generate $ go build $ go test ... 45
46. Running it for our obfuscate tool: //go:generate obfuscate Credentials type Credentials struct { username string password string } 46
47. Demo time! 47 48
48. Thanks 49
49. Questions?
50. Thank you 22nd January 2020 Anderson Queiroz Lead Backend Engineer Blacklane @AinSoph (http://twitter.com/AinSoph) contato@andersonq.eti.br (mailto:contato@andersonq.eti.br) https://www.linkedin.com/in/andersonq/ (https://www.linkedin.com/in/andersonq/) https://github.com/AndersonQ/talks (https://github.com/AndersonQ/talks)