3+ years of Go in Prod

An overview of useful techniques, conventions and talks collected over 3 years of using Go as the main language in my work projects

Transcript

1. 3+ years of Go in Prod Jaime Silvela @jsilvela GoMAD, 13 June 2018
2. Agenda • Intro • Codebase structure • Interfaces • Errors, Closures, Concurrency • Conclusion
3. Intro
4. My team • Insurance sector: web apps for the study of Catastrophe Models (i.e. earthquakes, flood…) • Intranet. Not many users. Complex requests • Go backend, Postgres, memcached, React, D3 • Over 30K lines of Go, 4 developers, 3 years
5. Things we do
6. Design of Go [standard languages: Java, C++] they’re also extremely good for large scale programming, and I don’t think we understand why, but we know that’s true Rob Pike Another Go at Language Design (2010)
7. Code Structure
8. Packages and sub-packages • Sub-packages don’t have semantic meaning. Just use for organization/convenience. • A package may depend on a sub-package (specialization) • A sub-package may depend on its parent package (polymorphism) • No import cycles allowed
9. Package depending on sub-package
10. Sub-package depending on package
11. Organizing your commands mycodebase/ library-directories/ cmd/ serve/ (package main) scanAlerts/ (package main) walkLibrary/ (package main) go build mycodebase/cmd/scanAlerts  scanAlerts(.exe)
12. Organizing your library code Suggestion (presentation from Ben Johnson, 2016) • Use “root” packages for domain types • Sub-packages group dependencies • Resolve dependencies in commands Similar to: Ports and Adapters a.k.a Hexagonal Architecture
13. Example: my layout mycodebase/ alerts/ core-types.go alertsdb/ db.go alertsweb/ handlers.go
14. The “domain” package core-types.go package alerts type AlertCluster struct {…} type AlertSummary struct {…} type Store interface { AlertClusters(cp Campaign) ([]AlertCluster, error) AlertSummaries() ([]AlertSummary, error) }
15. Sub-package for DB db.go package alertsdb import "mycodebase/alerts", "database/sql" … type DbStore struct { DB *sql.DB } func (s DbStore) AlertClusters(cpm alerts.Campaign) ([]alerts.AlertCluster, error) { } func (s DbStore) AlertSummaries() ([]alerts.AlertSummary, error) { }
16. Sub-package for web handlers.go package alertsweb import "mycodebase/alerts", "net/http", "encoding/json" type ClustersHandler struct { store alerts.Store } func (h ClustersHandler) ServeHTTP(w http.ResponseWriter …) { }
17. All you need is net/http
18. Decompose your ServerMux package alertsweb func MakeMux(st alerts.Store) *http.ServeMux { mux := http.NewServeMux() mux.Handle("/", homeHandler) mix.Handle("/foo", fooHandler) return mux }
19. Decompose your ServerMux package main func main() { … DB, err := sql.Open("postgres", "dbname=foo") … alertsStore := alertsdb.NewStore(DB) alertsMux := alertsweb.MakeMux(alertsStore) mux := http.NewServeMux() mux.Handle("/app/alerts/", http.StripPrefix("/app/alerts", alertsMux))
20. Interfaces
21. Go Interfaces = great OO Opinion: Go is a great language to learn OO. (Better than Smalltalk!) • No inheritance: Methods + Interfaces In Go, interfaces are satisfied implicitly. This makes them quite different from Java/C# interfaces
22. Prefer small interfaces • The bigger the interface, the weaker the abstraction. Rob Pike in Go Proverbs • Well designed interfaces are more likely to be small interfaces; the prevailing idiom is an interface contains only a single method. Dave Cheney SOLID Go Design
23. Practical example. Bad? package catmod type CatalogLister interface { Models() ([]Model, error) Perils() ([]Peril, error) Disclaimer(string) (string, error) Countries() ([]Country, error) GetLocationsAt(country string) ([]Location, error) ModelSettings() ([]ModelSetting, error }
24. Is this any better? package catmod type ModelLister interface { Models() ([]Model, error) } type PerilLister interface { Perils() ([]Peril, error) } type CatalogLister interface { ModelLister PerilLister
25. How is it used? package foobar func ProcesStuff(lister catmod.CatalogLister) { perils, err := lister.Perils() // do stuff with Peril list } • We only use one method from the interface. Can we leverage that?
26. Small interface at consumer package foobar type PerilLister interface { Perils() ([]catmod.Peril, error) } func ProcesStuff(lister PerilLister) { } • We don’t need to modify the catmod package • We can test ProcessStuff with a small mock
27. Benefits of small interfaces func CacheHandler(t time.Duration, h http.Handler) http.Handler { return http.HandlerFunc( func (w http.ResponseWriter, r *http.Request) { w.Header().Add("Cache-Control", fmt.Sprintf("max-age=%d, public, must-revalidate“, t/time.Second)) h.ServeHTTP(w, r) }) } … handler := CacheHandler(14*time.Hour, MyHandler)
28. Benefits of small interfaces type AppFields struct { Country, Peril, CATModel string } type AppHandler interface { ServeApp(http.ResponseWriter, *http.Request) (AppFields, int, error) } func WrapAppHandler(counter *prometheus.CounterVec, ah AppHandler) http.Handler { return http.HandlerFunc(func (w http.ResponseWriter, r *http.Request) { fields, statusCode, err := ah.ServeApp(h, r) // fields ==> Prometheus counter // statusCode, err ==> logging, metrics }) } • Inspired by Andrew Gerrand’s Error Handling and Go
29. Interfaces recap • If C++ and Java are about type hierarchies and the taxonomy of types, Go is about composition Rob Pike Less is exponentially more • The simple structure and natural applicability of lists are reflected in functions that are amazingly non-idiosyncratic. In Pascal the plethora of declarable data structures induces a specialization within functions that inhibits and penalizes casual cooperation Alan Perlis in Structure and Interpretation of Computer Programs
30. Errors, Closures, Concurrency
31. Closures (lambdas) • Go would be impoverished without them • Lambdas can express a domain-level concept type kernelFunc func(y1, y2, x float64) float64 • Body of defer blocks • Goroutines • Error flow • Composition: e.g. converting your HTTP handler into a cached handler
32. Closures and error flow func DeleteCatalog(recID int) error { … var err error delete := func(query string, recordID int) { if err != nil { return } _, err = DB.Exec(query, recordID) } … delete (qryEvents, recID) delete (qryCurves, recID) delete (qryVerifications, recID) delete (qryCatalog, recID) return err }
33. Concurrency • At very low cost, have extra servers in your binary, in addition to your “port 80” web server. • Profiling with pprof TCP port • Server and new port for Prometheus metrics • Handle slow web requests in background go-routines (See my talk at dotGo Paris) • Graceful shutdown (go-routine listening on a chan os.Signal for SIGTERM)
34. Conclusion
35. The bad? • Generics ... ?? • Not an issue, 90% of the time • But, if I had a penny for every DB Query() parser I’ve written… • Error handling … ?? • See: Errors are values; take advantage of closures • defer can interact strangely with errors • E.g. DB Transaction Commit / Rollback
36. After 3 years of Go • Small language: one language (cf. C++) • Code reviews are comfortable • If err != nil { … } makes code easy to understand • Concurrency easy enough for a beginner to get right • Excellent standard library. net/http is great! • Wonderful tooling: test coverage, profiling, race detector • Static binaries! • Two genius-level features: • Interfaces satisfied implicitly, plus usage conventions • Takes the pain out of static types
37. References • Rob Pike Another Go at Language Design: https://www.youtube.com/watch?v=7VcArS4Wpqk • Package naming conventions: https://blog.golang.org/package-names • Structuring Applications for Growth: https://go-talks.appspot.com/github.com/gophercon/2016-talks/BenJohnson-StructuringApplicationsForGrowth/main.slide#1 • Hexagonal Architecture: http://alistair.cockburn.us/Hexagonal+architecture • Go Proverbs: https://www.youtube.com/watch?v=PAAkCSZUG1c https://go-proverbs.github.io/ • SOLID Go Design: https://dave.cheney.net/2016/08/20/solid-go-design • Error Handling and Go: https://blog.golang.org/error-handling-and-go • Errors are values: https://blog.golang.org/errors-are-values • Handling slow requests in your Go server https://youtu.be/0qKqVSvB1G0
38. Thank you Questions?