Best Practices for Cloud-Native Go Services Elena Grahovac http://bit.ly/2OKCcRM

Lead TechOps Automation Engineer @N26 Podcast co-host @GolangShow hello@grahovac.pro webdeva rumyantseva

“Hello, World” “Production ready”

Go Best Practices (a checklist) 一 Effective Go: https://golang.org/doc/effective_go.html 一 Twelve Go Best Practices: https://talks.golang.org/2013/bestpractices.slide#2 一 Go Best Practices, Six Years in: https://peter.bourgon.org/go-best-practices-2016/#l ogging-and-instrumentation 一 Practical Go: Real world advice for writing maintainable Go programs: https://dave.cheney.net/practical-go/presentations/q con-china.html

How to be Cloud-Native? 一 Operate managed infrastructure 一 Develop isolated containerized services 一 Communicate through network 一 Follow microservice design principles 一 Use DevOps techniques

Cloud-Native Requirements 一 12 factor 一 Fault tolerance 一 Reliability 一 Maintainability 一 Operability 一 Observability 一 Security 一 Automation

Structuring in Practice

一 Flat 一 Layer-based 一 Module-based 一 Context-based 一 “Hexagonal” Kat Zień - How do you structure your Go apps: https://youtu.be/VQym87o91f8 “How do you structure your Go apps” by Kat Zień

Structuring in Practice 一 cmd for small and simple entry points If I have cmd/myctl and cmd/myserver I can simply do go get cmd/… to have myctl and myserver binaries inside $GOPATH/bin

Structuring in Practice 一 internal for packages that should be available for current project only If I don’t want a package being available for import by other projects, I’ll store it under internal See also: ‘Go 1.4 Internal Packages’ - https://docs.google.com/document/d/1e8kOo3r51b2BWtT s_1uADIA5djfXhPT36s6eHVRIvaU/edit

Structuring in Practice 一 vendor to store dependencies :) Other possible ideas (for inspiration): https://github.com/golang-standards/project-layout

Code Quality This and other wonderful gophers: https://github.com/ashleymcnamara/gophers

Code Quality Practices: 一 Testing 一 Static code analyzers 一 Profiling

Code Quality: Testing If you love the standard library and hate external dependencies: func TestDoSomething(t *testing.T) { // "want" is your expected result have, err := mypkg.DoSomething() if err != nil { t.Errorf( "%v", err) } else if !reflect.DeepEqual(have, want) { t.Errorf( "have %+v, want %+v", have, want) } }

Code Quality: Testing If you love the standard library and hate external dependencies: func TestDoSomething(t *testing.T) { // "want" is your expected result have, err := mypkg.DoSomething() if err != nil { t.Errorf("%v", err) } else if !reflect.DeepEqual(have, want) { t.Errorf("have %+v, want %+v", have, want) } }

Code Quality: Testing If you love to “assert”: https://github.com/stretchr/testify func TestDoSomething(t *testing.T) { // "want" is your expected result have, err := mypkg.DoSomething() require.NoError(t, err) assert.Equal(t, want, have) }

Code Quality: Testing Additional topics: 一 Why there is no “assert” in the standard library: https://golang.org/doc/faq#testing_framework 一 Learn testing: https://github.com/golang/go/wiki/LearnTesting 一 Advanced testing: https://about.sourcegraph.com/go/advanced-testing-in-go

Code Quality: Analysis Static code analyzers: 一 Check code style 一 Detect potential bugs 一 Scan for security problems

Code Quality: Analysis Tools: 一 Run the most popular linters: golangci-lint https://github.com/golangci/golangci-lint 一 Fast assessment for open source projects: https://goreportcard.com

Code Quality: Analysis When Develop/CI/CD-ing: 一 Local hooks (pre-commit) 一 Server hooks (push, pull request) 一 Dedicated Docker image 一 Continuous Integration stage

Code Quality: Runtime Profiler: 一 Simply import net/http/pprof to enable profiling handlers 一 Human-friendly interface: http://localhost:8080/debug/pprof 一 Tip: try github.com/google/pprof to check out new features

Code Quality: Runtime package main import ( "log" "net/http" _ "net/http/pprof" ) func main() { log.Println( http.ListenAndServe( ":8080", nil), ) } Hello, pprof!

Code Quality: Runtime Want to use custom URLs instead of debug/pprof? r := mux.NewRouter() r.HandleFunc("/my-prefix/pprof", pprof.Index) r.HandleFunc("/my-prefix/cmdline", pprof.Cmdline) r.HandleFunc("/my-prefix/profile", pprof.Profile) r.HandleFunc("/my-prefix/symbol", pprof.Symbol) r.HandleFunc("/my-prefix/trace", pprof.Trace) r.Handle( "/my-prefix/goroutine", pprof.Handler("goroutine"), ) r.Handle( "/my-prefix/heap", pprof.Handler("heap"), ) r.Handle( "/my-prefix/threadcreate", pprof.Handler("threadcreate"), ) r.Handle("/my-prefix/block",pprof.Handler("block"))

Observability & Operability This and other wonderful gophers: https://github.com/ashleymcnamara/gophers

Observability Logs: 一 Log errors or strange situations! 一 Log “stages” of your application 一 Try multiple log levels 一 Try “tags” to classify your logs 一 Tip: if something goes wrong, check performance of your logger

Observability Hello, logs! package main import ( log "github.com/sirupsen/logrus" ) func main() { log.WithFields(log.Fields{ "animal": "walrus", }).Info("A walrus appears") }

Observability Metrics: 一 Define, measure and report business and SLA metrics 一 Business metrics might be useful as well! Traces: 一 Tracing is a technique that shows how a single request goes through your service or services How to start? Check out OpenCensus: https://opencensus.io

Operability Health checks: 一 “healthy/unhealthy”: application state, business and performance metrics 一 “readiness”: report if application is ready to handle requests

Graceful Shutdown 一 Recover panics 一 Don’t panic, return errors 一 Handle errors 一 Listen to OS signals 一 Provide shutdown for your functional units

Versioning Use linker to provide a version, commit hash or any other data to identify the version: go build -ldflags "-X main.version=put_version_here" package main import ( "fmt" ) var ( version = "unset" ) func main() { fmt.Printf("The version is: %s\n", version) }

Configuration 一 Set configuration via env 一 Try github.com/kelseyhightower/envconfig as a systems solution 一 Keep secrets safe 一 Prepare a systems solution if you need to deal with secrets on the application level 一 Consider keeping configuration for the platform you use

Dependency Management

Dependency Management: Problem 一 Design & quality 一 Testing 一 Activity & maintenance 一 Licenses 一 Integrity & dependencies 一 Immutability & updates

Dependency Management: Checklist 一 Description 一 Documentation 一 Go Report Card 一 Issues & pull requests 一 Code coverage 一 Other reports 一 Repeat the same for dependencies of this dependency

Go mod 一 GO111MODULE=on go mod init 一 GO111MODULE=on go mod tidy 一 GO111MODULE=on go mod download 一 GO111MODULE=on go mod vendor Wiki: https://github.com/golang/go/wiki/Modules

GOPROXY: pros 一 Availability 一 Independency 一 Immutability 一 Archives are faster than git repos 一 Additional opportunities

GOPROXY: cons 一 Workflows around module management 一 Current implementations

Dockerization

Dockerization Simplest “monostage” image FROM scratch ENV PORT 8080 COPY bin/myapp /myapp EXPOSE $PORT CMD ["myapp"]

Multi-staging builds: go modules # Initial stage: download modules FROM artifactory/golang:1.13 ADD go.mod go.sum /m/ RUN cd /m && go mod download

Multi-staging builds: toolchain # Intermediate stage: Build the binary FROM artifactory/golang:1.13 as builder # add a non-privileged user RUN useradd -u 10001 myapp RUN mkdir -p /go/src/github.com/rumyantseva/myapp ADD . /go/src/github.com/rumyantseva/myapp WORKDIR /go/src/github.com/rumyantseva/myapp # Build the binary with go build RUN CGO_ENABLED=0 go build \ -o bin/myapp github.com/rumyantseva/myapp/cmd/myapp # Next stage …

Multi-staging builds: the binary # Final stage: Run the binary FROM scratch ENV PORT 8080 # certificates to interact with other services COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ # don't forget /etc/passwd from previous stage COPY --from=builder /etc/passwd /etc/passwd USER myapp # and finally the binary COPY --from=builder /go/src/github.com/rumyantseva/myapp/bin/myapp /myapp EXPOSE $PORT CMD ["myapp"]

Separate dockerfile for testing Example: https://github.com/rumyantseva/paris/blob/master/Dockerfile.test

Automation

Repeatable Actions 一 Define repeatable actions and prepare a tool to call them fast: • The actions you call when CI/CD-ing • Checkers and tests • Application building • Dealing with container images 一 GNU Make as a classic approach 一 A fancy alternative: https://github.com/go-task/task

Automate it! 一 Prepare a typical “Cloud-Native Hello World” service 一 Define templates based on it 一 Use code generation to produce new services from the templates 一 Consider the details specific for your infrastructure or project For inspiration: https://github.com/takama/caldera

Summary These and other wonderful gophers: https://github.com/ashleymcnamara/gophers

Summary 一 Think about structure before you start writing the application 一 Write unit tests 一 Try TDD to make your structure better and define how separated modules should interact to each other 一 Check your code not only for its style but also for potential bugs and security problems

Summary 一 Add a profiler before you released your first version to have the profile prepared as soon as you need it 一 Make sure that you don’t expose profiler externally 一 Log errors, stages and events produced by your application 一 Define, measure and report metrics 一 Try tracing

Summary 一 If your application handles lots of RPS, you don’t have to log, measure and trace every single request 一 Shutdown gracefully 一 Design configuration in advance 一 Keep version info inside the build

Summary 一 Manage dependencies 一 Think about security when containerize 一 Separate a container where you check the code and a container where you run the application 一 Define and describe repeatable actions 一 Automate

What’s Next? 一 You don’t have to agree with all the practices from this talk 一 ...But it might make sense to name a reason why you disagree 一 Discover, try and adopt new practices 一 ...Document them 一 Share your experience with the community 一 Spread the word!

hello@grahovac.pro webdeva rumyantseva These slides: http://bit.ly/2OKCcRM