Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Best practices for cloud native Go services

Best practices for cloud native Go services


Iskander (Alex) Sharipov

October 13, 2019


  1. Best Practices for Cloud-Native Go Services Elena Grahovac http://bit.ly/2OKCcRM

  2. Lead TechOps Automation Engineer @N26 Podcast co-host @GolangShow hello@grahovac.pro webdeva

  3. “Hello, World” “Production ready”

  4. Go Best Practices (a checklist) 一 Effective Go: https://golang.org/doc/effective_go.html 一

    Twelve Go Best Practices: https://talks.golang.org/2013/bestpractices.slide#2 一 Go Best Practices, Six Years in: https://peter.bourgon.org/go-best-practices-2016/#l ogging-and-instrumentation 一 Practical Go: Real world advice for writing maintainable Go programs: https://dave.cheney.net/practical-go/presentations/q con-china.html
  5. How to be Cloud-Native? 一 Operate managed infrastructure 一 Develop

    isolated containerized services 一 Communicate through network 一 Follow microservice design principles 一 Use DevOps techniques
  6. Cloud-Native Requirements 一 12 factor 一 Fault tolerance 一 Reliability

    一 Maintainability 一 Operability 一 Observability 一 Security 一 Automation
  7. Structuring in Practice

  8. 一 Flat 一 Layer-based 一 Module-based 一 Context-based 一 “Hexagonal”

    Kat Zień - How do you structure your Go apps: https://youtu.be/VQym87o91f8 “How do you structure your Go apps” by Kat Zień
  9. Structuring in Practice 一 cmd for small and simple entry

    points If I have cmd/myctl and cmd/myserver I can simply do go get cmd/… to have myctl and myserver binaries inside $GOPATH/bin
  10. Structuring in Practice 一 internal for packages that should be

    available for current project only If I don’t want a package being available for import by other projects, I’ll store it under internal See also: ‘Go 1.4 Internal Packages’ - https://docs.google.com/document/d/1e8kOo3r51b2BWtT s_1uADIA5djfXhPT36s6eHVRIvaU/edit
  11. Structuring in Practice 一 vendor to store dependencies :) Other

    possible ideas (for inspiration): https://github.com/golang-standards/project-layout
  12. Code Quality This and other wonderful gophers: https://github.com/ashleymcnamara/gophers

  13. Code Quality Practices: 一 Testing 一 Static code analyzers 一

  14. Code Quality: Testing If you love the standard library and

    hate external dependencies: func TestDoSomething(t *testing.T) { // "want" is your expected result have, err := mypkg.DoSomething() if err != nil { t.Errorf( "%v", err) } else if !reflect.DeepEqual(have, want) { t.Errorf( "have %+v, want %+v", have, want) } }
  15. Code Quality: Testing If you love the standard library and

    hate external dependencies: func TestDoSomething(t *testing.T) { // "want" is your expected result have, err := mypkg.DoSomething() if err != nil { t.Errorf("%v", err) } else if !reflect.DeepEqual(have, want) { t.Errorf("have %+v, want %+v", have, want) } }
  16. Code Quality: Testing If you love to “assert”: https://github.com/stretchr/testify func

    TestDoSomething(t *testing.T) { // "want" is your expected result have, err := mypkg.DoSomething() require.NoError(t, err) assert.Equal(t, want, have) }
  17. Code Quality: Testing Additional topics: 一 Why there is no

    “assert” in the standard library: https://golang.org/doc/faq#testing_framework 一 Learn testing: https://github.com/golang/go/wiki/LearnTesting 一 Advanced testing: https://about.sourcegraph.com/go/advanced-testing-in-go
  18. Code Quality: Analysis Static code analyzers: 一 Check code style

    一 Detect potential bugs 一 Scan for security problems
  19. Code Quality: Analysis Tools: 一 Run the most popular linters:

    golangci-lint https://github.com/golangci/golangci-lint 一 Fast assessment for open source projects: https://goreportcard.com
  20. Code Quality: Analysis When Develop/CI/CD-ing: 一 Local hooks (pre-commit) 一

    Server hooks (push, pull request) 一 Dedicated Docker image 一 Continuous Integration stage
  21. Code Quality: Runtime Profiler: 一 Simply import net/http/pprof to enable

    profiling handlers 一 Human-friendly interface: http://localhost:8080/debug/pprof 一 Tip: try github.com/google/pprof to check out new features
  22. Code Quality: Runtime package main import ( "log" "net/http" _

    "net/http/pprof" ) func main() { log.Println( http.ListenAndServe( ":8080", nil), ) } Hello, pprof!
  23. Code Quality: Runtime Want to use custom URLs instead of

    debug/pprof? r := mux.NewRouter() r.HandleFunc("/my-prefix/pprof", pprof.Index) r.HandleFunc("/my-prefix/cmdline", pprof.Cmdline) r.HandleFunc("/my-prefix/profile", pprof.Profile) r.HandleFunc("/my-prefix/symbol", pprof.Symbol) r.HandleFunc("/my-prefix/trace", pprof.Trace) r.Handle( "/my-prefix/goroutine", pprof.Handler("goroutine"), ) r.Handle( "/my-prefix/heap", pprof.Handler("heap"), ) r.Handle( "/my-prefix/threadcreate", pprof.Handler("threadcreate"), ) r.Handle("/my-prefix/block",pprof.Handler("block"))
  24. Observability & Operability This and other wonderful gophers: https://github.com/ashleymcnamara/gophers

  25. Observability Logs: 一 Log errors or strange situations! 一 Log

    “stages” of your application 一 Try multiple log levels 一 Try “tags” to classify your logs 一 Tip: if something goes wrong, check performance of your logger
  26. Observability Hello, logs! package main import ( log "github.com/sirupsen/logrus" )

    func main() { log.WithFields(log.Fields{ "animal": "walrus", }).Info("A walrus appears") }
  27. Observability Metrics: 一 Define, measure and report business and SLA

    metrics 一 Business metrics might be useful as well! Traces: 一 Tracing is a technique that shows how a single request goes through your service or services How to start? Check out OpenCensus: https://opencensus.io
  28. Operability Health checks: 一 “healthy/unhealthy”: application state, business and performance

    metrics 一 “readiness”: report if application is ready to handle requests
  29. Graceful Shutdown 一 Recover panics 一 Don’t panic, return errors

    一 Handle errors 一 Listen to OS signals 一 Provide shutdown for your functional units
  30. Versioning Use linker to provide a version, commit hash or

    any other data to identify the version: go build -ldflags "-X main.version=put_version_here" package main import ( "fmt" ) var ( version = "unset" ) func main() { fmt.Printf("The version is: %s\n", version) }
  31. Configuration 一 Set configuration via env 一 Try github.com/kelseyhightower/envconfig as

    a systems solution 一 Keep secrets safe 一 Prepare a systems solution if you need to deal with secrets on the application level 一 Consider keeping configuration for the platform you use
  32. Dependency Management

  33. Dependency Management: Problem 一 Design & quality 一 Testing 一

    Activity & maintenance 一 Licenses 一 Integrity & dependencies 一 Immutability & updates
  34. Dependency Management: Checklist 一 Description 一 Documentation 一 Go Report

    Card 一 Issues & pull requests 一 Code coverage 一 Other reports 一 Repeat the same for dependencies of this dependency
  35. Go mod 一 GO111MODULE=on go mod init 一 GO111MODULE=on go

    mod tidy 一 GO111MODULE=on go mod download 一 GO111MODULE=on go mod vendor Wiki: https://github.com/golang/go/wiki/Modules
  36. GOPROXY: pros 一 Availability 一 Independency 一 Immutability 一 Archives

    are faster than git repos 一 Additional opportunities
  37. GOPROXY: cons 一 Workflows around module management 一 Current implementations

  38. Dockerization

  39. Dockerization Simplest “monostage” image FROM scratch ENV PORT 8080 COPY

    bin/myapp /myapp EXPOSE $PORT CMD ["myapp"]
  40. Multi-staging builds: go modules # Initial stage: download modules FROM

    artifactory/golang:1.13 ADD go.mod go.sum /m/ RUN cd /m && go mod download
  41. Multi-staging builds: toolchain # Intermediate stage: Build the binary FROM

    artifactory/golang:1.13 as builder # add a non-privileged user RUN useradd -u 10001 myapp RUN mkdir -p /go/src/github.com/rumyantseva/myapp ADD . /go/src/github.com/rumyantseva/myapp WORKDIR /go/src/github.com/rumyantseva/myapp # Build the binary with go build RUN CGO_ENABLED=0 go build \ -o bin/myapp github.com/rumyantseva/myapp/cmd/myapp # Next stage …
  42. Multi-staging builds: the binary # Final stage: Run the binary

    FROM scratch ENV PORT 8080 # certificates to interact with other services COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ # don't forget /etc/passwd from previous stage COPY --from=builder /etc/passwd /etc/passwd USER myapp # and finally the binary COPY --from=builder /go/src/github.com/rumyantseva/myapp/bin/myapp /myapp EXPOSE $PORT CMD ["myapp"]
  43. Separate dockerfile for testing Example: https://github.com/rumyantseva/paris/blob/master/Dockerfile.test

  44. Automation

  45. Repeatable Actions 一 Define repeatable actions and prepare a tool

    to call them fast: • The actions you call when CI/CD-ing • Checkers and tests • Application building • Dealing with container images 一 GNU Make as a classic approach 一 A fancy alternative: https://github.com/go-task/task
  46. Automate it! 一 Prepare a typical “Cloud-Native Hello World” service

    一 Define templates based on it 一 Use code generation to produce new services from the templates 一 Consider the details specific for your infrastructure or project For inspiration: https://github.com/takama/caldera
  47. Summary These and other wonderful gophers: https://github.com/ashleymcnamara/gophers

  48. Summary 一 Think about structure before you start writing the

    application 一 Write unit tests 一 Try TDD to make your structure better and define how separated modules should interact to each other 一 Check your code not only for its style but also for potential bugs and security problems
  49. Summary 一 Add a profiler before you released your first

    version to have the profile prepared as soon as you need it 一 Make sure that you don’t expose profiler externally 一 Log errors, stages and events produced by your application 一 Define, measure and report metrics 一 Try tracing
  50. Summary 一 If your application handles lots of RPS, you

    don’t have to log, measure and trace every single request 一 Shutdown gracefully 一 Design configuration in advance 一 Keep version info inside the build
  51. Summary 一 Manage dependencies 一 Think about security when containerize

    一 Separate a container where you check the code and a container where you run the application 一 Define and describe repeatable actions 一 Automate
  52. What’s Next? 一 You don’t have to agree with all

    the practices from this talk 一 ...But it might make sense to name a reason why you disagree 一 Discover, try and adopt new practices 一 ...Document them 一 Share your experience with the community 一 Spread the word!
  53. hello@grahovac.pro webdeva rumyantseva These slides: http://bit.ly/2OKCcRM