Elena Grahovac_Best practices for cloud-native go services_Codemotion Berlin 2019

It is easy to fall in love with a new programming language, tool or architectural approach, especially as it gains in popularity. Usually, we start trying new technology by understanding a "tour" or the "quick start" guidance. However, in practice, it is not always so simple to bridge the gap between our first web server and a real production-ready application, this is especially true if this application should be prepared for cloud-native infrastructure. This talk is for those who touched Go and fell in love with it and now wonder how to achieve production readiness.

About:
Elena Grahovac: Lead TechOps Automation Engineer - N26
Elena is a Lead TechOps Automation Engineer at N26 and a co-host of the GolangShow podcast. With 11+ years of overall experience in IT, she values DevOps culture and passionate about automation, software architecture and site reliability engineering topics.

Transcript

1. Best Practices for Cloud-Native Go Services Elena Grahovac, N26 12-13 November, 2019
2. Lead TechOps Automation Engineer @N26 DevOps Enthusiast elena@grahovac.me webdeva rumyantseva Slides:
3. “Hello, World” “Production ready”
4. The principles of production readiness
5. 12 factor apps 一 Revision control 一 Dependencies 一 Config in the environment 一 Backing services as attached resources 一 Build and run stages 一 Stateless processes 一 Port binding 一 Environments as similar as possible 一 ... 12factor.net
6. Production readiness 一 Stability 一 Reliability 一 Scalability 一 Fault tolerance 一 Catastrophe-preparedness 一 Performance 一 Monitoring 一 Documentation by Susan J. Fowler
7. “S” principles 一 Security 一 Simplicity 一 Speed 一 Stability 一 Scalability “5S”
8. How to be Cloud Native? 一 Operate managed infrastructure 一 Develop isolated containerized services 一 Communicate through network 一 Follow microservice design principles 一 Use DevOps techniques
9. DevOps is not a one-way road!
10. What developers can do? 一 Structure the services properly 一 Be quality-oriented 一 Think about observability and operability 一 Shutdown gracefully
11. What developers can do? 一 Manage configuration wisely 一 Be accurate with dependencies 一 Containerize applications properly 一 Automate as much as possible
12. How to do it in Go?
13. Structuring
14. Structuring 一 Structuring is important 一 There is no one right way to structure apps 一 For inspiration: https://github.com/golang-standards/project-layout
15. Structuring in practice 一 cmd for small and simple entry points E.g. I might have cmd/myctl and cmd/myserver
16. Structuring in practice 一 internal for packages that should be available for current project only If I don’t want a package being available for import by other projects, I’ll store it under internal See also: ‘Go 1.4 Internal Packages’ - https://docs.google.com/document/d/1e8kOo3r51b2BWtTs_1uADIA5djfXhPT36s6e HVRIvaU/edit
17. Structuring in practice 一 vendor to store dependencies Yes, you might need it… still :)
18. Code quality
19. Code quality 一 Testing 一 Static code analysis 一 Profiling & tracing
20. Testing Testing deserves its own talk… Maybe even a few. 一 Why there is no “assert” in the standard library: https://golang.org/doc/faq#testing_framework 一 Learn testing: https://github.com/golang/go/wiki/LearnTesting
21. Fuzzy testing 一 My application works with a complex input 一 I validate it 一 I want to make sure that my validation is good enough 一 https://github.com/dvyukov/go-fuzz
22. Static code analysis 一 Check code style 一 Detect potential bugs 一 Scan for security problems
23. Static code analysis: how? 一 Run the most popular linters: golangci-lint https://github.com/golangci/golangci-lint 一 Fast assessment for open source projects: https://goreportcard.com
24. Static code analysis: when? 一 Local hooks (pre-commit): https://github.com/dnephin/pre-commit-golang 一 Server hooks (push, pull request) 一 Dedicated Docker image 一 Continuous Integration stage
25. Observability & Operability
26. Observability: logs 一 Log errors or strange situations! 一 Log “stages” of your application 一 Try multiple log levels 一 Try “tags” to classify your logs 一 Tip: if something goes wrong, check performance of your logger
27. Popular loggers 一 Logrus: https://github.com/sirupsen/logrus 一 Glog: https://github.com/golang/glog
28. Observability Metrics: 一 Define, measure and report business and SLA metrics 一 Business metrics might be useful as well! Traces: 一 Tracing is a technique that shows how a single request goes through your services
29. Observability in practice 一 Check this out: https://opentelemetry.io (OpenCensus + OpenTracing) 一 Not for production (yet)
30. Operability: health checks 一 “Healthy/Unhealthy”: application state, business and performance metrics 一 “Readiness”: report if application is ready to handle requests
31. Graceful shutdown 一 Recover panics 一 Don’t panic, return errors 一 Handle errors 一 Listen to OS signals 一 Provide shutdown for your functional units
32. Shutdown in practice 一 Example: main.go
33. Versioning Use linker to provide current version, commit hash or any other data to identify the version: go build -ldflags "-X main.version=put_version_here" package main import ( "fmt" ) var ( version = "unset" ) func main() { fmt.Printf("The version is: %s\n", version) }
34. Configuration
35. Configuration 一 Set configuration via env 一 Systems solution: https://github.com/kelseyhightower/envconfig 一 Keep secrets safe 一 Prepare a systems solution if you need to deal with secrets on the application level
36. Secrets
37. Secrets
38. Secrets: Solutions 一 Hashicorp Vault https://www.vaultproject.io 一 Google Cloud: KMS, HMS https://cloud.google.com/solutions/secrets-management/#tools 一 Other cloud solutions
39. Dependency management
40. Dependencies: problem definition 一 Design & quality 一 Testing 一 Activity & maintenance 一 Licenses 一 Integrity & dependencies 一 Immutability & updates
41. Dependencies checklist 一 Description 一 Documentation 一 Go Report Card 一 Issues & pull requests 一 Code coverage 一 Other reports 一 Repeat the same for dependencies of this dependency
42. GOPROXY: pros 一 Availability 一 Independency 一 Immutability 一 Archives are faster than git repos 一 Additional opportunities
43. GOPROXY: cons 一 Workflows around module management 一 Current implementations 一 Transitive dependencies still might cause problems
44. Builds 一 Compile a binary on the CI/CD node & copy it 一 Multi-staging builds
45. Dockerization
46. Dockerization 一 Mono-staging vs multi-staging 一 Testing vs building
47. The simplest “monostage” image FROM scratch ENV PORT 8080 COPY bin/myapp /myapp EXPOSE $PORT CMD ["myapp"]
48. Multi-staging builds: GOPROXY # Initial stage: download modules FROM registry/golang:1.13 AS modules ADD go.mod go.sum /m/ RUN cd /m && go mod download
49. Multi-staging builds: building phase # The second stage. Build the binary FROM registry/golang:1.13 AS builder COPY --from=modules /go/pkg /go/pkg # add a non-privileged user RUN useradd -u 10001 myapp RUN mkdir -p /go/src/github.com/rumyantseva/myapp ADD . /go/src/github.com/rumyantseva/myapp WORKDIR /go/src/github.com/rumyantseva/myapp # Build the binary with go build RUN CGO_ENABLED=0 go build \ -o bin/myapp github.com/rumyantseva/myapp/cmd/myapp
50. Multi-staging builds: running phase # The final stage. Run the binary FROM scratch ENV PORT 8080 # certificates to interact with other services COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ # don't forget /etc/passwd from previous stage COPY --from=builder /etc/passwd /etc/passwd USER myapp # and finally the binary COPY --from=builder /go/src/github.com/rumyantseva/myapp/bin/myapp /myapp EXPOSE $PORT CMD ["myapp"]
51. Multi-staging Dockerfile 一 Example: Dockerfile
52. Base image for testing FROM registry/golang:1.13 # Change here to update ENV VERSION 1.18.0 ENV CHECKSUM 0ef2c502035d5f12d6d3a30a7c4469cfcae4dd3828d15fbbfb799c8331cd51c4 # Make sure we have a fixed golangci-lint script with a chekcsum check RUN echo "${CHECKSUM} golangci-lint-${VERSION}-linux-amd64.tar.gz" > CHECKSUM # Download from Github the specified release and extract into the go/bin folder RUN curl -L "https://github.com/golangci/golangci-lint/ releases/download/v${VERSION}/golangci-lint-${VERSION}-linux-amd64.tar.gz" \ -o golangci-lint-${VERSION}-linux-amd64.tar.gz \ && shasum -a 256 -c CHECKSUM \ && tar xvzf golangci-lint-${VERSION}-linux-amd64.tar.gz \ --strip-components=1 \ -C ./bin \ golangci-lint-${VERSION}-linux-amd64/golangci-lint # Clean up RUN rm -rf CHECKSUM "golangci-lint-${VERSION}-linux-amd64.tar.gz"
53. Run linters and tests FROM registry/golang-linters RUN mkdir -p /go/src/github.com/rumyantseva/myapp ADD . /go/src/github.com/rumyantseva/myapp WORKDIR /go/src/github.com/rumyantseva/myapp # Run linters RUN golangci-lint run \ --no-config --issues-exit-code=1 \ --deadline=10m --exclude-use-default=false \ ./... # Run tests RUN go test -timeout=600s -v --race ./...
54. Linters and test from Dockerfile 一 Example: Dockerfile.testing
55. Automation
56. Repeatable actions 一 Define repeatable actions and prepare a tool to call them fast: • The actions you call when CI/CD-ing • Checkers and tests • Application building • Dealing with container images 一 GNU Make as a classic approach 一 A fancy alternative: https://github.com/go-task/task
57. Automate creation of new services 一 Prepare a typical “Cloud-Native Hello World” service 一 Define templates based on it 一 Use code generation to produce new services from the templates 一 Consider details specific for your infrastructure or project For inspiration: https://github.com/takama/caldera
58. What’s next? 一 You don’t have to agree with all the practices from this talk 一 When disagree, ask yourself why 一 Discover, try and adopt new practices 一 Document what you adopted 一 Share your experience with the community
59. elena@grahovac.me webdeva rumyantseva These slides: http://bit.ly/2X9UDRZ