Me • Sr. Software Engineer @Delivery Team • Creator of vim-go • Go contributor, author of many popular Go packages (i.e: color, structs, etc..) • Tool maker (i.e: gomodifytags, motion, etc...) • Coffee and bag geek
DigitalOcean at a glance • 3 million users have signed up for DO • 66 million droplets deployed (currently ~100k droplets/day) • It's been a month since Spaces launch. • We have 300 million objects, over 100 TB data
Programming languages used at DO • Ruby, Python, Perl, JS, C++ • Much of our infrastructure has been moved to Go • Current platforms are being ported to Go as well
First time using Go • First Go service was VNC proxy (by Mac Browning in March 2014) • Gained traction when engineers attended GopherCon 2014 • Was a huge success, other services followed quickly • metadata service, imagemgmt, metrics service, dns rewrite, etc..
VNC Console • Goroutines made it easy to duplex TCP and WebSocket connections • Interface usage enabled end to end testing • Go's stdlib was very powerful. Zero-downtime deploys were common, with no user interruptions.
Code sharing • Each single repo had different versions of dependencies • No official vendor support yet • Adding a package required to rewrite import paths • Took weeks to update deps across repos
Code structure (cont.) • doge: DigitalOcean Go Environment. Our internal "standard library" • exp: experimental stuff, not CI/CD checked • services: deprecated. Used to store services at DO, replaced by teams/ • teams: team specific code. All DO services are here • tools: internal tools, cmds. Mainly for CI/CD integration • script: shell scripts. Non Go related code cthulhu ├── docode │ └── src │ └── do │ ├── doge │ ├── exp │ ├── services │ ├── teams │ ├── tools │ └── vendor └── script
Monorepo issues • Vendoring is problematic • What if two services depends on two different versions of the same package? • Large GOPATH causes slow tooling performance (goimports, guru, etc..) • Needs good tooling and constant maintenance • Slow building
Direnv (optional) • It hooks into bash, zsh, fish, ... and automatically loads or unloads environment variables based on the current directory • https://github.com/direnv/direnv • Very handy to switch GOPATH's (personal and company) • Direnv is written in Go, compiles into a single binary and is very fast • In cthulhu: ln -s .env.sh .envrc
Frustrations of new engineers • What is GOPATH? • Vendoring packages • How to deal with other teams? • Finding the appropriate internal 'stdlib' is not easy
Three years ago (2014) • Third party packages were in third_party folder • GOPATH=${CTHULHU}/third_party:${CTHULHU}/docode • go get automatically puts dependencies to third_party first • No submodules are used, we rename .git to .checkout_git • If package is not version controlled, a file "import.md" had to be added
check3rdparty • Makes sure no .git is added • .checkout_git is allowed • import.md describes how to update/vendor package if it's not version controlled github.com/fatih/structs ├── .git (not allowed) ├── .checkout_git (ok) └── main.go github.com/fatih/structs ├── import.md (ok) └── main.go or
One year ago (2016) • Go 1.6 was released in 17 February 2016, with Vendor support enabled • We waited for 6 months after we made the switch to vendor/ folder • Removed third_party completely and moved all packages to vendor/ • Started to use govendor as it had vendor support
Added do/ prefix (2017) • Vendor tools don't like vendor/ being under GOPATH directly • https://github.com/kardianos/govendor/issues/237 • Clear ownership, we can see do/ stuff belongs to us • Allows us easily to open source stuff later when we rename do/ to do.co/
Slow on macOS • macOS file system (HFS+ and APFS) is case insensitive • both github.com/foo and github.com/FOO are the same • but for Go, import paths are case sensitive! • govendor tries to normalize it by converting all paths to lowercase • excessive usage of strings.ToLower() • a typical vendor takes many minutes
github.com/golang/dep ? • We're still evaluating it • Migrating from govendor to dep is not easy yet • We have an incomplete govendor.json • Some dependencies don't exist in public anymore • Github Enterprise doesn't work well with import paths (see: https://github.com/ golang/dep/issues/174)
CI/CD integration • Drone (github.com/drone/drone) is used for our monorepo • Runs for each branch & pull request and periodically for all DO packages • Also used for deployment (more on this later) • Default Go tools: gofmt, go vet and golint • Custom DO tools: gta, buildlint, explint, etc.. • Concourse (http://concourse.ci) and GoCD (https://www.gocd.org) is used for deployment
gofmt instead of goimports • We were using a custom goimports fork • Made sure that do/ prefix was in a block • Was causing problems, people were using gofmt or standard goimports • Is replaced with gofmt • goimports is still encouraged to group block of import paths, but not required
golint • Before golint around 1500 errors were detected • It took several weeks to fix all of them • Benefit: we have internal godoc running with high quality documentation
gta: Go Test Auto • Average build decreased from 20 minutes to 2 – 3 minutes • Finds differences between feature and master branch. • Returns a list of packages that needs to be tested/built • go tool compatible • go build $(gta), go test $(gta), etc... • Can be disabled with "-force-test" appended to the branch name
$ cat foo.go package main import "fmt" // +build // +build !race // +build !linux,linux func main() { fmt.Println("GoCon") } $ buildlint foo.go:5:1: empty build tag comment foo.go:6:1: found disallowed build tag "!race" foo.go:7:1: found positive and negative tags for "linux" in same group
explint • Makes sure exp can only be imported from exp itself, so production code can't use it • This is forbidden: cthulhu ├── docode │ └── src │ └── do │ ├── doge │ ├── exp │ ├── services │ ├── teams │ ├── tools │ └── vendor └── script package main import ( "do/exp/foo" ) func main() { foo.Foo() }
Upcoming ideas • Vendorlint • Makes sure new vendors or vendor updates are not done in the same commit as application code changes • Deplint • Warns user if a vendored package is already vendored in our global top-level vendor/ folder • i.e: src/teams/project/vendor/github.com/fatih/structs is already vendored in src/vendor/ github.com/fatih/structs • Autovendor • Automatically check for new updates and open a new Pull Request with changes
Before the merge • Every PR needs to be reviewed by its peers • Can be merged if peer gives +1 or LGTM • Reviewers are automatically tagged (with autoreview)
autoreview • A tool for tagging teams for reviewing Github PR's • Looks for OWNERS files relevant to changes in a PR • Adds comment that tags the missing owners • Modeled after Google's OWNERS system
OWNERS format The file is line-delimited, and supports the following patterns: @\S+ : a github team or person \S+@ : an email address (@digitalocean.com is inferred). #\S+ : a slack channel Example contents of an “OWNERS” file which contains all three: @digitalocean/storage storage@ #storage
githubjanitor • Checks for stale PR's • Warns after 10 days of inactivity • Closes PR's after 15 days of inactivity. • Resets timer if you push a commit
After the merge • Master is tested again. Conflicts can happen again, i.e: • CI check passed for PR foo • Meanwhile PR bar is merged, that changes some of the code of foo • PR foo is merged, master fails in this case
Deployment • Each team is responsible for their own deployment • Single source of truth: monorepo • Some teams have their own project-repos, not part of monorepo • Services written in Go are usually run inside a container
Binary server: gtartifacts • Remember gta? We also have gtartifacts! • HTTP server responsible of managing binaries • After a merge: • Gtartifacts builds changed binaries (via gta) • Publishes them to our internal storage server • Teams can fetch based on git SHA, via CLI or plain HTTP requests
What platform to use? • Chef • Still being used a lot. Old services usually • Not very fun to use it • DOCC (Kubernetes) • Our internal application runtime platform • Based on top of Kubernetes • Chef based deployments are slowly migrating to DOCC
DOCC (DigitalOcean Control Center) • Platform for deploying containerized applications • Services, not Servers • Declarative deploys • Deployments take seconds, instead of hours • Provides several features out of the box • Automatic TLS certs, alerting, metrics, persistent volumes, etc...
How to deploy? • Concourse or GoCD kicks in after master is merged • Pipeline builds binary and docker-image • Deploy to DOCC directly • Chef • Fetch form HTTP server and put to a "prodution/foo/bar/binname" path • Chef is configured to pull from that path on the next run • Chef-client runs periodically on each node
Why upgrade? (cont.) • Go 1.3: http.Server.ConnState hooks • Go 1.4: for range x {} loops • Go 1.5: internal packages • Go 1.6: net/http: HTTP/2 support • Go 1.7: testing: sub-tests and sub-benchmarks, context in stdlib • Go 1.8: more context support, HTTP/2 push • Go 1.9: alias support (we don't use it yet), testing helper function
Upgrading Go • Currently the monorepo uses Go 1.9.1 • Usually we wait days/weeks if a new version is released • We have a standardized upgrade process • Built a container with new Go version • Create a "-force-test" branch with new container • Run all unit tests, fix issues and merge in couple of weeks
Verdict • Monorepo let us solve a lot of problems, but needs constant maintenance • Go accelerated everything at DO • It became *the* language of our cloud • Stdlib > Docker > Kubernetes > DOCC > Droplets > Your website !
farslan
datarian
sugawani
chumajin
sinsoku
ikumatadokoro
kenchan
masaaki_k
toranoana
poteboy
orimanabu
okamototakuyasr2
shkitayama
schacon
shpigford
tammyeverts
pedronauck
nonsquared
trallard
kevinliebholz
jrom
revolveconf
brianwarren
phodgson
mthomps