Slide 1
Slide 1 text
Go at DigitalOcean
Fatih Arslan
Sr. Software Engineer @DigitalOcean – GoCon, Tokyo 2017
Slide 2
Slide 2 text
Me
• Sr. Software Engineer @Delivery Team
• Creator of vim-go
• Go contributor, author of many popular Go
packages (i.e: color, structs, etc..)
• Tool maker (i.e: gomodifytags, motion,
etc...)
• Coffee and bag geek
Slide 3
Slide 3 text
DigitalOcean
is a simple and robust cloud computing platform,
designed for developers.
Slide 4
Slide 4 text
No content
Slide 5
Slide 5 text
DigitalOcean at a glance
• 3 million users have signed up for DO
• 66 million droplets deployed (currently ~100k droplets/day)
• It's been a month since Spaces launch.
• We have 300 million objects, over 100 TB data
Slide 6
Slide 6 text
Programming languages used at DO
• Ruby, Python, Perl, JS, C++
• Much of our infrastructure has been moved to Go
• Current platforms are being ported to Go as well
Slide 7
Slide 7 text
State of Go
Slide 8
Slide 8 text
How did we start
using Go?
Slide 9
Slide 9 text
First time using Go
• First Go service was VNC proxy (by Mac Browning in March 2014)
• Gained traction when engineers attended GopherCon 2014
• Was a huge success, other services followed quickly
• metadata service, imagemgmt, metrics service, dns rewrite, etc..
Slide 10
Slide 10 text
No content
Slide 11
Slide 11 text
VNC Console
• Goroutines made it easy to duplex TCP and WebSocket connections
• Interface usage enabled end to end testing
• Go's stdlib was very powerful. Zero-downtime deploys were common, with
no user interruptions.
Slide 12
Slide 12 text
Some of the
first issues
Slide 13
Slide 13 text
Code sharing
was problematic
Slide 14
Slide 14 text
Code sharing
• Each single repo had different versions of dependencies
• No official vendor support yet
• Adding a package required to rewrite import paths
• Took weeks to update deps across repos
Slide 15
Slide 15 text
No unified
CI/CD integration
Slide 16
Slide 16 text
No internal stdlib
development
Slide 17
Slide 17 text
Monorepo?
Slide 18
Slide 18 text
No content
Slide 19
Slide 19 text
Cthulhu
Slide 20
Slide 20 text
First commit: Dec 3, 2014
Slide 21
Slide 21 text
Code structure
Slide 22
Slide 22 text
Code structure
cthulhu
├── docode
│ └── src
│ └── do
│ ├── doge
│ ├── exp
│ ├── services
│ ├── teams
│ ├── tools
│ └── vendor
└── script
Slide 23
Slide 23 text
Code structure (cont.)
• doge: DigitalOcean Go Environment. Our
internal "standard library"
• exp: experimental stuff, not CI/CD checked
• services: deprecated. Used to store services at
DO, replaced by teams/
• teams: team specific code. All DO services are
here
• tools: internal tools, cmds. Mainly for CI/CD
integration
• script: shell scripts. Non Go related code
cthulhu
├── docode
│ └── src
│ └── do
│ ├── doge
│ ├── exp
│ ├── services
│ ├── teams
│ ├── tools
│ └── vendor
└── script
Slide 24
Slide 24 text
Internal stdlib: doge
cthulhu
├── docode
│ └── src
│ └── do
│ ├── doge
│ ├── exp
│ ├── services
│ ├── teams
│ ├── tools
│ └── vendor
└── script
Slide 25
Slide 25 text
Internal stdlib: doge
• Over 30 packages
• context: Additional helper packages for Go's
context package
• dorpc: Wraps gRPC to provide DO specific
support
• httpclient: custom HTTP client tuned for our
internal infrastructure
• log: customer K/V logger that logs to our
centralized logging platform
• ...
doge
├── context
├── dorpc
├── ...
├── ...
├── httpclient
├── ...
├── ...
├── ...
├── log
├── ...
├── ...
└── version
Slide 26
Slide 26 text
Monorepo stats
Slide 27
Slide 27 text
Cthulhu overview
• 28,639 commits
• 824 branches
• 142 contributors
• 830,434 lines of DigitalOcean-authored Go code
• 2,136,373 lines of vendored Go code
Slide 28
Slide 28 text
Monoropo issues
Slide 29
Slide 29 text
Monorepo issues
• Vendoring is problematic
• What if two services depends on two different versions of the same
package?
• Large GOPATH causes slow tooling performance (goimports, guru, etc..)
• Needs good tooling and constant maintenance
• Slow building
Slide 30
Slide 30 text
Onboarding people
Slide 31
Slide 31 text
Onboarding
1. Clone repo: git clone cthulhu.git
2. Call .env.sh (sets GOPATH, PATH (GOPATH/bin))
3. Start coding!
Slide 32
Slide 32 text
Direnv (optional)
• It hooks into bash, zsh, fish, ... and automatically loads or unloads
environment variables based on the current directory
• https://github.com/direnv/direnv
• Very handy to switch GOPATH's (personal and company)
• Direnv is written in Go, compiles into a single binary and is very fast
• In cthulhu: ln -s .env.sh .envrc
Slide 33
Slide 33 text
Direnv (optional)
$ echo $GOPATH
/Users/fatih/go
$ cd cthulhu
direnv: loading .envrc
direnv: export ~GOPATH ~PATH
$ echo $GOPATH
/Users/fatih/Code/do/cthulhu/docode
Slide 34
Slide 34 text
What if you have never
programmed in Go before?
Slide 35
Slide 35 text
What if you are a new Gopher?
• Internal Go guide and documentation
• Slack #golang channel
• Go readability team
• Mentoring new developers
Slide 36
Slide 36 text
Frustrations of new engineers
• What is GOPATH?
• Vendoring packages
• How to deal with other teams?
• Finding the appropriate internal 'stdlib' is not easy
Slide 37
Slide 37 text
Dependency Management
Slide 38
Slide 38 text
Three years ago (2014)
Slide 39
Slide 39 text
Three years ago (2014)
cthulhu
├── docode
│ └── src
│ ├── doge
│ ├── services
│ └── tools
└── third_party
└── src
Slide 40
Slide 40 text
Three years ago (2014)
• Third party packages were in third_party folder
• GOPATH=${CTHULHU}/third_party:${CTHULHU}/docode
• go get automatically puts dependencies to third_party first
• No submodules are used, we rename .git to .checkout_git
• If package is not version controlled, a file "import.md" had to be added
Slide 41
Slide 41 text
Vendoring a package (2014)
$ go get github.com/fatih/structs
$ cd third_party/src/github.com/fatih/structs
$ mv .git .checkout_git
$ git add .
$ git commit -m "Vendored fatih/structs"
Slide 42
Slide 42 text
Updating a package (2014)
$ cd third_party/src/github.com/fatih/structs
# Update package to HEAD
$ mv .checkout_git .git
$ git pull origin master
$ mv .git .checkout_git
# add to monorepo
$ git add .
$ git commit -m "Vendored fatih/structs"
Slide 43
Slide 43 text
check3rdparty
Slide 44
Slide 44 text
check3rdparty
• Makes sure no .git is added
• .checkout_git is allowed
• import.md describes how to
update/vendor package if it's not
version controlled
github.com/fatih/structs
├── .git (not allowed)
├── .checkout_git (ok)
└── main.go
github.com/fatih/structs
├── import.md (ok)
└── main.go
or
Slide 45
Slide 45 text
One year ago (2016)
Slide 46
Slide 46 text
One year ago (2016)
• Go 1.6 was released in 17 February 2016, with Vendor support enabled
• We waited for 6 months after we made the switch to vendor/ folder
• Removed third_party completely and moved all packages to vendor/
• Started to use govendor as it had vendor support
Slide 47
Slide 47 text
One year ago (2016)
https://twitter.com/golang/status/700083070414643201
Slide 48
Slide 48 text
One year ago (2016)
Slide 49
Slide 49 text
Switch to vendor folder (2016)
cthulhu
└── docode
└── src
├── doge
├── services
├── tools
└── vendor
cthulhu
├── docode
│ └── src
│ ├── doge
│ ├── services
│ └── tools
└── third_party
└── src
Slide 50
Slide 50 text
Vendoring&Updating a package (now)
# vendors or updates package
$ govendor fetch github.com/fatih/structs
# add to monorepo
$ git add .
$ git commit -m "Vendored fatih/structs"
Slide 51
Slide 51 text
Added do/ prefix (2017)
cthulhu
├── docode
│ └── src
│ └── do
│ ├── doge
│ ├── services
│ ├── tools
│ └── vendor
cthulhu
├── docode
│ └── src
│ ├── doge
│ ├── services
│ ├── tools
│ └── vendor
Slide 52
Slide 52 text
Added do/ prefix (2017)
• Vendor tools don't like vendor/ being under GOPATH directly
• https://github.com/kardianos/govendor/issues/237
• Clear ownership, we can see do/ stuff belongs to us
• Allows us easily to open source stuff later when we rename do/ to do.co/
Slide 53
Slide 53 text
Issues with govendor
Slide 54
Slide 54 text
Slow on macOS
• macOS file system (HFS+ and APFS) is case insensitive
• both github.com/foo and github.com/FOO are the same
• but for Go, import paths are case sensitive!
• govendor tries to normalize it by converting all paths to lowercase
• excessive usage of strings.ToLower()
• a typical vendor takes many minutes
Slide 55
Slide 55 text
Each string operation
takes ~21 seconds
Before
Slide 56
Slide 56 text
Replace strings.ToLower() calls
Slide 57
Slide 57 text
memoize the lowered string
Slide 58
Slide 58 text
Lowered to ~5 seconds
( 400% improvement)
After
Slide 59
Slide 59 text
github.com/golang/dep ?
• We're still evaluating it
• Migrating from govendor to dep is not
easy yet
• We have an incomplete govendor.json
• Some dependencies don't exist in
public anymore
• Github Enterprise doesn't work well with
import paths (see: https://github.com/
golang/dep/issues/174)
Slide 60
Slide 60 text
CI/CD integration
Slide 61
Slide 61 text
CI/CD integration
• Drone (github.com/drone/drone) is used for our monorepo
• Runs for each branch & pull request and periodically for all DO packages
• Also used for deployment (more on this later)
• Default Go tools: gofmt, go vet and golint
• Custom DO tools: gta, buildlint, explint, etc..
• Concourse (http://concourse.ci) and GoCD (https://www.gocd.org) is used for
deployment
Slide 62
Slide 62 text
gofmt instead of goimports
• We were using a custom goimports fork
• Made sure that do/ prefix was in a block
• Was causing problems, people were using gofmt or standard goimports
• Is replaced with gofmt
• goimports is still encouraged to group block of import paths, but not
required
Slide 63
Slide 63 text
golint
• Before golint around 1500 errors were detected
• It took several weeks to fix all of them
• Benefit: we have internal godoc running with high quality documentation
Slide 64
Slide 64 text
Custom tools
•gta
•buildlint
•githubjanitor
•explint
•autoreview
Slide 65
Slide 65 text
In early 2016, CI builds took an
average of 20 minutes
Slide 66
Slide 66 text
How did we improve
CI build duration?
Slide 67
Slide 67 text
gta:
Go Test Auto
Slide 68
Slide 68 text
src
├── foo
│ └── foo.go
├── bar
│ ├── bar.go
│ └── baz
│ └── baz.go
├── qux
│ └── qux.go
└── example
└── example.go
package qux
import "fmt"
func Hello() {
fmt.Println("Hello GoCon")
}
Slide 69
Slide 69 text
src
├── foo
│ └── foo.go
├── bar
│ ├── bar.go
│ └── baz
│ └── baz.go
├── qux
│ └── qux.go
└── example
└── example.go
package foo
import "qux"
func main() {
qux.Hello()
}
Package foo
imports qux
package qux
import "fmt"
func Hello() {
fmt.Println("Hello GoCon")
}
Slide 70
Slide 70 text
$ git diff
--- a/docode/src/qux/qux.go
+++ b/docode/src/qux/qux.go
@@ -3,5 +3,5 @@ package qux
import "fmt"
func Hello() {
- fmt.Println("Hello GoCon!")
+ fmt.Println("こんにちは。GoCon!")
}
src
├── foo
│ └── foo.go
├── bar
│ ├── bar.go
│ └── baz
│ └── baz.go
├── qux
│ └── qux.go
└── example
└── example.go
Change something in Qux
Slide 71
Slide 71 text
src
├── foo
│ └── foo.go
├── bar
│ ├── bar.go
│ └── baz
│ └── baz.go
├── qux
│ └── qux.go
└── example
└── example.go
$ gta
foo
qux
Run gta
Slide 72
Slide 72 text
src
├── foo
│ └── foo.go
├── bar
│ ├── bar.go
│ └── baz
│ └── baz.go
├── qux
│ └── qux.go
└── example
└── example.go
$ go build -v ./...
foo
bar
bar/baz
qux
example
go build packages
Slide 73
Slide 73 text
src
├── foo
│ └── foo.go
├── bar
│ ├── bar.go
│ └── baz
│ └── baz.go
├── qux
│ └── qux.go
└── example
└── example.go
$ go build -v ./...
$ go build -v $(gta)
foo
qux
go build with gta
Slide 74
Slide 74 text
gta: Go Test Auto
• Average build decreased from 20 minutes to 2 – 3 minutes
• Finds differences between feature and master branch.
• Returns a list of packages that needs to be tested/built
• go tool compatible
• go build $(gta), go test $(gta), etc...
• Can be disabled with "-force-test" appended to the branch name
Slide 75
Slide 75 text
buildlint
Slide 76
Slide 76 text
buildlint
• Checks build tags
• Example: // +build linux
• Finds problems
• Disallows race:
• Positive and negative of the same tag:
• Empty build tag:
// +build !race
// +build linux,!linux
// +build
Slide 77
Slide 77 text
$ cat foo.go
package main
import "fmt"
// +build
// +build !race
// +build !linux,linux
func main() {
fmt.Println("GoCon")
}
Slide 78
Slide 78 text
$ cat foo.go
package main
import "fmt"
// +build
// +build !race
// +build !linux,linux
func main() {
fmt.Println("GoCon")
}
$ buildlint
foo.go:5:1: empty build tag comment
foo.go:6:1: found disallowed build tag "!race"
foo.go:7:1: found positive and negative tags
for "linux" in same group
Slide 79
Slide 79 text
explint
Slide 80
Slide 80 text
explint
• Makes sure exp can only be imported from
exp itself, so production code can't use it
• This is forbidden:
cthulhu
├── docode
│ └── src
│ └── do
│ ├── doge
│ ├── exp
│ ├── services
│ ├── teams
│ ├── tools
│ └── vendor
└── script
package main
import (
"do/exp/foo"
)
func main() {
foo.Foo()
}
Slide 81
Slide 81 text
Upcoming ideas
Slide 82
Slide 82 text
Upcoming ideas
• Vendorlint
• Makes sure new vendors or vendor updates are not done in the same commit as application
code changes
• Deplint
• Warns user if a vendored package is already vendored in our global top-level vendor/ folder
• i.e: src/teams/project/vendor/github.com/fatih/structs is already vendored in src/vendor/
github.com/fatih/structs
• Autovendor
• Automatically check for new updates and open a new Pull Request with changes
Slide 83
Slide 83 text
Before merging the PR
Slide 84
Slide 84 text
Before the merge
• Every PR needs to be reviewed by its peers
• Can be merged if peer gives +1 or LGTM
• Reviewers are automatically tagged (with autoreview)
Slide 85
Slide 85 text
autoreview
Slide 86
Slide 86 text
autoreview
• A tool for tagging teams for reviewing Github PR's
• Looks for OWNERS files relevant to changes in a PR
• Adds comment that tags the missing owners
• Modeled after Google's OWNERS system
Slide 87
Slide 87 text
OWNERS format
The file is line-delimited, and supports the following patterns:
@\S+ : a github team or person
\S+@ : an email address (@digitalocean.com is inferred).
#\S+ : a slack channel
Example contents of an “OWNERS” file which contains all three:
@digitalocean/storage
storage@
#storage
Slide 88
Slide 88 text
autoreview
teams
├── delivery
│ ├── OWNERS
│ └── project1
│ ├── server.go
│ ├── server_test.go
│ └── main.go
└────── project2
├── foo.go
└── main.go
If anything under
project1 or project2
changes
Slide 89
Slide 89 text
autoreview
Slide 90
Slide 90 text
Changes PR status to pending
Slide 91
Slide 91 text
Success if signoff comment is added
• LGTM, lgtm, +1
• :shipit: ( )
• :+1: ()
• :ship::it: (#)
Some signoff comments:
Slide 92
Slide 92 text
githubjanitor
Slide 93
Slide 93 text
githubjanitor
• Checks for stale PR's
• Warns after 10 days of inactivity
• Closes PR's after 15 days of inactivity.
• Resets timer if you push a commit
Slide 94
Slide 94 text
githubjanitor
Slide 95
Slide 95 text
After merging the PR
Slide 96
Slide 96 text
After the merge
• Master is tested again. Conflicts can happen again, i.e:
• CI check passed for PR foo
• Meanwhile PR bar is merged, that changes some of the code of foo
• PR foo is merged, master fails in this case
Slide 97
Slide 97 text
Deployment & Delivery
Slide 98
Slide 98 text
Deployment
• Each team is responsible for their own deployment
• Single source of truth: monorepo
• Some teams have their own project-repos, not part of monorepo
• Services written in Go are usually run inside a container
Slide 99
Slide 99 text
Binary server: gtartifacts
• Remember gta? We also have gtartifacts!
• HTTP server responsible of managing binaries
• After a merge:
• Gtartifacts builds changed binaries (via gta)
• Publishes them to our internal storage server
• Teams can fetch based on git SHA, via CLI or plain HTTP requests
Slide 100
Slide 100 text
No content
Slide 101
Slide 101 text
CLI: artifactctl
Slide 102
Slide 102 text
Using artifactctl
Slide 103
Slide 103 text
What platform to use?
• Chef
• Still being used a lot. Old services usually
• Not very fun to use it
• DOCC (Kubernetes)
• Our internal application runtime platform
• Based on top of Kubernetes
• Chef based deployments are slowly migrating to DOCC
Slide 104
Slide 104 text
DOCC (DigitalOcean Control Center)
• Platform for deploying containerized applications
• Services, not Servers
• Declarative deploys
• Deployments take seconds, instead of hours
• Provides several features out of the box
• Automatic TLS certs, alerting, metrics, persistent volumes, etc...
Slide 105
Slide 105 text
No content
Slide 106
Slide 106 text
DOCC numbers
• 850 applications
• 3000 docker containers
• 3500 deploys in last 30 days
• 15+ separate kubernetes clusters (3500+ cores, 10TB of memory)
Slide 107
Slide 107 text
How to deploy?
• Concourse or GoCD kicks in after master is merged
• Pipeline builds binary and docker-image
• Deploy to DOCC directly
• Chef
• Fetch form HTTP server and put to a "prodution/foo/bar/binname" path
• Chef is configured to pull from that path on the next run
• Chef-client runs periodically on each node
Slide 108
Slide 108 text
Upgrading Go
Slide 109
Slide 109 text
Why upgrade?
• Increased compilation speed
• Increased performance in stdlib
• Smaller Binaries
• Safer code
• Consistency with Go community
Slide 110
Slide 110 text
Why upgrade? (cont.)
• Go 1.3: http.Server.ConnState hooks
• Go 1.4: for range x {} loops
• Go 1.5: internal packages
• Go 1.6: net/http: HTTP/2 support
• Go 1.7: testing: sub-tests and sub-benchmarks, context in stdlib
• Go 1.8: more context support, HTTP/2 push
• Go 1.9: alias support (we don't use it yet), testing helper function
Slide 111
Slide 111 text
Upgrading Go
• Currently the monorepo uses Go 1.9.1
• Usually we wait days/weeks if a new version is released
• We have a standardized upgrade process
• Built a container with new Go version
• Create a "-force-test" branch with new container
• Run all unit tests, fix issues and merge in couple of weeks
Slide 112
Slide 112 text
Moving forward
• Build matrix among multiple Go versions
• Policy of trying RC releases in staging
Slide 113
Slide 113 text
Verdict
Slide 114
Slide 114 text
Verdict
• Monorepo let us solve a lot of problems, but needs constant maintenance
• Go accelerated everything at DO
• It became *the* language of our cloud
• Stdlib > Docker > Kubernetes > DOCC > Droplets > Your website !