Go at DigitalOcean

B1019ca5714cf8e9951868d6bc517827?s=47 Fatih Arslan
November 05, 2017

Go at DigitalOcean

How do we use Go at DigitalOcean

B1019ca5714cf8e9951868d6bc517827?s=128

Fatih Arslan

November 05, 2017
Tweet

Transcript

  1. 2.

    Me • Sr. Software Engineer @Delivery Team • Creator of

    vim-go • Go contributor, author of many popular Go packages (i.e: color, structs, etc..) • Tool maker (i.e: gomodifytags, motion, etc...) • Coffee and bag geek
  2. 4.
  3. 5.

    DigitalOcean at a glance • 3 million users have signed

    up for DO • 66 million droplets deployed (currently ~100k droplets/day) • It's been a month since Spaces launch. • We have 300 million objects, over 100 TB data
  4. 6.

    Programming languages used at DO • Ruby, Python, Perl, JS,

    C++ • Much of our infrastructure has been moved to Go • Current platforms are being ported to Go as well
  5. 9.

    First time using Go • First Go service was VNC

    proxy (by Mac Browning in March 2014) • Gained traction when engineers attended GopherCon 2014 • Was a huge success, other services followed quickly • metadata service, imagemgmt, metrics service, dns rewrite, etc..
  6. 10.
  7. 11.

    VNC Console • Goroutines made it easy to duplex TCP

    and WebSocket connections • Interface usage enabled end to end testing • Go's stdlib was very powerful. Zero-downtime deploys were common, with no user interruptions.
  8. 14.

    Code sharing • Each single repo had different versions of

    dependencies • No official vendor support yet • Adding a package required to rewrite import paths • Took weeks to update deps across repos
  9. 17.
  10. 18.
  11. 19.
  12. 22.

    Code structure cthulhu ├── docode │ └── src │ └──

    do │ ├── doge │ ├── exp │ ├── services │ ├── teams │ ├── tools │ └── vendor └── script
  13. 23.

    Code structure (cont.) • doge: DigitalOcean Go Environment. Our internal

    "standard library" • exp: experimental stuff, not CI/CD checked • services: deprecated. Used to store services at DO, replaced by teams/ • teams: team specific code. All DO services are here • tools: internal tools, cmds. Mainly for CI/CD integration • script: shell scripts. Non Go related code cthulhu ├── docode │ └── src │ └── do │ ├── doge │ ├── exp │ ├── services │ ├── teams │ ├── tools │ └── vendor └── script
  14. 24.

    Internal stdlib: doge cthulhu ├── docode │ └── src │

    └── do │ ├── doge │ ├── exp │ ├── services │ ├── teams │ ├── tools │ └── vendor └── script
  15. 25.

    Internal stdlib: doge • Over 30 packages • context: Additional

    helper packages for Go's context package • dorpc: Wraps gRPC to provide DO specific support • httpclient: custom HTTP client tuned for our internal infrastructure • log: customer K/V logger that logs to our centralized logging platform • ... doge ├── context ├── dorpc ├── ... ├── ... ├── httpclient ├── ... ├── ... ├── ... ├── log ├── ... ├── ... └── version
  16. 27.

    Cthulhu overview • 28,639 commits • 824 branches • 142

    contributors • 830,434 lines of DigitalOcean-authored Go code • 2,136,373 lines of vendored Go code
  17. 29.

    Monorepo issues • Vendoring is problematic • What if two

    services depends on two different versions of the same package? • Large GOPATH causes slow tooling performance (goimports, guru, etc..) • Needs good tooling and constant maintenance • Slow building
  18. 31.

    Onboarding 1. Clone repo: git clone cthulhu.git 2. Call .env.sh

    (sets GOPATH, PATH (GOPATH/bin)) 3. Start coding!
  19. 32.

    Direnv (optional) • It hooks into bash, zsh, fish, ...

    and automatically loads or unloads environment variables based on the current directory • https://github.com/direnv/direnv • Very handy to switch GOPATH's (personal and company) • Direnv is written in Go, compiles into a single binary and is very fast • In cthulhu: ln -s .env.sh .envrc
  20. 33.

    Direnv (optional) $ echo $GOPATH /Users/fatih/go $ cd cthulhu direnv:

    loading .envrc direnv: export ~GOPATH ~PATH $ echo $GOPATH /Users/fatih/Code/do/cthulhu/docode
  21. 35.

    What if you are a new Gopher? • Internal Go

    guide and documentation • Slack #golang channel • Go readability team • Mentoring new developers
  22. 36.

    Frustrations of new engineers • What is GOPATH? • Vendoring

    packages • How to deal with other teams? • Finding the appropriate internal 'stdlib' is not easy
  23. 39.

    Three years ago (2014) cthulhu ├── docode │ └── src

    │ ├── doge │ ├── services │ └── tools └── third_party └── src
  24. 40.

    Three years ago (2014) • Third party packages were in

    third_party folder • GOPATH=${CTHULHU}/third_party:${CTHULHU}/docode • go get automatically puts dependencies to third_party first • No submodules are used, we rename .git to .checkout_git • If package is not version controlled, a file "import.md" had to be added
  25. 41.

    Vendoring a package (2014) $ go get github.com/fatih/structs $ cd

    third_party/src/github.com/fatih/structs $ mv .git .checkout_git $ git add . $ git commit -m "Vendored fatih/structs"
  26. 42.

    Updating a package (2014) $ cd third_party/src/github.com/fatih/structs # Update package

    to HEAD $ mv .checkout_git .git $ git pull origin master $ mv .git .checkout_git # add to monorepo $ git add . $ git commit -m "Vendored fatih/structs"
  27. 44.

    check3rdparty • Makes sure no .git is added • .checkout_git

    is allowed • import.md describes how to update/vendor package if it's not version controlled github.com/fatih/structs ├── .git (not allowed) ├── .checkout_git (ok) └── main.go github.com/fatih/structs ├── import.md (ok) └── main.go or
  28. 46.

    One year ago (2016) • Go 1.6 was released in

    17 February 2016, with Vendor support enabled • We waited for 6 months after we made the switch to vendor/ folder • Removed third_party completely and moved all packages to vendor/ • Started to use govendor as it had vendor support
  29. 49.

    Switch to vendor folder (2016) cthulhu └── docode └── src

    ├── doge ├── services ├── tools └── vendor cthulhu ├── docode │ └── src │ ├── doge │ ├── services │ └── tools └── third_party └── src
  30. 50.

    Vendoring&Updating a package (now) # vendors or updates package $

    govendor fetch github.com/fatih/structs # add to monorepo $ git add . $ git commit -m "Vendored fatih/structs"
  31. 51.

    Added do/ prefix (2017) cthulhu ├── docode │ └── src

    │ └── do │ ├── doge │ ├── services │ ├── tools │ └── vendor cthulhu ├── docode │ └── src │ ├── doge │ ├── services │ ├── tools │ └── vendor
  32. 52.

    Added do/ prefix (2017) • Vendor tools don't like vendor/

    being under GOPATH directly • https://github.com/kardianos/govendor/issues/237 • Clear ownership, we can see do/ stuff belongs to us • Allows us easily to open source stuff later when we rename do/ to do.co/
  33. 54.

    Slow on macOS • macOS file system (HFS+ and APFS)

    is case insensitive • both github.com/foo and github.com/FOO are the same • but for Go, import paths are case sensitive! • govendor tries to normalize it by converting all paths to lowercase • excessive usage of strings.ToLower() • a typical vendor takes many minutes
  34. 59.

    github.com/golang/dep ? • We're still evaluating it • Migrating from

    govendor to dep is not easy yet • We have an incomplete govendor.json • Some dependencies don't exist in public anymore • Github Enterprise doesn't work well with import paths (see: https://github.com/ golang/dep/issues/174)
  35. 61.

    CI/CD integration • Drone (github.com/drone/drone) is used for our monorepo

    • Runs for each branch & pull request and periodically for all DO packages • Also used for deployment (more on this later) • Default Go tools: gofmt, go vet and golint • Custom DO tools: gta, buildlint, explint, etc.. • Concourse (http://concourse.ci) and GoCD (https://www.gocd.org) is used for deployment
  36. 62.

    gofmt instead of goimports • We were using a custom

    goimports fork • Made sure that do/ prefix was in a block • Was causing problems, people were using gofmt or standard goimports • Is replaced with gofmt • goimports is still encouraged to group block of import paths, but not required
  37. 63.

    golint • Before golint around 1500 errors were detected •

    It took several weeks to fix all of them • Benefit: we have internal godoc running with high quality documentation
  38. 68.

    src ├── foo │ └── foo.go ├── bar │ ├──

    bar.go │ └── baz │ └── baz.go ├── qux │ └── qux.go └── example └── example.go package qux import "fmt" func Hello() { fmt.Println("Hello GoCon") }
  39. 69.

    src ├── foo │ └── foo.go ├── bar │ ├──

    bar.go │ └── baz │ └── baz.go ├── qux │ └── qux.go └── example └── example.go package foo import "qux" func main() { qux.Hello() } Package foo imports qux package qux import "fmt" func Hello() { fmt.Println("Hello GoCon") }
  40. 70.

    $ git diff --- a/docode/src/qux/qux.go +++ b/docode/src/qux/qux.go @@ -3,5 +3,5

    @@ package qux import "fmt" func Hello() { - fmt.Println("Hello GoCon!") + fmt.Println("こんにちは。GoCon!") } src ├── foo │ └── foo.go ├── bar │ ├── bar.go │ └── baz │ └── baz.go ├── qux │ └── qux.go └── example └── example.go Change something in Qux
  41. 71.

    src ├── foo │ └── foo.go ├── bar │ ├──

    bar.go │ └── baz │ └── baz.go ├── qux │ └── qux.go └── example └── example.go $ gta foo qux Run gta
  42. 72.

    src ├── foo │ └── foo.go ├── bar │ ├──

    bar.go │ └── baz │ └── baz.go ├── qux │ └── qux.go └── example └── example.go $ go build -v ./... foo bar bar/baz qux example go build packages
  43. 73.

    src ├── foo │ └── foo.go ├── bar │ ├──

    bar.go │ └── baz │ └── baz.go ├── qux │ └── qux.go └── example └── example.go $ go build -v ./... $ go build -v $(gta) foo qux go build with gta
  44. 74.

    gta: Go Test Auto • Average build decreased from 20

    minutes to 2 – 3 minutes • Finds differences between feature and master branch. • Returns a list of packages that needs to be tested/built • go tool compatible • go build $(gta), go test $(gta), etc... • Can be disabled with "-force-test" appended to the branch name
  45. 75.
  46. 76.

    buildlint • Checks build tags • Example: // +build linux

    • Finds problems • Disallows race: • Positive and negative of the same tag: • Empty build tag: // +build !race // +build linux,!linux // +build
  47. 77.

    $ cat foo.go package main import "fmt" // +build //

    +build !race // +build !linux,linux func main() { fmt.Println("GoCon") }
  48. 78.

    $ cat foo.go package main import "fmt" // +build //

    +build !race // +build !linux,linux func main() { fmt.Println("GoCon") } $ buildlint foo.go:5:1: empty build tag comment foo.go:6:1: found disallowed build tag "!race" foo.go:7:1: found positive and negative tags for "linux" in same group
  49. 79.
  50. 80.

    explint • Makes sure exp can only be imported from

    exp itself, so production code can't use it • This is forbidden: cthulhu ├── docode │ └── src │ └── do │ ├── doge │ ├── exp │ ├── services │ ├── teams │ ├── tools │ └── vendor └── script package main import ( "do/exp/foo" ) func main() { foo.Foo() }
  51. 82.

    Upcoming ideas • Vendorlint • Makes sure new vendors or

    vendor updates are not done in the same commit as application code changes • Deplint • Warns user if a vendored package is already vendored in our global top-level vendor/ folder • i.e: src/teams/project/vendor/github.com/fatih/structs is already vendored in src/vendor/ github.com/fatih/structs • Autovendor • Automatically check for new updates and open a new Pull Request with changes
  52. 84.

    Before the merge • Every PR needs to be reviewed

    by its peers • Can be merged if peer gives +1 or LGTM • Reviewers are automatically tagged (with autoreview)
  53. 86.

    autoreview • A tool for tagging teams for reviewing Github

    PR's • Looks for OWNERS files relevant to changes in a PR • Adds comment that tags the missing owners • Modeled after Google's OWNERS system
  54. 87.

    OWNERS format The file is line-delimited, and supports the following

    patterns: @\S+ : a github team or person \S+@ : an email address (@digitalocean.com is inferred). #\S+ : a slack channel
 Example contents of an “OWNERS” file which contains all three: @digitalocean/storage storage@ #storage
  55. 88.

    autoreview teams ├── delivery │ ├── OWNERS │ └── project1

    │ ├── server.go │ ├── server_test.go │ └── main.go └────── project2 ├── foo.go └── main.go If anything under project1 or project2 changes
  56. 91.

    Success if signoff comment is added • LGTM, lgtm, +1

    • :shipit: ( ) • :+1: () • :ship::it: (#) Some signoff comments:
  57. 93.

    githubjanitor • Checks for stale PR's • Warns after 10

    days of inactivity • Closes PR's after 15 days of inactivity. • Resets timer if you push a commit
  58. 96.

    After the merge • Master is tested again. Conflicts can

    happen again, i.e: • CI check passed for PR foo • Meanwhile PR bar is merged, that changes some of the code of foo • PR foo is merged, master fails in this case
  59. 98.

    Deployment • Each team is responsible for their own deployment

    • Single source of truth: monorepo • Some teams have their own project-repos, not part of monorepo • Services written in Go are usually run inside a container
  60. 99.

    Binary server: gtartifacts • Remember gta? We also have gtartifacts!

    • HTTP server responsible of managing binaries • After a merge: • Gtartifacts builds changed binaries (via gta) • Publishes them to our internal storage server • Teams can fetch based on git SHA, via CLI or plain HTTP requests
  61. 100.
  62. 103.

    What platform to use? • Chef • Still being used

    a lot. Old services usually • Not very fun to use it • DOCC (Kubernetes) • Our internal application runtime platform • Based on top of Kubernetes • Chef based deployments are slowly migrating to DOCC
  63. 104.

    DOCC (DigitalOcean Control Center) • Platform for deploying containerized applications

    • Services, not Servers • Declarative deploys • Deployments take seconds, instead of hours • Provides several features out of the box • Automatic TLS certs, alerting, metrics, persistent volumes, etc...
  64. 105.
  65. 106.

    DOCC numbers • 850 applications • 3000 docker containers •

    3500 deploys in last 30 days • 15+ separate kubernetes clusters (3500+ cores, 10TB of memory)
  66. 107.

    How to deploy? • Concourse or GoCD kicks in after

    master is merged • Pipeline builds binary and docker-image • Deploy to DOCC directly • Chef • Fetch form HTTP server and put to a "prodution/foo/bar/binname" path • Chef is configured to pull from that path on the next run • Chef-client runs periodically on each node
  67. 109.

    Why upgrade? • Increased compilation speed • Increased performance in

    stdlib • Smaller Binaries • Safer code • Consistency with Go community
  68. 110.

    Why upgrade? (cont.) • Go 1.3: http.Server.ConnState hooks • Go

    1.4: for range x {} loops • Go 1.5: internal packages • Go 1.6: net/http: HTTP/2 support • Go 1.7: testing: sub-tests and sub-benchmarks, context in stdlib • Go 1.8: more context support, HTTP/2 push • Go 1.9: alias support (we don't use it yet), testing helper function
  69. 111.

    Upgrading Go • Currently the monorepo uses Go 1.9.1 •

    Usually we wait days/weeks if a new version is released • We have a standardized upgrade process • Built a container with new Go version • Create a "-force-test" branch with new container • Run all unit tests, fix issues and merge in couple of weeks
  70. 112.

    Moving forward • Build matrix among multiple Go versions •

    Policy of trying RC releases in staging
  71. 113.
  72. 114.

    Verdict • Monorepo let us solve a lot of problems,

    but needs constant maintenance • Go accelerated everything at DO • It became *the* language of our cloud • Stdlib > Docker > Kubernetes > DOCC > Droplets > Your website !