Go at DigitalOcean

B1019ca5714cf8e9951868d6bc517827?s=47 Fatih Arslan
November 05, 2017

Go at DigitalOcean

How do we use Go at DigitalOcean

B1019ca5714cf8e9951868d6bc517827?s=128

Fatih Arslan

November 05, 2017
Tweet

Transcript

  1. Go at DigitalOcean Fatih Arslan Sr. Software Engineer @DigitalOcean –

    GoCon, Tokyo 2017
  2. Me • Sr. Software Engineer @Delivery Team • Creator of

    vim-go • Go contributor, author of many popular Go packages (i.e: color, structs, etc..) • Tool maker (i.e: gomodifytags, motion, etc...) • Coffee and bag geek
  3. DigitalOcean is a simple and robust cloud computing platform, designed

    for developers.
  4. None
  5. DigitalOcean at a glance • 3 million users have signed

    up for DO • 66 million droplets deployed (currently ~100k droplets/day) • It's been a month since Spaces launch. • We have 300 million objects, over 100 TB data
  6. Programming languages used at DO • Ruby, Python, Perl, JS,

    C++ • Much of our infrastructure has been moved to Go • Current platforms are being ported to Go as well
  7. State of Go

  8. How did we start using Go?

  9. First time using Go • First Go service was VNC

    proxy (by Mac Browning in March 2014) • Gained traction when engineers attended GopherCon 2014 • Was a huge success, other services followed quickly • metadata service, imagemgmt, metrics service, dns rewrite, etc..
  10. None
  11. VNC Console • Goroutines made it easy to duplex TCP

    and WebSocket connections • Interface usage enabled end to end testing • Go's stdlib was very powerful. Zero-downtime deploys were common, with no user interruptions.
  12. Some of the first issues

  13. Code sharing was problematic

  14. Code sharing • Each single repo had different versions of

    dependencies • No official vendor support yet • Adding a package required to rewrite import paths • Took weeks to update deps across repos
  15. No unified CI/CD integration

  16. No internal stdlib development

  17. Monorepo?

  18. None
  19. Cthulhu

  20. First commit: Dec 3, 2014

  21. Code structure

  22. Code structure cthulhu ├── docode │ └── src │ └──

    do │ ├── doge │ ├── exp │ ├── services │ ├── teams │ ├── tools │ └── vendor └── script
  23. Code structure (cont.) • doge: DigitalOcean Go Environment. Our internal

    "standard library" • exp: experimental stuff, not CI/CD checked • services: deprecated. Used to store services at DO, replaced by teams/ • teams: team specific code. All DO services are here • tools: internal tools, cmds. Mainly for CI/CD integration • script: shell scripts. Non Go related code cthulhu ├── docode │ └── src │ └── do │ ├── doge │ ├── exp │ ├── services │ ├── teams │ ├── tools │ └── vendor └── script
  24. Internal stdlib: doge cthulhu ├── docode │ └── src │

    └── do │ ├── doge │ ├── exp │ ├── services │ ├── teams │ ├── tools │ └── vendor └── script
  25. Internal stdlib: doge • Over 30 packages • context: Additional

    helper packages for Go's context package • dorpc: Wraps gRPC to provide DO specific support • httpclient: custom HTTP client tuned for our internal infrastructure • log: customer K/V logger that logs to our centralized logging platform • ... doge ├── context ├── dorpc ├── ... ├── ... ├── httpclient ├── ... ├── ... ├── ... ├── log ├── ... ├── ... └── version
  26. Monorepo stats

  27. Cthulhu overview • 28,639 commits • 824 branches • 142

    contributors • 830,434 lines of DigitalOcean-authored Go code • 2,136,373 lines of vendored Go code
  28. Monoropo issues

  29. Monorepo issues • Vendoring is problematic • What if two

    services depends on two different versions of the same package? • Large GOPATH causes slow tooling performance (goimports, guru, etc..) • Needs good tooling and constant maintenance • Slow building
  30. Onboarding people

  31. Onboarding 1. Clone repo: git clone cthulhu.git 2. Call .env.sh

    (sets GOPATH, PATH (GOPATH/bin)) 3. Start coding!
  32. Direnv (optional) • It hooks into bash, zsh, fish, ...

    and automatically loads or unloads environment variables based on the current directory • https://github.com/direnv/direnv • Very handy to switch GOPATH's (personal and company) • Direnv is written in Go, compiles into a single binary and is very fast • In cthulhu: ln -s .env.sh .envrc
  33. Direnv (optional) $ echo $GOPATH /Users/fatih/go $ cd cthulhu direnv:

    loading .envrc direnv: export ~GOPATH ~PATH $ echo $GOPATH /Users/fatih/Code/do/cthulhu/docode
  34. What if you have never programmed in Go before?

  35. What if you are a new Gopher? • Internal Go

    guide and documentation • Slack #golang channel • Go readability team • Mentoring new developers
  36. Frustrations of new engineers • What is GOPATH? • Vendoring

    packages • How to deal with other teams? • Finding the appropriate internal 'stdlib' is not easy
  37. Dependency Management

  38. Three years ago (2014)

  39. Three years ago (2014) cthulhu ├── docode │ └── src

    │ ├── doge │ ├── services │ └── tools └── third_party └── src
  40. Three years ago (2014) • Third party packages were in

    third_party folder • GOPATH=${CTHULHU}/third_party:${CTHULHU}/docode • go get automatically puts dependencies to third_party first • No submodules are used, we rename .git to .checkout_git • If package is not version controlled, a file "import.md" had to be added
  41. Vendoring a package (2014) $ go get github.com/fatih/structs $ cd

    third_party/src/github.com/fatih/structs $ mv .git .checkout_git $ git add . $ git commit -m "Vendored fatih/structs"
  42. Updating a package (2014) $ cd third_party/src/github.com/fatih/structs # Update package

    to HEAD $ mv .checkout_git .git $ git pull origin master $ mv .git .checkout_git # add to monorepo $ git add . $ git commit -m "Vendored fatih/structs"
  43. check3rdparty

  44. check3rdparty • Makes sure no .git is added • .checkout_git

    is allowed • import.md describes how to update/vendor package if it's not version controlled github.com/fatih/structs ├── .git (not allowed) ├── .checkout_git (ok) └── main.go github.com/fatih/structs ├── import.md (ok) └── main.go or
  45. One year ago (2016)

  46. One year ago (2016) • Go 1.6 was released in

    17 February 2016, with Vendor support enabled • We waited for 6 months after we made the switch to vendor/ folder • Removed third_party completely and moved all packages to vendor/ • Started to use govendor as it had vendor support
  47. One year ago (2016) https://twitter.com/golang/status/700083070414643201

  48. One year ago (2016)

  49. Switch to vendor folder (2016) cthulhu └── docode └── src

    ├── doge ├── services ├── tools └── vendor cthulhu ├── docode │ └── src │ ├── doge │ ├── services │ └── tools └── third_party └── src
  50. Vendoring&Updating a package (now) # vendors or updates package $

    govendor fetch github.com/fatih/structs # add to monorepo $ git add . $ git commit -m "Vendored fatih/structs"
  51. Added do/ prefix (2017) cthulhu ├── docode │ └── src

    │ └── do │ ├── doge │ ├── services │ ├── tools │ └── vendor cthulhu ├── docode │ └── src │ ├── doge │ ├── services │ ├── tools │ └── vendor
  52. Added do/ prefix (2017) • Vendor tools don't like vendor/

    being under GOPATH directly • https://github.com/kardianos/govendor/issues/237 • Clear ownership, we can see do/ stuff belongs to us • Allows us easily to open source stuff later when we rename do/ to do.co/
  53. Issues with govendor

  54. Slow on macOS • macOS file system (HFS+ and APFS)

    is case insensitive • both github.com/foo and github.com/FOO are the same • but for Go, import paths are case sensitive! • govendor tries to normalize it by converting all paths to lowercase • excessive usage of strings.ToLower() • a typical vendor takes many minutes
  55. Each string operation takes ~21 seconds Before

  56. Replace strings.ToLower() calls

  57. memoize the lowered string

  58. Lowered to ~5 seconds ( 400% improvement) After

  59. github.com/golang/dep ? • We're still evaluating it • Migrating from

    govendor to dep is not easy yet • We have an incomplete govendor.json • Some dependencies don't exist in public anymore • Github Enterprise doesn't work well with import paths (see: https://github.com/ golang/dep/issues/174)
  60. CI/CD integration

  61. CI/CD integration • Drone (github.com/drone/drone) is used for our monorepo

    • Runs for each branch & pull request and periodically for all DO packages • Also used for deployment (more on this later) • Default Go tools: gofmt, go vet and golint • Custom DO tools: gta, buildlint, explint, etc.. • Concourse (http://concourse.ci) and GoCD (https://www.gocd.org) is used for deployment
  62. gofmt instead of goimports • We were using a custom

    goimports fork • Made sure that do/ prefix was in a block • Was causing problems, people were using gofmt or standard goimports • Is replaced with gofmt • goimports is still encouraged to group block of import paths, but not required
  63. golint • Before golint around 1500 errors were detected •

    It took several weeks to fix all of them • Benefit: we have internal godoc running with high quality documentation
  64. Custom tools •gta •buildlint •githubjanitor •explint •autoreview

  65. In early 2016, CI builds took an average of 20

    minutes
  66. How did we improve CI build duration?

  67. gta: Go Test Auto

  68. src ├── foo │ └── foo.go ├── bar │ ├──

    bar.go │ └── baz │ └── baz.go ├── qux │ └── qux.go └── example └── example.go package qux import "fmt" func Hello() { fmt.Println("Hello GoCon") }
  69. src ├── foo │ └── foo.go ├── bar │ ├──

    bar.go │ └── baz │ └── baz.go ├── qux │ └── qux.go └── example └── example.go package foo import "qux" func main() { qux.Hello() } Package foo imports qux package qux import "fmt" func Hello() { fmt.Println("Hello GoCon") }
  70. $ git diff --- a/docode/src/qux/qux.go +++ b/docode/src/qux/qux.go @@ -3,5 +3,5

    @@ package qux import "fmt" func Hello() { - fmt.Println("Hello GoCon!") + fmt.Println("こんにちは。GoCon!") } src ├── foo │ └── foo.go ├── bar │ ├── bar.go │ └── baz │ └── baz.go ├── qux │ └── qux.go └── example └── example.go Change something in Qux
  71. src ├── foo │ └── foo.go ├── bar │ ├──

    bar.go │ └── baz │ └── baz.go ├── qux │ └── qux.go └── example └── example.go $ gta foo qux Run gta
  72. src ├── foo │ └── foo.go ├── bar │ ├──

    bar.go │ └── baz │ └── baz.go ├── qux │ └── qux.go └── example └── example.go $ go build -v ./... foo bar bar/baz qux example go build packages
  73. src ├── foo │ └── foo.go ├── bar │ ├──

    bar.go │ └── baz │ └── baz.go ├── qux │ └── qux.go └── example └── example.go $ go build -v ./... $ go build -v $(gta) foo qux go build with gta
  74. gta: Go Test Auto • Average build decreased from 20

    minutes to 2 – 3 minutes • Finds differences between feature and master branch. • Returns a list of packages that needs to be tested/built • go tool compatible • go build $(gta), go test $(gta), etc... • Can be disabled with "-force-test" appended to the branch name
  75. buildlint

  76. buildlint • Checks build tags • Example: // +build linux

    • Finds problems • Disallows race: • Positive and negative of the same tag: • Empty build tag: // +build !race // +build linux,!linux // +build
  77. $ cat foo.go package main import "fmt" // +build //

    +build !race // +build !linux,linux func main() { fmt.Println("GoCon") }
  78. $ cat foo.go package main import "fmt" // +build //

    +build !race // +build !linux,linux func main() { fmt.Println("GoCon") } $ buildlint foo.go:5:1: empty build tag comment foo.go:6:1: found disallowed build tag "!race" foo.go:7:1: found positive and negative tags for "linux" in same group
  79. explint

  80. explint • Makes sure exp can only be imported from

    exp itself, so production code can't use it • This is forbidden: cthulhu ├── docode │ └── src │ └── do │ ├── doge │ ├── exp │ ├── services │ ├── teams │ ├── tools │ └── vendor └── script package main import ( "do/exp/foo" ) func main() { foo.Foo() }
  81. Upcoming ideas

  82. Upcoming ideas • Vendorlint • Makes sure new vendors or

    vendor updates are not done in the same commit as application code changes • Deplint • Warns user if a vendored package is already vendored in our global top-level vendor/ folder • i.e: src/teams/project/vendor/github.com/fatih/structs is already vendored in src/vendor/ github.com/fatih/structs • Autovendor • Automatically check for new updates and open a new Pull Request with changes
  83. Before merging the PR

  84. Before the merge • Every PR needs to be reviewed

    by its peers • Can be merged if peer gives +1 or LGTM • Reviewers are automatically tagged (with autoreview)
  85. autoreview

  86. autoreview • A tool for tagging teams for reviewing Github

    PR's • Looks for OWNERS files relevant to changes in a PR • Adds comment that tags the missing owners • Modeled after Google's OWNERS system
  87. OWNERS format The file is line-delimited, and supports the following

    patterns: @\S+ : a github team or person \S+@ : an email address (@digitalocean.com is inferred). #\S+ : a slack channel
 Example contents of an “OWNERS” file which contains all three: @digitalocean/storage storage@ #storage
  88. autoreview teams ├── delivery │ ├── OWNERS │ └── project1

    │ ├── server.go │ ├── server_test.go │ └── main.go └────── project2 ├── foo.go └── main.go If anything under project1 or project2 changes
  89. autoreview

  90. Changes PR status to pending

  91. Success if signoff comment is added • LGTM, lgtm, +1

    • :shipit: ( ) • :+1: () • :ship::it: (#) Some signoff comments:
  92. githubjanitor

  93. githubjanitor • Checks for stale PR's • Warns after 10

    days of inactivity • Closes PR's after 15 days of inactivity. • Resets timer if you push a commit
  94. githubjanitor

  95. After merging the PR

  96. After the merge • Master is tested again. Conflicts can

    happen again, i.e: • CI check passed for PR foo • Meanwhile PR bar is merged, that changes some of the code of foo • PR foo is merged, master fails in this case
  97. Deployment & Delivery

  98. Deployment • Each team is responsible for their own deployment

    • Single source of truth: monorepo • Some teams have their own project-repos, not part of monorepo • Services written in Go are usually run inside a container
  99. Binary server: gtartifacts • Remember gta? We also have gtartifacts!

    • HTTP server responsible of managing binaries • After a merge: • Gtartifacts builds changed binaries (via gta) • Publishes them to our internal storage server • Teams can fetch based on git SHA, via CLI or plain HTTP requests
  100. None
  101. CLI: artifactctl

  102. Using artifactctl

  103. What platform to use? • Chef • Still being used

    a lot. Old services usually • Not very fun to use it • DOCC (Kubernetes) • Our internal application runtime platform • Based on top of Kubernetes • Chef based deployments are slowly migrating to DOCC
  104. DOCC (DigitalOcean Control Center) • Platform for deploying containerized applications

    • Services, not Servers • Declarative deploys • Deployments take seconds, instead of hours • Provides several features out of the box • Automatic TLS certs, alerting, metrics, persistent volumes, etc...
  105. None
  106. DOCC numbers • 850 applications • 3000 docker containers •

    3500 deploys in last 30 days • 15+ separate kubernetes clusters (3500+ cores, 10TB of memory)
  107. How to deploy? • Concourse or GoCD kicks in after

    master is merged • Pipeline builds binary and docker-image • Deploy to DOCC directly • Chef • Fetch form HTTP server and put to a "prodution/foo/bar/binname" path • Chef is configured to pull from that path on the next run • Chef-client runs periodically on each node
  108. Upgrading Go

  109. Why upgrade? • Increased compilation speed • Increased performance in

    stdlib • Smaller Binaries • Safer code • Consistency with Go community
  110. Why upgrade? (cont.) • Go 1.3: http.Server.ConnState hooks • Go

    1.4: for range x {} loops • Go 1.5: internal packages • Go 1.6: net/http: HTTP/2 support • Go 1.7: testing: sub-tests and sub-benchmarks, context in stdlib • Go 1.8: more context support, HTTP/2 push • Go 1.9: alias support (we don't use it yet), testing helper function
  111. Upgrading Go • Currently the monorepo uses Go 1.9.1 •

    Usually we wait days/weeks if a new version is released • We have a standardized upgrade process • Built a container with new Go version • Create a "-force-test" branch with new container • Run all unit tests, fix issues and merge in couple of weeks
  112. Moving forward • Build matrix among multiple Go versions •

    Policy of trying RC releases in staging
  113. Verdict

  114. Verdict • Monorepo let us solve a lot of problems,

    but needs constant maintenance • Go accelerated everything at DO • It became *the* language of our cloud • Stdlib > Docker > Kubernetes > DOCC > Droplets > Your website !
  115. Thanks! Fatih Arslan @fatih @fatih fatih@digitalocean.com