Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Go at DigitalOcean

Fatih Arslan
November 05, 2017

Go at DigitalOcean

How do we use Go at DigitalOcean

Fatih Arslan

November 05, 2017
Tweet

More Decks by Fatih Arslan

Other Decks in Technology

Transcript

  1. Go at DigitalOcean
    Fatih Arslan
    Sr. Software Engineer @DigitalOcean – GoCon, Tokyo 2017

    View full-size slide

  2. Me
    • Sr. Software Engineer @Delivery Team
    • Creator of vim-go
    • Go contributor, author of many popular Go
    packages (i.e: color, structs, etc..)
    • Tool maker (i.e: gomodifytags, motion,
    etc...)
    • Coffee and bag geek

    View full-size slide

  3. DigitalOcean
    is a simple and robust cloud computing platform,
    designed for developers.

    View full-size slide

  4. DigitalOcean at a glance
    • 3 million users have signed up for DO
    • 66 million droplets deployed (currently ~100k droplets/day)
    • It's been a month since Spaces launch.
    • We have 300 million objects, over 100 TB data

    View full-size slide

  5. Programming languages used at DO
    • Ruby, Python, Perl, JS, C++
    • Much of our infrastructure has been moved to Go
    • Current platforms are being ported to Go as well

    View full-size slide

  6. How did we start
    using Go?

    View full-size slide

  7. First time using Go
    • First Go service was VNC proxy (by Mac Browning in March 2014)
    • Gained traction when engineers attended GopherCon 2014
    • Was a huge success, other services followed quickly
    • metadata service, imagemgmt, metrics service, dns rewrite, etc..

    View full-size slide

  8. VNC Console
    • Goroutines made it easy to duplex TCP and WebSocket connections
    • Interface usage enabled end to end testing
    • Go's stdlib was very powerful. Zero-downtime deploys were common, with
    no user interruptions.

    View full-size slide

  9. Some of the
    first issues

    View full-size slide

  10. Code sharing
    was problematic

    View full-size slide

  11. Code sharing
    • Each single repo had different versions of dependencies
    • No official vendor support yet
    • Adding a package required to rewrite import paths
    • Took weeks to update deps across repos

    View full-size slide

  12. No unified
    CI/CD integration

    View full-size slide

  13. No internal stdlib
    development

    View full-size slide

  14. First commit: Dec 3, 2014

    View full-size slide

  15. Code structure

    View full-size slide

  16. Code structure
    cthulhu
    ├── docode
    │ └── src
    │ └── do
    │ ├── doge
    │ ├── exp
    │ ├── services
    │ ├── teams
    │ ├── tools
    │ └── vendor
    └── script

    View full-size slide

  17. Code structure (cont.)
    • doge: DigitalOcean Go Environment. Our
    internal "standard library"
    • exp: experimental stuff, not CI/CD checked
    • services: deprecated. Used to store services at
    DO, replaced by teams/
    • teams: team specific code. All DO services are
    here
    • tools: internal tools, cmds. Mainly for CI/CD
    integration
    • script: shell scripts. Non Go related code
    cthulhu
    ├── docode
    │ └── src
    │ └── do
    │ ├── doge
    │ ├── exp
    │ ├── services
    │ ├── teams
    │ ├── tools
    │ └── vendor
    └── script

    View full-size slide

  18. Internal stdlib: doge
    cthulhu
    ├── docode
    │ └── src
    │ └── do
    │ ├── doge
    │ ├── exp
    │ ├── services
    │ ├── teams
    │ ├── tools
    │ └── vendor
    └── script

    View full-size slide

  19. Internal stdlib: doge
    • Over 30 packages
    • context: Additional helper packages for Go's
    context package
    • dorpc: Wraps gRPC to provide DO specific
    support
    • httpclient: custom HTTP client tuned for our
    internal infrastructure
    • log: customer K/V logger that logs to our
    centralized logging platform
    • ...
    doge
    ├── context
    ├── dorpc
    ├── ...
    ├── ...
    ├── httpclient
    ├── ...
    ├── ...
    ├── ...
    ├── log
    ├── ...
    ├── ...
    └── version

    View full-size slide

  20. Monorepo stats

    View full-size slide

  21. Cthulhu overview
    • 28,639 commits
    • 824 branches
    • 142 contributors
    • 830,434 lines of DigitalOcean-authored Go code
    • 2,136,373 lines of vendored Go code

    View full-size slide

  22. Monoropo issues

    View full-size slide

  23. Monorepo issues
    • Vendoring is problematic
    • What if two services depends on two different versions of the same
    package?
    • Large GOPATH causes slow tooling performance (goimports, guru, etc..)
    • Needs good tooling and constant maintenance
    • Slow building

    View full-size slide

  24. Onboarding people

    View full-size slide

  25. Onboarding
    1. Clone repo: git clone cthulhu.git
    2. Call .env.sh (sets GOPATH, PATH (GOPATH/bin))
    3. Start coding!

    View full-size slide

  26. Direnv (optional)
    • It hooks into bash, zsh, fish, ... and automatically loads or unloads
    environment variables based on the current directory
    • https://github.com/direnv/direnv
    • Very handy to switch GOPATH's (personal and company)
    • Direnv is written in Go, compiles into a single binary and is very fast
    • In cthulhu: ln -s .env.sh .envrc

    View full-size slide

  27. Direnv (optional)
    $ echo $GOPATH
    /Users/fatih/go
    $ cd cthulhu
    direnv: loading .envrc
    direnv: export ~GOPATH ~PATH
    $ echo $GOPATH
    /Users/fatih/Code/do/cthulhu/docode

    View full-size slide

  28. What if you have never
    programmed in Go before?

    View full-size slide

  29. What if you are a new Gopher?
    • Internal Go guide and documentation
    • Slack #golang channel
    • Go readability team
    • Mentoring new developers

    View full-size slide

  30. Frustrations of new engineers
    • What is GOPATH?
    • Vendoring packages
    • How to deal with other teams?
    • Finding the appropriate internal 'stdlib' is not easy

    View full-size slide

  31. Dependency Management

    View full-size slide

  32. Three years ago (2014)

    View full-size slide

  33. Three years ago (2014)
    cthulhu
    ├── docode
    │ └── src
    │ ├── doge
    │ ├── services
    │ └── tools
    └── third_party
    └── src

    View full-size slide

  34. Three years ago (2014)
    • Third party packages were in third_party folder
    • GOPATH=${CTHULHU}/third_party:${CTHULHU}/docode
    • go get automatically puts dependencies to third_party first
    • No submodules are used, we rename .git to .checkout_git
    • If package is not version controlled, a file "import.md" had to be added

    View full-size slide

  35. Vendoring a package (2014)
    $ go get github.com/fatih/structs
    $ cd third_party/src/github.com/fatih/structs
    $ mv .git .checkout_git
    $ git add .
    $ git commit -m "Vendored fatih/structs"

    View full-size slide

  36. Updating a package (2014)
    $ cd third_party/src/github.com/fatih/structs
    # Update package to HEAD
    $ mv .checkout_git .git
    $ git pull origin master
    $ mv .git .checkout_git
    # add to monorepo
    $ git add .
    $ git commit -m "Vendored fatih/structs"

    View full-size slide

  37. check3rdparty

    View full-size slide

  38. check3rdparty
    • Makes sure no .git is added
    • .checkout_git is allowed
    • import.md describes how to
    update/vendor package if it's not
    version controlled
    github.com/fatih/structs
    ├── .git (not allowed)
    ├── .checkout_git (ok)
    └── main.go
    github.com/fatih/structs
    ├── import.md (ok)
    └── main.go
    or

    View full-size slide

  39. One year ago (2016)

    View full-size slide

  40. One year ago (2016)
    • Go 1.6 was released in 17 February 2016, with Vendor support enabled
    • We waited for 6 months after we made the switch to vendor/ folder
    • Removed third_party completely and moved all packages to vendor/
    • Started to use govendor as it had vendor support

    View full-size slide

  41. One year ago (2016)
    https://twitter.com/golang/status/700083070414643201

    View full-size slide

  42. One year ago (2016)

    View full-size slide

  43. Switch to vendor folder (2016)
    cthulhu
    └── docode
    └── src
    ├── doge
    ├── services
    ├── tools
    └── vendor
    cthulhu
    ├── docode
    │ └── src
    │ ├── doge
    │ ├── services
    │ └── tools
    └── third_party
    └── src

    View full-size slide

  44. Vendoring&Updating a package (now)
    # vendors or updates package
    $ govendor fetch github.com/fatih/structs
    # add to monorepo
    $ git add .
    $ git commit -m "Vendored fatih/structs"

    View full-size slide

  45. Added do/ prefix (2017)
    cthulhu
    ├── docode
    │ └── src
    │ └── do
    │ ├── doge
    │ ├── services
    │ ├── tools
    │ └── vendor
    cthulhu
    ├── docode
    │ └── src
    │ ├── doge
    │ ├── services
    │ ├── tools
    │ └── vendor

    View full-size slide

  46. Added do/ prefix (2017)
    • Vendor tools don't like vendor/ being under GOPATH directly
    • https://github.com/kardianos/govendor/issues/237
    • Clear ownership, we can see do/ stuff belongs to us
    • Allows us easily to open source stuff later when we rename do/ to do.co/

    View full-size slide

  47. Issues with govendor

    View full-size slide

  48. Slow on macOS
    • macOS file system (HFS+ and APFS) is case insensitive
    • both github.com/foo and github.com/FOO are the same
    • but for Go, import paths are case sensitive!
    • govendor tries to normalize it by converting all paths to lowercase
    • excessive usage of strings.ToLower()
    • a typical vendor takes many minutes

    View full-size slide

  49. Each string operation
    takes ~21 seconds
    Before

    View full-size slide

  50. Replace strings.ToLower() calls

    View full-size slide

  51. memoize the lowered string

    View full-size slide

  52. Lowered to ~5 seconds
    ( 400% improvement)
    After

    View full-size slide

  53. github.com/golang/dep ?
    • We're still evaluating it
    • Migrating from govendor to dep is not
    easy yet
    • We have an incomplete govendor.json
    • Some dependencies don't exist in
    public anymore
    • Github Enterprise doesn't work well with
    import paths (see: https://github.com/
    golang/dep/issues/174)

    View full-size slide

  54. CI/CD integration

    View full-size slide

  55. CI/CD integration
    • Drone (github.com/drone/drone) is used for our monorepo
    • Runs for each branch & pull request and periodically for all DO packages
    • Also used for deployment (more on this later)
    • Default Go tools: gofmt, go vet and golint
    • Custom DO tools: gta, buildlint, explint, etc..
    • Concourse (http://concourse.ci) and GoCD (https://www.gocd.org) is used for
    deployment

    View full-size slide

  56. gofmt instead of goimports
    • We were using a custom goimports fork
    • Made sure that do/ prefix was in a block
    • Was causing problems, people were using gofmt or standard goimports
    • Is replaced with gofmt
    • goimports is still encouraged to group block of import paths, but not
    required

    View full-size slide

  57. golint
    • Before golint around 1500 errors were detected
    • It took several weeks to fix all of them
    • Benefit: we have internal godoc running with high quality documentation

    View full-size slide

  58. Custom tools
    •gta
    •buildlint
    •githubjanitor
    •explint
    •autoreview

    View full-size slide

  59. In early 2016, CI builds took an
    average of 20 minutes

    View full-size slide

  60. How did we improve
    CI build duration?

    View full-size slide

  61. gta:
    Go Test Auto

    View full-size slide

  62. src
    ├── foo
    │ └── foo.go
    ├── bar
    │ ├── bar.go
    │ └── baz
    │ └── baz.go
    ├── qux
    │ └── qux.go
    └── example
    └── example.go
    package qux
    import "fmt"
    func Hello() {
    fmt.Println("Hello GoCon")
    }

    View full-size slide

  63. src
    ├── foo
    │ └── foo.go
    ├── bar
    │ ├── bar.go
    │ └── baz
    │ └── baz.go
    ├── qux
    │ └── qux.go
    └── example
    └── example.go
    package foo
    import "qux"
    func main() {
    qux.Hello()
    }
    Package foo
    imports qux
    package qux
    import "fmt"
    func Hello() {
    fmt.Println("Hello GoCon")
    }

    View full-size slide

  64. $ git diff
    --- a/docode/src/qux/qux.go
    +++ b/docode/src/qux/qux.go
    @@ -3,5 +3,5 @@ package qux
    import "fmt"
    func Hello() {
    - fmt.Println("Hello GoCon!")
    + fmt.Println("こんにちは。GoCon!")
    }
    src
    ├── foo
    │ └── foo.go
    ├── bar
    │ ├── bar.go
    │ └── baz
    │ └── baz.go
    ├── qux
    │ └── qux.go
    └── example
    └── example.go
    Change something in Qux

    View full-size slide

  65. src
    ├── foo
    │ └── foo.go
    ├── bar
    │ ├── bar.go
    │ └── baz
    │ └── baz.go
    ├── qux
    │ └── qux.go
    └── example
    └── example.go
    $ gta
    foo
    qux
    Run gta

    View full-size slide

  66. src
    ├── foo
    │ └── foo.go
    ├── bar
    │ ├── bar.go
    │ └── baz
    │ └── baz.go
    ├── qux
    │ └── qux.go
    └── example
    └── example.go
    $ go build -v ./...
    foo
    bar
    bar/baz
    qux
    example
    go build packages

    View full-size slide

  67. src
    ├── foo
    │ └── foo.go
    ├── bar
    │ ├── bar.go
    │ └── baz
    │ └── baz.go
    ├── qux
    │ └── qux.go
    └── example
    └── example.go
    $ go build -v ./...
    $ go build -v $(gta)
    foo
    qux
    go build with gta

    View full-size slide

  68. gta: Go Test Auto
    • Average build decreased from 20 minutes to 2 – 3 minutes
    • Finds differences between feature and master branch.
    • Returns a list of packages that needs to be tested/built
    • go tool compatible
    • go build $(gta), go test $(gta), etc...
    • Can be disabled with "-force-test" appended to the branch name

    View full-size slide

  69. buildlint
    • Checks build tags
    • Example: // +build linux
    • Finds problems
    • Disallows race:
    • Positive and negative of the same tag:
    • Empty build tag:
    // +build !race
    // +build linux,!linux
    // +build

    View full-size slide

  70. $ cat foo.go
    package main
    import "fmt"
    // +build
    // +build !race
    // +build !linux,linux
    func main() {
    fmt.Println("GoCon")
    }

    View full-size slide

  71. $ cat foo.go
    package main
    import "fmt"
    // +build
    // +build !race
    // +build !linux,linux
    func main() {
    fmt.Println("GoCon")
    }
    $ buildlint
    foo.go:5:1: empty build tag comment
    foo.go:6:1: found disallowed build tag "!race"
    foo.go:7:1: found positive and negative tags
    for "linux" in same group

    View full-size slide

  72. explint
    • Makes sure exp can only be imported from
    exp itself, so production code can't use it
    • This is forbidden:
    cthulhu
    ├── docode
    │ └── src
    │ └── do
    │ ├── doge
    │ ├── exp
    │ ├── services
    │ ├── teams
    │ ├── tools
    │ └── vendor
    └── script
    package main
    import (
    "do/exp/foo"
    )
    func main() {
    foo.Foo()
    }

    View full-size slide

  73. Upcoming ideas

    View full-size slide

  74. Upcoming ideas
    • Vendorlint
    • Makes sure new vendors or vendor updates are not done in the same commit as application
    code changes
    • Deplint
    • Warns user if a vendored package is already vendored in our global top-level vendor/ folder
    • i.e: src/teams/project/vendor/github.com/fatih/structs is already vendored in src/vendor/
    github.com/fatih/structs
    • Autovendor
    • Automatically check for new updates and open a new Pull Request with changes

    View full-size slide

  75. Before merging the PR

    View full-size slide

  76. Before the merge
    • Every PR needs to be reviewed by its peers
    • Can be merged if peer gives +1 or LGTM
    • Reviewers are automatically tagged (with autoreview)

    View full-size slide

  77. autoreview
    • A tool for tagging teams for reviewing Github PR's
    • Looks for OWNERS files relevant to changes in a PR
    • Adds comment that tags the missing owners
    • Modeled after Google's OWNERS system

    View full-size slide

  78. OWNERS format
    The file is line-delimited, and supports the following patterns:
    @\S+ : a github team or person
    \S+@ : an email address (@digitalocean.com is inferred).
    #\S+ : a slack channel

    Example contents of an “OWNERS” file which contains all three:
    @digitalocean/storage
    storage@
    #storage

    View full-size slide

  79. autoreview
    teams
    ├── delivery
    │ ├── OWNERS
    │ └── project1
    │ ├── server.go
    │ ├── server_test.go
    │ └── main.go
    └────── project2
    ├── foo.go
    └── main.go
    If anything under
    project1 or project2
    changes

    View full-size slide

  80. Changes PR status to pending

    View full-size slide

  81. Success if signoff comment is added
    • LGTM, lgtm, +1
    • :shipit: ( )
    • :+1: ()
    • :ship::it: (#)
    Some signoff comments:

    View full-size slide

  82. githubjanitor

    View full-size slide

  83. githubjanitor
    • Checks for stale PR's
    • Warns after 10 days of inactivity
    • Closes PR's after 15 days of inactivity.
    • Resets timer if you push a commit

    View full-size slide

  84. githubjanitor

    View full-size slide

  85. After merging the PR

    View full-size slide

  86. After the merge
    • Master is tested again. Conflicts can happen again, i.e:
    • CI check passed for PR foo
    • Meanwhile PR bar is merged, that changes some of the code of foo
    • PR foo is merged, master fails in this case

    View full-size slide

  87. Deployment & Delivery

    View full-size slide

  88. Deployment
    • Each team is responsible for their own deployment
    • Single source of truth: monorepo
    • Some teams have their own project-repos, not part of monorepo
    • Services written in Go are usually run inside a container

    View full-size slide

  89. Binary server: gtartifacts
    • Remember gta? We also have gtartifacts!
    • HTTP server responsible of managing binaries
    • After a merge:
    • Gtartifacts builds changed binaries (via gta)
    • Publishes them to our internal storage server
    • Teams can fetch based on git SHA, via CLI or plain HTTP requests

    View full-size slide

  90. CLI: artifactctl

    View full-size slide

  91. Using artifactctl

    View full-size slide

  92. What platform to use?
    • Chef
    • Still being used a lot. Old services usually
    • Not very fun to use it
    • DOCC (Kubernetes)
    • Our internal application runtime platform
    • Based on top of Kubernetes
    • Chef based deployments are slowly migrating to DOCC

    View full-size slide

  93. DOCC (DigitalOcean Control Center)
    • Platform for deploying containerized applications
    • Services, not Servers
    • Declarative deploys
    • Deployments take seconds, instead of hours
    • Provides several features out of the box
    • Automatic TLS certs, alerting, metrics, persistent volumes, etc...

    View full-size slide

  94. DOCC numbers
    • 850 applications
    • 3000 docker containers
    • 3500 deploys in last 30 days
    • 15+ separate kubernetes clusters (3500+ cores, 10TB of memory)

    View full-size slide

  95. How to deploy?
    • Concourse or GoCD kicks in after master is merged
    • Pipeline builds binary and docker-image
    • Deploy to DOCC directly
    • Chef
    • Fetch form HTTP server and put to a "prodution/foo/bar/binname" path
    • Chef is configured to pull from that path on the next run
    • Chef-client runs periodically on each node

    View full-size slide

  96. Upgrading Go

    View full-size slide

  97. Why upgrade?
    • Increased compilation speed
    • Increased performance in stdlib
    • Smaller Binaries
    • Safer code
    • Consistency with Go community

    View full-size slide

  98. Why upgrade? (cont.)
    • Go 1.3: http.Server.ConnState hooks
    • Go 1.4: for range x {} loops
    • Go 1.5: internal packages
    • Go 1.6: net/http: HTTP/2 support
    • Go 1.7: testing: sub-tests and sub-benchmarks, context in stdlib
    • Go 1.8: more context support, HTTP/2 push
    • Go 1.9: alias support (we don't use it yet), testing helper function

    View full-size slide

  99. Upgrading Go
    • Currently the monorepo uses Go 1.9.1
    • Usually we wait days/weeks if a new version is released
    • We have a standardized upgrade process
    • Built a container with new Go version
    • Create a "-force-test" branch with new container
    • Run all unit tests, fix issues and merge in couple of weeks

    View full-size slide

  100. Moving forward
    • Build matrix among multiple Go versions
    • Policy of trying RC releases in staging

    View full-size slide

  101. Verdict
    • Monorepo let us solve a lot of problems, but needs constant maintenance
    • Go accelerated everything at DO
    • It became *the* language of our cloud
    • Stdlib > Docker > Kubernetes > DOCC > Droplets > Your website !

    View full-size slide

  102. Thanks!
    Fatih Arslan
    @fatih
    @fatih
    [email protected]

    View full-size slide