Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Go Modules

Avatar for Joe Joe
January 31, 2019

Go Modules

An overview of Go Modules and the general dependency management problems that arise in most languages.

Avatar for Joe

Joe

January 31, 2019
Tweet

More Decks by Joe

Other Decks in Programming

Transcript

  1. Go Modules Past, Present & Near-future GoSV, Netflix HQ, 31

    Jan 2019 Joe Blubaugh (@joeblubaugh, @LightStepHQ)
  2. Why do I care about Go Modules? Engineering @ LightStep

    ~30 programmers ~15 Go services, since 1.5 multiple kube clusters hybrid on-prem releases monorepo trivia: (my first go app was for 1.0!)
  3. the dependency problem go before dependency management mvs - go

    mod’s version selection algorithm go modules today go modules in the near-future Agenda
  4. Given A and it’s imports, we need to derive and

    fetch the full graph of packages needed to build A. Essentially “just” graph traversal. Getting “new” stuff is more complex... Managing the chaos of code changes |-F-| D E | | B C | | A----
  5. Adding versions: - assume only one package with same name.

    - therefore, can’t have two versions of the same package in one binary. Managing the chaos of code changes |-F.1-| F.2 D E.1 | | | E.3 G.3 B C.1 | | | | C.2- A------
  6. When we update a package and its dependencies change, we

    need to re-build the graph. A *cannot* be built if we’re upgrading C.1 to C.2 Managing the chaos of code changes |-F.1-| F.2 D E.1 | | | E.3 G.3 B C.1 | | | | C.2- A------
  7. - How do I *know* what’s in my build? -

    How do I safely use code from external sources? - How do I publish code that others can use? - What happens when a dep “dies” like uuid? Managing the chaos of code changes |-F-| D E | | B - C | A
  8. go get; go get -u Lots of $GOPATH hacking, git

    submodule-ing, import path rewriting was used to keep the source we built programs from consistent from build-to-build. Handling updates for deps-of-deps is an error-prone process. dep management before go mod
  9. gopkg.in/$foo/$bar.v$N Mirrors versioned packages on Github so they work with

    go get Redirects to github.com/foo/bar with branch / tag vN, vN.M, vN.M.O Helps us avoid getting a major version update we aren’t ready for, but still get minor updates. If $foo/$bar’s dependencies don’t use Gopkg.in, still vulnerable to breakage from changes in dependencies. dep management before go mod
  10. The vendor/ directory. (starting in Go 1.6) When go build

    is run from a directory d/, it resolves import paths - in a vendor/ directory rooted at d/ - in vendor/ directories rooted at parents of d/ - in the $GOPATH Put all your dependencies in vendor/ and what’s in $GOPATH won’t affect your build. (But you still need to be in $GOPATH!) dep management before go mod
  11. 3rd party tools like Glide and dep - manage your

    vendor/ directory for reproducible builds - Include a manifest specifying constraints for *direct dependencies* (Gopkg.toml) - output a version lockfile of some kind. (Gopkg.lock) - Updates the lockfile by finding a set of versions that satisfy the constraints in the constraint file. But how do you find that set of versions? dep management before go mod
  12. - Derive the dependency graph (N, E) from recursing along

    import paths - Accumulate constraints from modules along the way. (pom.xml, Gopkg.toml, Cargo.toml) - Find a set of N in the graph that can satisfy the constraints - This problem is NP complete! - But we have some good heuristic approaches using SAT solvers Most dependency management systems
  13. changes some assumptions of many version selection systems: 1. requires

    Semantic Import Versioning github.com/foo (v0 and v1) github.com/foo/v2 2. exclusions only used from the *main module* 3. Requirements *must specify one and only one version*. require github.com/foo/v2 v2.1.1 (roughly means >=v2.1.1) Minimal Version Selection
  14. Why make these changes? Enables a linear-time version selection algorithm

    (MVS). A.1 -> A.1 -> A.1 -> A.1 B.2 B.2 B.2 C.1 C.1 C.1.2 C.1.2 Traverse the graph, building a list of all imports and their required versions. Keep the maximum version listed in any requirements. Never select a version that is newer than *any* of the requirements. Minimal Version Selection
  15. Why make these changes? Treats different major versions as different

    named packages, allowing “broken diamond” dependencies when major versions differ. (Similar to the gopkg.in/yaml.v2 case). Minimal Version Selection
  16. With modules enabled, tooling is really simple: go mod init

    to set up your module’s go.mod file Add a dependency with an import statement while programming. go build, go test will fetch missing imports and update go.mod Get more specific: go get github.com/foo/[email protected] Great stuff
  17. module github.com/googleapis/gapic-generator-go require ( github.com/golang-commonmark/html v0.0.0-20180910111043-7d7c804e1d46 // indirect github.com/golang-commonmark/linkify v0.0.0-20180910111149-f05efb453a0e

    // indirect github.com/golang-commonmark/markdown v0.0.0-20180910011815-a8f139058164 github.com/golang-commonmark/mdurl v0.0.0-20180910110917-8d018c6567d6 // indirect github.com/golang-commonmark/puny v0.0.0-20180910110745-050be392d8b8 // indirect github.com/golang/protobuf v1.2.0 github.com/google/go-cmp v0.2.0 github.com/opennota/wd v0.0.0-20180911144301-b446539ab1e7 // indirect github.com/russross/blackfriday v2.0.0+incompatible // indirect github.com/shurcooL/sanitized_anchor_name v0.0.0-20170918181015-86672fcb3f95 // indirect golang.org/x/sync v0.0.0-20181108010431-42b317875d0f // indirect google.golang.org/genproto v0.0.0-20180914223249-4b56f30a1fd9 gopkg.in/yaml.v2 v2.2.1 ) replace google.golang.org/genproto => ./vendor/google.golang.org/genproto go.mod is *nice* to read
  18. Develop outside of $GOPATH! Module cache handles downloading and verifying

    your dependencies. (flashbacks to ~/.m2/*) You can still put dependencies in a vendor/ folder and check them in. Great for CI Helps solve the “left-pad problem” where a module goes away. go mod vendor Great stuff
  19. go mod will never fetch a newer version than all

    the dependencies specify! This can help with the “malicious bitcoin miner” problem but may also slow the speed of fixes through the ecosystem. Updating a dep to the newest version is simple: go get -u module/name MVS is fast (linear time!) MVS gives the base binary control over exclusion and replacement, not your dependencies. Great stuff
  20. Migration from other dependency managers is fairly easy: go mod

    init mod-aware go can handle code that does & doesn’t support modules using “pseudo-versions”: v0.0.0-20150612182905-d4adf43b75b9 Great stuff
  21. Major versions need module paths that include their version numbers:

    import my/foo/v3/mypkg I suspect this will lead to slower adoption of major versions by most users, and fewer changes Requires some serious finagling of your builds to support both Go modules and older Go code if you have releases >= v2.0.0 For Libraries
  22. *lots* of CLI tools still need updating - lots of

    tools need to move off of go/build and onto golang.org/x/tools/go/packages Particularly hard on us vim users … the big corporate IDEs are in great shape! Tooling
  23. go mod only includes directories that include go files. If

    you were relying on .c, .protos, etc in subdirectories, you’ll need to use some tools to help you pull those back in. Repos with multiple modules under edit need to do a tricky replace dance in their modules files. Distributing versioned go tools “for free” by abusing your dependency manager doesn’t really work well with go mod Local code management
  24. - module support will be on by default - $GOPATH

    mode will be deprecated Go 1.13 (August 2019)
  25. Module proxies: fast mirrors of modules, hosted on more reliable

    infra. (How do I know what’s in my build?) Plans for a Google-hosted proxy & separate notary service to provide verifiable source code hashes. (How do I safely use code from external sources?) (How do I know what’s in my build?) Module proxies & notary
  26. rsc’s blog posts on SAT solvers and MVS, and his

    series on modules Go Modules in 2019 Sam Boyer’s So you want to write a package manager These slides: speakerdeck.com/joeblubaugh/go-modules Appendix