How to manage tool dependencies in Go

talked on golang.tokyo #20
https://golangtokyo.connpass.com/event/111077/

talked about izumin5210/gex https://github.com/izumin5210/gex

Transcript

1. How to manage tool dependencies
   golang.tokyo #20 - @izumi5210
   

   View Slide

2. @izumin5210
   Application Engineer, Wantedly People
   Wantedly, Inc.
   

   View Slide

3. tool dependencies ?
   Executable tools that installed

   via `go get`
   e.g.mockgen, protoc-gen-grpc-gateway, ...
   

   View Slide

4. Why should we manage
   versions of tools?
   

   View Slide

5. Why should we manage
   versions of tools?
   ⬇
   Behaviors of tools may change
   when developers install them via `go get`
   (e.g. an output file format of code generator)
   

   View Slide

6. mockgen@v1.0.1
   mockgen@v1.1.0
   mockgen@v1.2.0
   mockgen@bd3c8e8
   mockgen@v1.1.1
   mockgen@v1.0.0
   mockgen@ddf6e33
   Why should we manage
   versions of tools?
   ⬇
   Behaviors of tools may change
   when developers install them via `go get`
   (e.g. an output file format of code generator)
   

   View Slide

7. Mock implementation that I wanna create
   Mocks that updated unexpectedly
   Because I use mockgen@v1.2.0
   but they are generated by mockgen@v1.0.1
   If you do not manage
   tool dependencies ...
   

   View Slide

8. How about existing package managers?
   (e.g. dep ,go mod)
   `go build` is the out of scope of their responsibility
   dep document says*
   if you need the tool to be installed you still run the following
   (manually or from a `Makefile`) after each `dep ensure`

   ```

   cd vendor/pkg/to/install

   go install .

   ```
   * https://golang.github.io/dep/docs/Gopkg.toml.html#required
   

   View Slide

9. How about existing package managers?
   (e.g. dep ,go mod)
   # Makefile
   DEP_COMMANDS := \
   vendor/github.com/golang/mock/mockgen
   .PHONY: dep
   dep: Gopkg.toml Gopkg.lock
   @dep ensure -v -vendor-only
   pkgs="$(DEP_COMMANDS)"; \
   for pkg in $$pkgs; do \
   cd $$pkg; \
   GOBIN="$(shell pwd)/bin" go install .; \
   cd -; \
   done
   # Gopkg.toml
   required = [
   "github.com/golang/mock/mockgen",
   ]
   
   

   View Slide

10. https://github.com/golang/go/issues/25922
    cmd/go: clarify best practice for tool dependencies
    #25922 · golang/go
    

    View Slide

11. cmd/go: clarify best practice for tool dependencies
    #25922 · golang/go
    // tools.go
    // +build tools
    package tools
    import (
    _ "golang.org/x/tools/cmd/stringer"
    )
    // go.mod
    module example.com/hello
    require golang.org/x/tools v0.0.0-20180813205110-a434f64ace81 // indirect
    go get
    export GOBIN=$(pwd)/bin
    go install golang.org/x/tools/cmd/stringer
    

    View Slide

12. cmd/go: clarify best practice for tool dependencies
    #25922 · golang/go
    Other 3rd-party tools (e.g. virtualgo, retool, ...) need to

    introduce new management mechanisms
    "tools.go" method can use existing dependency management
    mechanisms (e.g. dep, go mod)
    

    View Slide

13. cmd/go: clarify best practice for tool dependencies
    #25922 · golang/go
    // tools.go
    // +build tools
    package tools
    import (
    _ "golang.org/x/tools/cmd/stringer"
    )
    // go.mod
    module example.com/hello
    require golang.org/x/tools v0.0.0-20180813205110-a434f64ace81 // indirect
    go get
    export GOBIN=$(pwd)/bin
    go install golang.org/x/tools/cmd/stringer
    Best practice?
    

    View Slide

14. cmd/go: clarify best practice for tool dependencies
    #25922 · golang/go
    // tools.go
    // +build tools
    package tools
    import (
    _ "golang.org/x/tools/cmd/stringer"
    )
    // go.mod
    module example.com/hello
    require golang.org/x/tools v0.0.0-20180813205110-a434f64ace81 // indirect
    go get
    export GOBIN=$(pwd)/bin
    go install golang.org/x/tools/cmd/stringer
    We want to also automate this step!
    

    View Slide

15. gex: manage tools with `tools.go`
    github.com/izumin5210/gex
    $ gex --add github.com/golang/mock/mockgen
    $ cat tools.go
    // Code generated by github.com/izumin5210/gex. DO NOT EDIT.
    // +build tools
    package tools
    // tool dependencies
    import (
    _ "github.com/golang/mock/mockgen"
    )
    $ cat go.mod | grep mock
    github.com/golang/mock v1.1.1 // indirect
    $ gex mockgen
    # prints mockgen's help text...
    

    View Slide

16. gex: manage tools with `tools.go`
    github.com/izumin5210/gex
    $ gex --add github.com/golang/mock/mockgen
    $ cat tools.go
    // Code generated by github.com/izumin5210/gex. DO NOT EDIT.
    // +build tools
    package tools
    // tool dependencies
    import (
    _ "github.com/golang/mock/mockgen"
    )
    $ cat go.mod | grep mock
    github.com/golang/mock v1.1.1 // indirect
    $ gex mockgen
    # prints mockgen's help text...
    Install mockgen via gex
    - gex manages dependencies version using `dep` or `go mod`
    - So `gex --add ...` runs `go get ...` or `dep ensure -add ...` internally
    - Build mockgen and output to `$PWD/bin`
    

    View Slide

17. gex: manage tools with `tools.go`
    github.com/izumin5210/gex
    $ gex --add github.com/golang/mock/mockgen
    $ cat tools.go
    // Code generated by github.com/izumin5210/gex. DO NOT EDIT.
    // +build tools
    package tools
    // tool dependencies
    import (
    _ "github.com/golang/mock/mockgen"
    )
    $ cat go.mod | grep mock
    github.com/golang/mock v1.1.1 // indirect
    $ gex mockgen
    # prints mockgen's help text...
    `tools.go` will generate by `gex`
    

    View Slide

18. gex: manage tools with `tools.go`
    github.com/izumin5210/gex
    $ gex --add github.com/golang/mock/mockgen
    $ cat tools.go
    // Code generated by github.com/izumin5210/gex. DO NOT EDIT.
    // +build tools
    package tools
    // tool dependencies
    import (
    _ "github.com/golang/mock/mockgen"
    )
    $ cat go.mod | grep mock
    github.com/golang/mock v1.1.1 // indirect
    $ gex mockgen
    # prints mockgen's help text...
    `go.mod` will modify because `gex` run `go get`
    

    View Slide

19. gex: manage tools with `tools.go`
    github.com/izumin5210/gex
    $ gex --add github.com/golang/mock/mockgen
    $ cat tools.go
    // Code generated by github.com/izumin5210/gex. DO NOT EDIT.
    // +build tools
    package tools
    // tool dependencies
    import (
    _ "github.com/golang/mock/mockgen"
    )
    $ cat go.mod | grep mock
    github.com/golang/mock v1.1.1 // indirect
    $ gex mockgen
    # prints mockgen's help text...
    Run versioned mockgen
    - Execute `$PWD/bin/mockgen`
    - If `$PWD/bin/mockgen` does not exist, run `go build -o mockgen ...` automatically
    

    View Slide

20. gex: manage tools with `tools.go`
    github.com/izumin5210/gex
    Does not introduce new mechanism to manage tool deps.
    - `gex` is very thin wrapper command of `dep` or `go mod`
    - For example, if `go mod` was to support to manage tools officially,

    we would be able to move from `gex` easily
    Only 2 commands that you use: `--add` and `--build`
    (I believe) tools management mechanism will be introduced

    into `golang/go` officially
    - When the time coms, we graduate from`gex`
    

    View Slide

21. How to manage tool dependencies
    golang.tokyo #20 - @izumi5210
    Let's manage tools with `gex` efficiently,
    and happy hacking with Go!
    

    View Slide

22. https://www.wantedly.com/projects/268180
    

    View Slide