Go Workspaces for Kubernetes

My lightning talk at KubeCon NA '22 in Detroit

25 slides in a hair over 5 minutes

Transcript

1. Paying Down Debt:
   Using Go Workspaces in Kubernetes
   KubeCon North America
   October 2022
   Tim Hockin
   @thockin
   

   View Slide

2. $ ls -no staging/src/k8s.io/ | head -4
   drwxr-xr-x 26 8119 4096 Oct 8 14:52 api
   drwxr-xr-x 5 8119 4096 Oct 8 14:52 apimachinery
   drwxr-xr-x 5 8119 4096 Oct 8 14:52 apiserver
   drwxr-xr-x 23 8119 4096 Oct 8 14:52 client-go
   $ ls -no vendor/k8s.io/ | head -4
   lrwxrwxrwx 1 8119 28 Oct 10 21:23 api -> ../../staging/src/k8s.io/api
   lrwxrwxrwx 1 8119 37 Oct 10 21:23 apimachinery ->
   ../../staging/src/k8s.io/apimachinery
   lrwxrwxrwx 1 8119 34 Oct 10 21:23 apiserver -> ../../staging/src/k8s.io/apiserver
   lrwxrwxrwx 1 8119 34 Oct 10 21:23 client-go -> ../../staging/src/k8s.io/client-go
   $ find . -name go.mod | sort | head -4
   ./go.mod
   ./hack/tools/go.mod
   ./staging/src/k8s.io/api/go.mod
   ./staging/src/k8s.io/apimachinery/go.mod
   ./staging/src/k8s.io/apiserver/go.mod
   $ find . -name go.mod | wc -l
   31
   

   View Slide

3. Kubernetes is not a “normal” Go project
   (and not always for great reasons)
   

   View Slide

4. Once upon a time, there was a
   monorepo, and everyone was happy
   

   View Slide

5. ...except people who tried to
   import “k8s.io/kubernetes”.
   Because screw them, amirite?
   

   View Slide

6. Unfortunately, that included anyone
   who wanted to use our client-go library
   and API definitions.
   

   View Slide

7. Actual render of Kubernetes’ deps
   

   View Slide

8. View Slide

9. $ ls -no staging/src/k8s.io/ | head -4
   drwxr-xr-x 26 8119 4096 Oct 8 14:52 api
   drwxr-xr-x 5 8119 4096 Oct 8 14:52 apimachinery
   drwxr-xr-x 5 8119 4096 Oct 8 14:52 apiserver
   drwxr-xr-x 23 8119 4096 Oct 8 14:52 client-go
   $ ls -no vendor/k8s.io/ | head -4
   lrwxrwxrwx 1 8119 28 Oct 10 21:23 api -> ../../staging/src/k8s.io/api
   lrwxrwxrwx 1 8119 37 Oct 10 21:23 apimachinery -> ../../staging/src/k8s.io/apimachinery
   lrwxrwxrwx 1 8119 34 Oct 10 21:23 apiserver -> ../../staging/src/k8s.io/apiserver
   lrwxrwxrwx 1 8119 34 Oct 10 21:23 client-go -> ../../staging/src/k8s.io/client-go
   

   View Slide

10. $ find staging/src/k8s.io/code-generator/ -type f -name \*.go \
    | grep -v _test.go \
    | xargs cat | wc -l
    33416
    $ find vendor/k8s.io/gengo/ -type f -name \*.go \
    | grep -v _test.go \
    | xargs cat | wc -l
    7484
    $ ls -no hack/run-in-gopath.sh
    -rwxr-xr-x 1 8119 1164 Oct 10 21:23 hack/run-in-gopath.sh
    $ grep staging.*vendor hack/update-codegen.sh
    | sed 's|^./staging/src|vendor|'
    

    View Slide

11. View Slide

12. $ go list -m
    k8s.io/kubernetes
    $ go list ./pkg/proxy/iptables/
    k8s.io/kubernetes/pkg/proxy/iptables
    $ go list ./staging/src/k8s.io/api/core/v1
    main module (k8s.io/kubernetes) does not contain package
    k8s.io/kubernetes/staging/src/k8s.io/api/core/v1
    $ (cd ./staging/src/k8s.io/api/; go list -m; go list ./core/v1)
    k8s.io/api
    k8s.io/api/core/v1
    

    View Slide

13. View Slide

14. $ go help work | head -20
    Work provides access to operations on workspaces.
    Note that support for workspaces is built into many other commands, not
    just 'go work'.
    See 'go help modules' for information about Go's module system of which
    workspaces are a part.
    See https://go.dev/ref/mod#workspaces for an in-depth reference on
    workspaces.
    See https://go.dev/doc/tutorial/workspaces for an introductory
    tutorial on workspaces.
    A workspace is specified by a go.work file that specifies a set of
    module directories with the "use" directive. These modules are used as
    root modules by the go command for builds and related operations. A
    workspace that does not specify modules to be used cannot be used to do
    builds from local modules.
    

    View Slide

15. $ head -10 go.work
    go 1.19
    use (
    .
    ./staging/src/k8s.io/api
    ./staging/src/k8s.io/apiextensions-apiserver
    ./staging/src/k8s.io/apimachinery
    ./staging/src/k8s.io/apiserver
    ./staging/src/k8s.io/cli-runtime
    ./staging/src/k8s.io/client-go
    $ go list ./pkg/proxy/iptables/
    k8s.io/kubernetes/pkg/proxy/iptables
    $ go list ./staging/src/k8s.io/api/core/v1
    k8s.io/api/core/v1
    

    View Slide

16. Remember all the goop from earlier?
    It’s time to clean house!
    

    View Slide

17. $ git diff
    diff --git a/hack/lib/golang.sh b/hack/lib/golang.sh
    index 3508203563e..cce9103c024 100755
    --- a/hack/lib/golang.sh
    +++ b/hack/lib/golang.sh
    @@ -565,6 +565,9 @@ kube::golang::setup_env() {
    # This seems to matter to some tools
    export GO15VENDOREXPERIMENT=1
    +
    + # Explicitly turn on modules.
    + export GO111MODULE=on
    }
    # kube::golang::setup_env will check that the `go` commands is available in
    

    View Slide

18. $ git diff | diffstat -w 20
    args/args.go | 28 ++---------
    generator/execute.go | 17 ------
    generator/generator.go | 18 ++++++-
    parser/parse.go | 291 +++++++++++++++++++++++++++++++---------
    4 files changed, 292 insertions(+), 62 deletions(-)
    $ git diff | grep FIXME | wc -l
    38
    

    View Slide

19. $ git diff 838012 | diffstat -w 20
    gengo2/v2/args/args.go | 3
    gengo2/v2/examples/deepcopy-gen/generators/deepcopy.go | 24 -
    gengo2/v2/examples/defaulter-gen/generators/defaulter.go | 81 +++--
    gengo2/v2/parser/parse.go | 2
    go.mod | 4
    go.work | 1
    hack/update-codegen.sh | 46 --
    kube-openapi-hack/cmd/openapi-gen/args/args.go | 6
    kube-openapi-hack/go.mod | 36 ++
    kube-openapi-hack/pkg/generators/api_linter.go | 4
    kube-openapi-hack/pkg/generators/config.go | 10
    kube-openapi-hack/pkg/generators/enum.go | 4
    kube-openapi-hack/pkg/generators/extension.go | 4
    kube-openapi-hack/pkg/generators/openapi.go | 6
    kube-openapi-hack/pkg/generators/rules/idl_tag.go | 2
    kube-openapi-hack/pkg/generators/rules/names_match.go | 2
    kube-openapi-hack/pkg/generators/rules/omitempty_match_case.go | 2
    kube-openapi-hack/pkg/generators/union.go | 2
    staging/src/k8s.io/client-go/scale/scheme/autoscalingv1/doc.go | 2
    staging/src/k8s.io/client-go/scale/scheme/extensionsv1beta1/doc.go | 2
    staging/src/k8s.io/code-generator/cmd/conversion-gen/args/args.go | 2
    staging/src/k8s.io/code-generator/cmd/conversion-gen/generators/conversion.go | 157 ++++++----
    staging/src/k8s.io/code-generator/cmd/conversion-gen/main.go | 1
    staging/src/k8s.io/code-generator/cmd/deepcopy-gen/args/args.go | 4
    staging/src/k8s.io/code-generator/cmd/deepcopy-gen/main.go | 2
    staging/src/k8s.io/code-generator/cmd/defaulter-gen/args/args.go | 4
    staging/src/k8s.io/code-generator/cmd/defaulter-gen/main.go | 2
    staging/src/k8s.io/code-generator/cmd/prerelease-lifecycle-gen/args/args.go | 2
    staging/src/k8s.io/code-generator/cmd/prerelease-lifecycle-gen/prerelease-lifecycle-generators/status.go | 11
    29 files changed, 255 insertions(+), 181 deletions(-)
    

    View Slide

20. View Slide

21. Goal: “Plausibly acceptable”
    

    View Slide

22. View Slide

23. $ head TODO.workspaces
    * Rebase frequently
    * Delete about 2/3 of gengo
    - totally overhaul how flags are passed, it doesn’t work well any more
    - lots of badness in partial-failure cases
    * Lots of newly touched code still needs test changes
    * Figure out what to do with vendoring
    * Fix all corner-cases and FIXME comments
    * Back-pop gengo changes as gengo/v2
    * Fix last “GOPATH” and “vendor” and “GO111MODULE” references
    * Try CI again (LOL, prechecks go boom!)
    * Make commits as clean as possible
    * Fix remaining hack/* scripts
    

    View Slide

24. My working branch compiles and regenerates code correctly
    ● 83 commits
    ● 1404 changed files
    ● +55,748, −48,440 LOC
    ● Not done yet
    Most of that is generated files with incidental changes (whew!)
    ● Still going to be a BEAR to review
    ETA: Kubernetes v1.27.x
    Special thanks to the Go team, in particular Michael Matloob!
    Status
    

    View Slide

25. x/tools/packages calls `exec(go)`, which is REALLY SLOW at our scale
    ● manifests as slow tools, forcing us towards more monolithic codegen
    ● also manifests as slow and memory-hoggy gopls
    ● OTOH: slowness prompted a huge Makefile simplification
    Lack of support for vendoring leaves us to figure it out on our own
    ● Go’s module proxy does not guarantee “forever” cache
    ● We have been burned by modules disappearing
    ● Use-case: clone repo, get on a plane, hack -- does not work now!
    Issues
    

    View Slide

26. Thank you
    

    View Slide