Paying Down Debt: Using Go Workspaces in Kubernetes KubeCon North America October 2022 Tim Hockin @thockin

$ ls -no staging/src/k8s.io/ | head -4 drwxr-xr-x 26 8119 4096 Oct 8 14:52 api drwxr-xr-x 5 8119 4096 Oct 8 14:52 apimachinery drwxr-xr-x 5 8119 4096 Oct 8 14:52 apiserver drwxr-xr-x 23 8119 4096 Oct 8 14:52 client-go $ ls -no vendor/k8s.io/ | head -4 lrwxrwxrwx 1 8119 28 Oct 10 21:23 api -> ../../staging/src/k8s.io/api lrwxrwxrwx 1 8119 37 Oct 10 21:23 apimachinery -> ../../staging/src/k8s.io/apimachinery lrwxrwxrwx 1 8119 34 Oct 10 21:23 apiserver -> ../../staging/src/k8s.io/apiserver lrwxrwxrwx 1 8119 34 Oct 10 21:23 client-go -> ../../staging/src/k8s.io/client-go $ find . -name go.mod | sort | head -4 ./go.mod ./hack/tools/go.mod ./staging/src/k8s.io/api/go.mod ./staging/src/k8s.io/apimachinery/go.mod ./staging/src/k8s.io/apiserver/go.mod $ find . -name go.mod | wc -l 31

Kubernetes is not a “normal” Go project (and not always for great reasons)

Once upon a time, there was a monorepo, and everyone was happy

...except people who tried to import “k8s.io/kubernetes”. Because screw them, amirite?

Unfortunately, that included anyone who wanted to use our client-go library and API definitions.

Actual render of Kubernetes’ deps

No content

$ ls -no staging/src/k8s.io/ | head -4 drwxr-xr-x 26 8119 4096 Oct 8 14:52 api drwxr-xr-x 5 8119 4096 Oct 8 14:52 apimachinery drwxr-xr-x 5 8119 4096 Oct 8 14:52 apiserver drwxr-xr-x 23 8119 4096 Oct 8 14:52 client-go $ ls -no vendor/k8s.io/ | head -4 lrwxrwxrwx 1 8119 28 Oct 10 21:23 api -> ../../staging/src/k8s.io/api lrwxrwxrwx 1 8119 37 Oct 10 21:23 apimachinery -> ../../staging/src/k8s.io/apimachinery lrwxrwxrwx 1 8119 34 Oct 10 21:23 apiserver -> ../../staging/src/k8s.io/apiserver lrwxrwxrwx 1 8119 34 Oct 10 21:23 client-go -> ../../staging/src/k8s.io/client-go

$ find staging/src/k8s.io/code-generator/ -type f -name \*.go \ | grep -v _test.go \ | xargs cat | wc -l 33416 $ find vendor/k8s.io/gengo/ -type f -name \*.go \ | grep -v _test.go \ | xargs cat | wc -l 7484 $ ls -no hack/run-in-gopath.sh -rwxr-xr-x 1 8119 1164 Oct 10 21:23 hack/run-in-gopath.sh $ grep staging.*vendor hack/update-codegen.sh | sed 's|^./staging/src|vendor|'

No content

$ go list -m k8s.io/kubernetes $ go list ./pkg/proxy/iptables/ k8s.io/kubernetes/pkg/proxy/iptables $ go list ./staging/src/k8s.io/api/core/v1 main module (k8s.io/kubernetes) does not contain package k8s.io/kubernetes/staging/src/k8s.io/api/core/v1 $ (cd ./staging/src/k8s.io/api/; go list -m; go list ./core/v1) k8s.io/api k8s.io/api/core/v1

No content

$ go help work | head -20 Work provides access to operations on workspaces. Note that support for workspaces is built into many other commands, not just 'go work'. See 'go help modules' for information about Go's module system of which workspaces are a part. See https://go.dev/ref/mod#workspaces for an in-depth reference on workspaces. See https://go.dev/doc/tutorial/workspaces for an introductory tutorial on workspaces. A workspace is specified by a go.work file that specifies a set of module directories with the "use" directive. These modules are used as root modules by the go command for builds and related operations. A workspace that does not specify modules to be used cannot be used to do builds from local modules.

$ head -10 go.work go 1.19 use ( . ./staging/src/k8s.io/api ./staging/src/k8s.io/apiextensions-apiserver ./staging/src/k8s.io/apimachinery ./staging/src/k8s.io/apiserver ./staging/src/k8s.io/cli-runtime ./staging/src/k8s.io/client-go $ go list ./pkg/proxy/iptables/ k8s.io/kubernetes/pkg/proxy/iptables $ go list ./staging/src/k8s.io/api/core/v1 k8s.io/api/core/v1

Remember all the goop from earlier? It’s time to clean house!

$ git diff diff --git a/hack/lib/golang.sh b/hack/lib/golang.sh index 3508203563e..cce9103c024 100755 --- a/hack/lib/golang.sh +++ b/hack/lib/golang.sh @@ -565,6 +565,9 @@ kube::golang::setup_env() { # This seems to matter to some tools export GO15VENDOREXPERIMENT=1 + + # Explicitly turn on modules. + export GO111MODULE=on } # kube::golang::setup_env will check that the `go` commands is available in

$ git diff | diffstat -w 20 args/args.go | 28 ++--------- generator/execute.go | 17 ------ generator/generator.go | 18 ++++++- parser/parse.go | 291 +++++++++++++++++++++++++++++++--------- 4 files changed, 292 insertions(+), 62 deletions(-) $ git diff | grep FIXME | wc -l 38

$ git diff 838012 | diffstat -w 20 gengo2/v2/args/args.go | 3 gengo2/v2/examples/deepcopy-gen/generators/deepcopy.go | 24 - gengo2/v2/examples/defaulter-gen/generators/defaulter.go | 81 +++-- gengo2/v2/parser/parse.go | 2 go.mod | 4 go.work | 1 hack/update-codegen.sh | 46 -- kube-openapi-hack/cmd/openapi-gen/args/args.go | 6 kube-openapi-hack/go.mod | 36 ++ kube-openapi-hack/pkg/generators/api_linter.go | 4 kube-openapi-hack/pkg/generators/config.go | 10 kube-openapi-hack/pkg/generators/enum.go | 4 kube-openapi-hack/pkg/generators/extension.go | 4 kube-openapi-hack/pkg/generators/openapi.go | 6 kube-openapi-hack/pkg/generators/rules/idl_tag.go | 2 kube-openapi-hack/pkg/generators/rules/names_match.go | 2 kube-openapi-hack/pkg/generators/rules/omitempty_match_case.go | 2 kube-openapi-hack/pkg/generators/union.go | 2 staging/src/k8s.io/client-go/scale/scheme/autoscalingv1/doc.go | 2 staging/src/k8s.io/client-go/scale/scheme/extensionsv1beta1/doc.go | 2 staging/src/k8s.io/code-generator/cmd/conversion-gen/args/args.go | 2 staging/src/k8s.io/code-generator/cmd/conversion-gen/generators/conversion.go | 157 ++++++---- staging/src/k8s.io/code-generator/cmd/conversion-gen/main.go | 1 staging/src/k8s.io/code-generator/cmd/deepcopy-gen/args/args.go | 4 staging/src/k8s.io/code-generator/cmd/deepcopy-gen/main.go | 2 staging/src/k8s.io/code-generator/cmd/defaulter-gen/args/args.go | 4 staging/src/k8s.io/code-generator/cmd/defaulter-gen/main.go | 2 staging/src/k8s.io/code-generator/cmd/prerelease-lifecycle-gen/args/args.go | 2 staging/src/k8s.io/code-generator/cmd/prerelease-lifecycle-gen/prerelease-lifecycle-generators/status.go | 11 29 files changed, 255 insertions(+), 181 deletions(-)

No content

Goal: “Plausibly acceptable”

No content

$ head TODO.workspaces * Rebase frequently * Delete about 2/3 of gengo - totally overhaul how flags are passed, it doesn’t work well any more - lots of badness in partial-failure cases * Lots of newly touched code still needs test changes * Figure out what to do with vendoring * Fix all corner-cases and FIXME comments * Back-pop gengo changes as gengo/v2 * Fix last “GOPATH” and “vendor” and “GO111MODULE” references * Try CI again (LOL, prechecks go boom!) * Make commits as clean as possible * Fix remaining hack/* scripts

My working branch compiles and regenerates code correctly ● 83 commits ● 1404 changed files ● +55,748, −48,440 LOC ● Not done yet Most of that is generated files with incidental changes (whew!) ● Still going to be a BEAR to review ETA: Kubernetes v1.27.x Special thanks to the Go team, in particular Michael Matloob! Status

x/tools/packages calls `exec(go)`, which is REALLY SLOW at our scale ● manifests as slow tools, forcing us towards more monolithic codegen ● also manifests as slow and memory-hoggy gopls ● OTOH: slowness prompted a huge Makefile simplification Lack of support for vendoring leaves us to figure it out on our own ● Go’s module proxy does not guarantee “forever” cache ● We have been burned by modules disappearing ● Use-case: clone repo, get on a plane, hack -- does not work now! Issues

Thank you