Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Go Workspaces for Kubernetes

Tim Hockin
October 25, 2022

Go Workspaces for Kubernetes

My lightning talk at KubeCon NA '22 in Detroit

25 slides in a hair over 5 minutes

Tim Hockin

October 25, 2022
Tweet

More Decks by Tim Hockin

Other Decks in Technology

Transcript

  1. Paying Down Debt:
    Using Go Workspaces in Kubernetes
    KubeCon North America
    October 2022
    Tim Hockin
    @thockin

    View Slide

  2. $ ls -no staging/src/k8s.io/ | head -4
    drwxr-xr-x 26 8119 4096 Oct 8 14:52 api
    drwxr-xr-x 5 8119 4096 Oct 8 14:52 apimachinery
    drwxr-xr-x 5 8119 4096 Oct 8 14:52 apiserver
    drwxr-xr-x 23 8119 4096 Oct 8 14:52 client-go
    $ ls -no vendor/k8s.io/ | head -4
    lrwxrwxrwx 1 8119 28 Oct 10 21:23 api -> ../../staging/src/k8s.io/api
    lrwxrwxrwx 1 8119 37 Oct 10 21:23 apimachinery ->
    ../../staging/src/k8s.io/apimachinery
    lrwxrwxrwx 1 8119 34 Oct 10 21:23 apiserver -> ../../staging/src/k8s.io/apiserver
    lrwxrwxrwx 1 8119 34 Oct 10 21:23 client-go -> ../../staging/src/k8s.io/client-go
    $ find . -name go.mod | sort | head -4
    ./go.mod
    ./hack/tools/go.mod
    ./staging/src/k8s.io/api/go.mod
    ./staging/src/k8s.io/apimachinery/go.mod
    ./staging/src/k8s.io/apiserver/go.mod
    $ find . -name go.mod | wc -l
    31

    View Slide

  3. Kubernetes is not a “normal” Go project
    (and not always for great reasons)

    View Slide

  4. Once upon a time, there was a
    monorepo, and everyone was happy

    View Slide

  5. ...except people who tried to
    import “k8s.io/kubernetes”.
    Because screw them, amirite?

    View Slide

  6. Unfortunately, that included anyone
    who wanted to use our client-go library
    and API definitions.

    View Slide

  7. Actual render of Kubernetes’ deps

    View Slide

  8. View Slide

  9. $ ls -no staging/src/k8s.io/ | head -4
    drwxr-xr-x 26 8119 4096 Oct 8 14:52 api
    drwxr-xr-x 5 8119 4096 Oct 8 14:52 apimachinery
    drwxr-xr-x 5 8119 4096 Oct 8 14:52 apiserver
    drwxr-xr-x 23 8119 4096 Oct 8 14:52 client-go
    $ ls -no vendor/k8s.io/ | head -4
    lrwxrwxrwx 1 8119 28 Oct 10 21:23 api -> ../../staging/src/k8s.io/api
    lrwxrwxrwx 1 8119 37 Oct 10 21:23 apimachinery -> ../../staging/src/k8s.io/apimachinery
    lrwxrwxrwx 1 8119 34 Oct 10 21:23 apiserver -> ../../staging/src/k8s.io/apiserver
    lrwxrwxrwx 1 8119 34 Oct 10 21:23 client-go -> ../../staging/src/k8s.io/client-go

    View Slide

  10. $ find staging/src/k8s.io/code-generator/ -type f -name \*.go \
    | grep -v _test.go \
    | xargs cat | wc -l
    33416
    $ find vendor/k8s.io/gengo/ -type f -name \*.go \
    | grep -v _test.go \
    | xargs cat | wc -l
    7484
    $ ls -no hack/run-in-gopath.sh
    -rwxr-xr-x 1 8119 1164 Oct 10 21:23 hack/run-in-gopath.sh
    $ grep staging.*vendor hack/update-codegen.sh
    | sed 's|^./staging/src|vendor|'

    View Slide

  11. View Slide

  12. $ go list -m
    k8s.io/kubernetes
    $ go list ./pkg/proxy/iptables/
    k8s.io/kubernetes/pkg/proxy/iptables
    $ go list ./staging/src/k8s.io/api/core/v1
    main module (k8s.io/kubernetes) does not contain package
    k8s.io/kubernetes/staging/src/k8s.io/api/core/v1
    $ (cd ./staging/src/k8s.io/api/; go list -m; go list ./core/v1)
    k8s.io/api
    k8s.io/api/core/v1

    View Slide

  13. View Slide

  14. $ go help work | head -20
    Work provides access to operations on workspaces.
    Note that support for workspaces is built into many other commands, not
    just 'go work'.
    See 'go help modules' for information about Go's module system of which
    workspaces are a part.
    See https://go.dev/ref/mod#workspaces for an in-depth reference on
    workspaces.
    See https://go.dev/doc/tutorial/workspaces for an introductory
    tutorial on workspaces.
    A workspace is specified by a go.work file that specifies a set of
    module directories with the "use" directive. These modules are used as
    root modules by the go command for builds and related operations. A
    workspace that does not specify modules to be used cannot be used to do
    builds from local modules.

    View Slide

  15. $ head -10 go.work
    go 1.19
    use (
    .
    ./staging/src/k8s.io/api
    ./staging/src/k8s.io/apiextensions-apiserver
    ./staging/src/k8s.io/apimachinery
    ./staging/src/k8s.io/apiserver
    ./staging/src/k8s.io/cli-runtime
    ./staging/src/k8s.io/client-go
    $ go list ./pkg/proxy/iptables/
    k8s.io/kubernetes/pkg/proxy/iptables
    $ go list ./staging/src/k8s.io/api/core/v1
    k8s.io/api/core/v1

    View Slide

  16. Remember all the goop from earlier?
    It’s time to clean house!

    View Slide

  17. $ git diff
    diff --git a/hack/lib/golang.sh b/hack/lib/golang.sh
    index 3508203563e..cce9103c024 100755
    --- a/hack/lib/golang.sh
    +++ b/hack/lib/golang.sh
    @@ -565,6 +565,9 @@ kube::golang::setup_env() {
    # This seems to matter to some tools
    export GO15VENDOREXPERIMENT=1
    +
    + # Explicitly turn on modules.
    + export GO111MODULE=on
    }
    # kube::golang::setup_env will check that the `go` commands is available in

    View Slide

  18. $ git diff | diffstat -w 20
    args/args.go | 28 ++---------
    generator/execute.go | 17 ------
    generator/generator.go | 18 ++++++-
    parser/parse.go | 291 +++++++++++++++++++++++++++++++---------
    4 files changed, 292 insertions(+), 62 deletions(-)
    $ git diff | grep FIXME | wc -l
    38

    View Slide

  19. $ git diff 838012 | diffstat -w 20
    gengo2/v2/args/args.go | 3
    gengo2/v2/examples/deepcopy-gen/generators/deepcopy.go | 24 -
    gengo2/v2/examples/defaulter-gen/generators/defaulter.go | 81 +++--
    gengo2/v2/parser/parse.go | 2
    go.mod | 4
    go.work | 1
    hack/update-codegen.sh | 46 --
    kube-openapi-hack/cmd/openapi-gen/args/args.go | 6
    kube-openapi-hack/go.mod | 36 ++
    kube-openapi-hack/pkg/generators/api_linter.go | 4
    kube-openapi-hack/pkg/generators/config.go | 10
    kube-openapi-hack/pkg/generators/enum.go | 4
    kube-openapi-hack/pkg/generators/extension.go | 4
    kube-openapi-hack/pkg/generators/openapi.go | 6
    kube-openapi-hack/pkg/generators/rules/idl_tag.go | 2
    kube-openapi-hack/pkg/generators/rules/names_match.go | 2
    kube-openapi-hack/pkg/generators/rules/omitempty_match_case.go | 2
    kube-openapi-hack/pkg/generators/union.go | 2
    staging/src/k8s.io/client-go/scale/scheme/autoscalingv1/doc.go | 2
    staging/src/k8s.io/client-go/scale/scheme/extensionsv1beta1/doc.go | 2
    staging/src/k8s.io/code-generator/cmd/conversion-gen/args/args.go | 2
    staging/src/k8s.io/code-generator/cmd/conversion-gen/generators/conversion.go | 157 ++++++----
    staging/src/k8s.io/code-generator/cmd/conversion-gen/main.go | 1
    staging/src/k8s.io/code-generator/cmd/deepcopy-gen/args/args.go | 4
    staging/src/k8s.io/code-generator/cmd/deepcopy-gen/main.go | 2
    staging/src/k8s.io/code-generator/cmd/defaulter-gen/args/args.go | 4
    staging/src/k8s.io/code-generator/cmd/defaulter-gen/main.go | 2
    staging/src/k8s.io/code-generator/cmd/prerelease-lifecycle-gen/args/args.go | 2
    staging/src/k8s.io/code-generator/cmd/prerelease-lifecycle-gen/prerelease-lifecycle-generators/status.go | 11
    29 files changed, 255 insertions(+), 181 deletions(-)

    View Slide

  20. View Slide

  21. Goal: “Plausibly acceptable”

    View Slide

  22. View Slide

  23. $ head TODO.workspaces
    * Rebase frequently
    * Delete about 2/3 of gengo
    - totally overhaul how flags are passed, it doesn’t work well any more
    - lots of badness in partial-failure cases
    * Lots of newly touched code still needs test changes
    * Figure out what to do with vendoring
    * Fix all corner-cases and FIXME comments
    * Back-pop gengo changes as gengo/v2
    * Fix last “GOPATH” and “vendor” and “GO111MODULE” references
    * Try CI again (LOL, prechecks go boom!)
    * Make commits as clean as possible
    * Fix remaining hack/* scripts

    View Slide

  24. My working branch compiles and regenerates code correctly
    ● 83 commits
    ● 1404 changed files
    ● +55,748, −48,440 LOC
    ● Not done yet
    Most of that is generated files with incidental changes (whew!)
    ● Still going to be a BEAR to review
    ETA: Kubernetes v1.27.x
    Special thanks to the Go team, in particular Michael Matloob!
    Status

    View Slide

  25. x/tools/packages calls `exec(go)`, which is REALLY SLOW at our scale
    ● manifests as slow tools, forcing us towards more monolithic codegen
    ● also manifests as slow and memory-hoggy gopls
    ● OTOH: slowness prompted a huge Makefile simplification
    Lack of support for vendoring leaves us to figure it out on our own
    ● Go’s module proxy does not guarantee “forever” cache
    ● We have been burned by modules disappearing
    ● Use-case: clone repo, get on a plane, hack -- does not work now!
    Issues

    View Slide

  26. Thank you

    View Slide