Kubernetes API Codebase Tour

1. API Codebase Tour Stefan Schimanski / @the_sttts / Red Hat Hacking the kube-apiserver
2. Defining API types
4. v1alpha1 types: staging/src/k8s.io/api/auditregistration/v1alpha1 • types.go – actual Golang types (with JSON and Proto tags) • register.go – registration code: AddToScheme internal types: pkg/apis/auditregistration • types.go – internal (hub) Golang types (without JSON/Proto) • register.go – registration code: AddToScheme Installer: pkg/apis/auditregistration/install: func Install(scheme *runtime.Scheme) Golang types
5. Scheme: register Golang types & Golang funcs w/ GroupVersionKind k8s.io/apimachinery/pkg/runtime.Scheme GroupVersionKinds conversions defaulters reflect.Type Scheme Codec
6. v1alpha1 types: staging/src/k8s.io/api/auditregistration/v1alpha1 • types.go – actual Golang types (with JSON and Proto tags) • register.go – registration code: AddToScheme internal types: pkg/apis/auditregistration • types.go – internal (hub) Golang types (without JSON/Proto) • register.go – registration code: AddToScheme Installer: pkg/apis/auditregistration/install: func Install(scheme *runtime.Scheme) Golang types
7. Conversions: pkg/apis/auditregistration/v1alpha1 • conversion.go – custom conversions • zz_generated.conversion.go – generated conversions Defaults: zz_generated_defaults.go DeepCopy: zz_generated_deepcopy.go Generated Code not in k8s.io/api!
8. Serving the API
9. apiserver binary generic apiserver in k8s.io/apiserver 404 authentication authorization impersonation panic recovery request-timeout audit max-in-flight handler chain mux data flow calls back to knows no API groups yet Scheme empty /version /apis /openapi/v2 /swagger.json /healthz /metrics
10. apiserver binary generic apiserver in k8s.io/apiserver 404 authentication authorization impersonation panic recovery request-timeout audit max-in-flight handler chain mux data flow calls back to knows no API groups yet Scheme empty /version /apis /openapi/v2 /swagger.json /healthz /metrics
11. func DefaultBuildHandlerChain(apiHandler http.Handler, c *Config) http.Handler { handler := genericapifilters.WithAuthorization(apiHandler, ...) handler = genericfilters.WithMaxInFlightLimit(handler, ...) handler = genericapifilters.WithImpersonation(handler, ...) handler = genericapifilters.WithAudit(handler, ...) failedHandler := genericapifilters.Unauthorized(...) failedHandler = genericapifilters.WithFailedAuthenticationAudit(failedHandler, ...) handler = genericapifilters.WithAuthentication(handler, ..., failedHandler, ...) handler = genericfilters.WithCORS(handler, ...) handler = genericfilters.WithTimeoutForNonLongRunningRequests(handler, ...) handler = genericfilters.WithWaitGroup(handler, ...) handler = genericapifilters.WithRequestInfo(handler, ...) handler = genericfilters.WithPanicRecovery(handler) return handler } k8s.io/apiserver/pkg/server/config.go
12. kube-apiserver generic apiserver 404 authentication authorization impersonation panic recovery request-timeout audit max-in-flight handler chain mux data flow calls back to knows no API groups yet Scheme /version /apis /openapi/v2 /swagger.json /healthz /metrics core/v1 Pod core/v1 Pod core/v1 Pod
13. kube-apiserver apiserver 404 resource handler request conversion & defaulting REST logic result conversion validation admission decoding GET CREATE LIST UPDATE DELETE WATCH PATCH encoding mutating webhooks validating webhooks authentication authorization impersonation panic recovery request-timeout audit max-in-flight handler chain mux data flow calls back to Scheme core/v1 Pod core/v1 Pod core/v1 Pod via InstallAPIGroup(info)
14. kube-apiserver apiserver resource handler resource handler 404 resource handler request conversion & defaulting REST logic result conversion validation admission decoding GET CREATE LIST UPDATE DELETE WATCH PATCH encoding mutating webhooks validating webhooks authentication authorization impersonation panic recovery request-timeout audit max-in-flight handler chain Scheme core/v1 Pod core/v1 Pod core/v1 Pod data flow calls back to mux no storage logic yet
15. kube-apiserver apiserver resource handler resource handler 404 etcd resource handler request conversion & defaulting storage conversion & defaulting REST logic result conversion validation admission decoding GET CREATE LIST UPDATE DELETE WATCH PATCH encoding mutating webhooks validating webhooks authentication authorization impersonation panic recovery request-timeout audit max-in-flight handler chain API Group “core” API Group “core” API Group “core” PodStorage PodStorage PodStorage Generic Registry Pod Strategy - PrepareForUpdate - PrepareForCreate - Validate ... create update ... mux Scheme core/v1 Pod core/v1 Pod core/v1 Pod data flow calls back to
16. kube-apiserver apiserver resource handler resource handler 404 etcd resource handler request conversion & defaulting storage conversion & defaulting REST logic result conversion validation admission decoding GET CREATE LIST UPDATE DELETE WATCH PATCH encoding mutating webhooks validating webhooks authentication authorization impersonation panic recovery request-timeout audit max-in-flight handler chain API Group “core” API Group “core” API Group “core” PodStorage PodStorage PodStorage Generic Registry Pod Strategy - PrepareForUpdate - PrepareForCreate - Validate ... create update ... mux Scheme core/v1 Pod core/v1 Pod core/v1 Pod data flow calls back to v1 v1 v1 int int v1 int v1 int v2 v1 int int v1 hub/internal version
17. kube-apiserver apiserver resource handler resource handler 404 etcd resource handler request conversion & defaulting storage conversion & defaulting REST logic result conversion validation admission decoding GET CREATE LIST UPDATE DELETE WATCH PATCH encoding mutating webhooks validating webhooks authentication authorization impersonation panic recovery request-timeout audit max-in-flight handler chain API Group “core” API Group “core” API Group “core” PodStorage PodStorage PodStorage Generic Registry Pod Strategy - PrepareForUpdate - PrepareForCreate - Validate ... create update ... mux Scheme core/v1 Pod core/v1 Pod core/v1 Pod data flow calls back to conversions defaulting
18. kube-apiserver CRDs aggregator kube- aggregator & CRDs apiserver resource handler resource handler 404 etcd aggregated apiservers resource handler request conversion & defaulting storage conversion & defaulting REST logic result conversion validation admission decoding GET CREATE LIST UPDATE DELETE WATCH PATCH encoding mutating webhooks validating webhooks authentication authorization impersonation panic recovery request-timeout audit max-in-flight handler chain API Group “core” API Group “core” API Group “core” PodStorage PodStorage PodStorage Generic Registry Pod Strategy - PrepareForUpdate - PrepareForCreate - Validate ... create update ... mux Scheme core/v1 Pod core/v1 Pod core/v1 Pod data flow calls back to
19. kube-apiserver kube- aggregator apiserver resource handler resource handler 404 etcd aggregated apiservers resource handler request conversion & defaulting storage conversion & defaulting REST logic result conversion validation admission decoding GET CREATE LIST UPDATE DELETE WATCH PATCH encoding mutating webhooks validating webhooks authentication authorization impersonation panic recovery request-timeout audit max-in-flight handler chain API Group “core” API Group “core” API Group “core” PodStorage PodStorage PodStorage Generic Registry Pod Strategy - PrepareForUpdate - PrepareForCreate - Validate ... create update ... mux Scheme core/v1 Pod core/v1 Pod core/v1 Pod data flow calls back to pkg/registry pkg/apis + k8s.io/api k8s.io/apiserver/pkg/endpoints/handlers k8s.io/apiserver/pkg/admission k8s.io/apiserver/plugin/pkg/admission plugins/pkg/admission k8s.io/apiserver/pkg/endpoints/filters k8s.io/kube-aggregator k8s.io/apiextensions-apiserver k8s.io/apiserver/pkg/storage/etcd3 k8s.io/apiserver/pkg/registry/generic
20. API Group “core” API Group “core” API Group “auditregistration.k8s.io” PodStorage PodStorage AuditSinkStorage Generic Registry AuditSink Strategy - PrepareForUpdate - PrepareForCreate - Validate ... create update ... staging/src/k8s.io/apiserver/pkg/registry/generic/registry pkg/apis/auditregistration/validation “The registry” of a resource
21. Plumbing into kube-apiserver pkg/master/import_known_versions.go import ( _ "k8s.io/kubernetes/pkg/apis/auditregistration/install" ) pkg/master/master.go import ( auditregistrationrest "k8s.io/kubernetes/pkg/registry/auditregistration/rest" ) restStorageProviders := []RESTStorageProvider{ auditregistrationrest.RESTStorageProvider{}, autoscalingrest.RESTStorageProvider{}, … } apiserver.InstallAPIs(…, restStorageProviders…) legacyscheme.Scheme installs handlers into the mux func init()
22. Build system plumbing • hack/.golint_failures ignore lint errors due to generated code • hack/lib/init.sh add to KUBE_AVAILABLE_GROUP_VERSIONS, used by many hack/ scripts • hack/update-generated-protobuf-dockerized.sh generate Protobuf code, independent from KUBE_AVAILABLE_GROUP_VERSIONS for some reason
23. $ make WHAT=cmd/hyperkube $ RUNTIME_CONFIG=auditregistration.k8s.io/v1alpha1=true \ hack/local-up-cluster.sh $ kubectl get --raw /apis | grep auditregistration.k8s.io
24. Live Debugging
25. kube-apiserver CRDs aggregator kube- aggregator & CRDs apiserver resource handler resource handler 404 etcd aggregated apiservers resource handler request conversion & defaulting storage conversion & defaulting REST logic result conversion validation admission decoding GET CREATE LIST UPDATE DELETE WATCH PATCH encoding mutating webhooks validating webhooks authentication authorization impersonation panic recovery request-timeout audit max-in-flight handler chain API Group “core” API Group “core” API Group “core” PodStorage PodStorage PodStorage Generic Registry Pod Strategy - PrepareForUpdate - PrepareForCreate - Validate ... create update ... mux Scheme core/v1 Pod core/v1 Pod core/v1 Pod data flow calls back to
26. Live Debugging * perfectly written down in xmudrii’s https://xmudrii.com/posts/debugging-kubernetes/ *
27. kube-apiserver CRDs aggregator kube- aggregator & CRDs apiserver resource handler resource handler 404 etcd aggregated apiservers resource handler request conversion & defaulting storage conversion & defaulting REST logic result conversion validation admission decoding GET CREATE LIST UPDATE DELETE WATCH PATCH encoding mutating webhooks validating webhooks authentication authorization impersonation panic recovery request-timeout audit max-in-flight handler chain API Group “core” API Group “core” API Group “core” PodStorage PodStorage PodStorage Generic Registry Pod Strategy - PrepareForUpdate - PrepareForCreate - Validate ... create update ... mux Scheme core/v1 Pod core/v1 Pod core/v1 Pod data flow calls back to
28. @lavalamp’s “Live API Code Review” after the break