Left Shifting Validation Logic in Kubernetes Operators

Transcript

1. Left Shifting Validation Logic in Kubernetes Operators WELCOME Sergiusz Urbaniak Team Lead, Kubernetes Atlas Operator Roo Thorp Software Engineer, Kubernetes Atlas Operator
2. What is Data Validation? https://en.wikipedia.org/wiki/Data_validation
3. • Kubernetes offers declarative APIs • New Kubernetes APIs can be declared via Custom Resource Definitions (CRDs) • CRDs encode the schema of our API • We want to validate inputs for our API What are we validating actually?
4. The anatomy of a CRD apiVersion: my.domain/v1 kind: MyCustomResource metadata: name: in-operator-check namespace: default spec: field: value status: conditions: - status: True type: Ready Type information Entity information identity of a resource Specification (desired state) controlled by the user (producer) Status information controlled by the operator (consumer)
5. The anatomy of a CRD apiVersion: my.domain/v1 kind: MyCustomResource metadata: name: in-operator-check namespace: default spec: field: value status: conditions: - status: True type: Ready This is our validation target
6. What are our validation options? Validation Strategy Kubernetes Version Required Code Change Necessary? No Validation * No Custom status based validation * Yes OpenAPI Validation GA as 1.8 Yes Complete structural schema GA as of 1.16 Yes Common Expression Language GA as of 1.29 Yes Validating Admission Webhooks * Yes Validating Admission Policies GA as of 1.30 No CRD Ratcheting Beta as of 1.30, enabled by default No
7. Store arbitrary key/value pairs in custom resources No Validation $ cat <<EOF | kubectl apply -f - apiVersion: my.domain/v1 kind: NoValidation metadata: name: simple-json spec: foo: fooValue EOF $ cat <<EOF | kubectl apply -f - apiVersion: my.domain/v1 kind: NoValidation metadata: name: deep-json spec: lets: go: crazy: with: this: value EOF $ cat <<EOF | kubectl apply -f - apiVersion: my.domain/v1 kind: NoValidation metadata: name: gibberish spec: | Arbitrary value, even multiline invalid JSON :) EOF
8. No Validation - Danger Zone base64( ) $ cat <<EOF | kubectl apply -f - apiVersion: my.domain/v1 kind: NoValidation metadata: name: no-validation-check-foo spec: godzilla-movie: R0lGQps5v…AEBADs= EOF
9. Submission of faulty data Security concerns Operator needs to smart Operator needs to be tolerant Drawbacks
10. • Switching gears … • Operator can be “opinionated” on what constraints are allowed No Validation Status based validation $ cat <<EOF | kubectl apply -f - apiVersion: my.domain/v1 kind: StatusBasedValidation metadata: name: in-operator-check spec: foo: 123 EOF Challenge • Validate input field “foo” is a number • Fail validation if field is “invalid”
11. Status based validation apiVersion: my.domain/v1 kind: StatusBasedValidation metadata: name: status-based-invalid namespace: default spec: foo: invalid status: conditions: - message: 'The spec is invalid: It must contain a field "foo" of type "int"' reason: InvalidSpec status: "False" type: Ready
12. Status based validation - Danger Zone $ cat <<EOF | kubectl apply -f - apiVersion: my.domain/v1 kind: StatusBasedValidation metadata: name: in-operator-check spec: foo: 123 # valid field godzilla-movie: R0lGQps5v…AEBADs= # unknown field EOF JSON marshalling (in Go) ignores unknown fields!
13. Status Polling Late schema checks Lots of boilerplate code Injection of unknown fields Drawbacks
14. A fresh start with three ingredients Don’t wait for status conditions. Get feedback when executing “kubectl apply”. Left-shift Validation No custom code for schema validation. Reuse OpenAPI OpenAPI Validation Don’t ignore unknown fields. Complete Structural Schema …. and more!
15. Left shifting validation logic - Benefits CRD intrinsics External API Validation Validation left-shift Operator Fails during kubectl apply Fails after kubectl apply User
16. Why left shift validation logic? Failures during kubectl apply • Resource is not stored in Kubernetes • Easier GitOps flows • Enhanced UX • Early feedback to the user • “kubectl —dry-run” works Failures after kubectl apply • Invalid resource is stored in Kubernetes • Complex GitOps flows ◦ Need to check status • Worse UX • Late feedback to the user • “kubectl —dry-run” does not work
17. Complete Structural Schema • Reject unknown fields • Enables “pruning” Cleanup of unknown fields • Enabled by default! • Can opt-out at field-level i.e. JSON fields $ cat <<EOF | kubectl apply -f - apiVersion: my.domain/v1 kind: LeftShifted metadata: name: foo spec: foo: 123 # known field godzilla-movie: R0lGQps5v…AEBADs= # UNKNOWN field, REJECTED EOF The request is invalid: patch: Invalid value: … strict decoding error: unknown field "spec.godzilla-movie"
18. OpenAPI support • CRDs are translated into OpenAPI specifications type MyCRD struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec MyCRDSpec `json:"spec,omitempty"` } type MyCRDSpec struct { Foo int `json:"foo,omitempty"` }
19. OpenAPI support • CRDs are translated into OpenAPI specifications type MyCRD struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec MyCRDSpec `json:"spec,omitempty"` } type MyCRDSpec struct { // +kubebuilder:validation:Minimum=1 // +kubebuilder:validation:Maximum=10 Foo int `json:"foo,omitempty"` } Available OpenAPI Validation Tags: https://book.kubebuilder.io/reference/markers/crd-validation
20. Validating Admission Webhooks • Uses webhooks to inspect requests before they reach the Kube API • Validation happens as a Go function(s) • Potentially very powerful and flexible • Useful for more complex data types and validation • Was the primary alternative to OpenAPI validations (until recently) Network call! Network call!
21. Network failures result in rejection Errors in webhook result in rejection Webhook availability becomes a SPoF Lots of code to write (and deploy!) Drawbacks
22. Common Expression Language • CEL is a simple language for expression evaluation • Extends the current OpenAPI validation • Allows for more complex validation logic than built in OpenAPI validation • Makes use of the new x-kubernetes-validations extension • Can be defined using kubebuilder annotations, to fit nicely into current tooling // +kubebuilder:validation:XValidation:rule=(self.minReplicas <= self.replicas),message=”Replicas must be less than or equal to minimum replicas” …becomes… openAPIV3Schema: type: object properties: spec: type: object x-kubernetes-validations: - rule: "self.minReplicas <= self.replicas" - message: "Replicas must be less than or equal to minimum replicas"
23. Common Expression Language Make references to the current state using oldSelf… // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="Value is immutable" Use built in functions for more complex rules… // +kubebuilder:validation:XValidation:rule="oldSelf.all(x, x in self)",message="Values may only be added" … or combine them for even more complex cases! // +kubebuilder:validation:XValidation:rule=”self.envars.filter(e, e.name == 'MY_ENV').all(e, e.value.matches('^[a-zA-Z]*$')”,message=”” Extend the Kubernetes API with CustomResourceDefinitions: Validation Rules CRD Validation - The Kubebuilder Book Google CEL Specification
24. Testing CEL
25. Validating Admission Policies • Rules defined using CEL • Defined as a YAML resource, so can be included in an existing CRD bundle • Can be written and shared by Operator developers and users Spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: ["apps"] apiVersions: ["v1"] operations: ["CREATE", "UPDATE"] resources: ["deployments"] Validations: - expression: "object.spec.replicas <= params.maxReplicas" messageExpression: "'object.spec.replicas must be no greater than ' + string(params.maxReplicas)" reason: Invalid # Determines the HTTP Response Code Validating Admission Policy | Kubernetes
26. CRD Ratcheting • CRD ratcheting allows an update to proceed on an invalid resource, provided the change did not affect the invalid fields. ◦ “any invalid part of the resource that remains invalid must have already been wrong” • This means we can enforce or even tighten validation going forward, without breaking current configurations. • While this is not validation in itself, it makes implementing and improving validation much easier. • CRD Ratcheting is not supported for all validations; some will continue to error on update as usual, including required and any use of oldSelf.
27. CRD Ratcheting Example Create custom resource… apiVersion: atlas.mongodb.com/v1 kind: AtlasDatabaseUser ... username: theuser x509Type: INVALID atlasdatabaseuser.atlas.mongodb.com/my-database-user created
28. CRD Ratcheting Example Create custom resource with an invalid field… apiVersion: atlas.mongodb.com/v1 kind: AtlasDatabaseUser ... username: theuser x509Type: INVALID atlasdatabaseuser.atlas.mongodb.com/my-database-user created Update CRD with tightened validation… x509Type: + default: NONE + enum: + - NONE + - MANAGED + - CUSTOMER
29. CRD Ratcheting Example Create custom resource with an invalid field… apiVersion: atlas.mongodb.com/v1 kind: AtlasDatabaseUser ... username: theuser x509Type: INVALID atlasdatabaseuser.atlas.mongodb.com/my-database-user created Update CRD with tightened validation… x509Type: + default: NONE + enum: + - NONE + - MANAGED + - CUSTOMER Change the username (which is valid), not x509Type (which is invalid)... apiVersion: atlas.mongodb.com/v1 kind: AtlasDatabaseUser ... username: theuser_CHANGED # this value is edited x509Type: INVALID
30. Without CRD Ratcheting (Prior to 1.30) Create custom resource with an invalid field… apiVersion: atlas.mongodb.com/v1 kind: AtlasDatabaseUser ... username: theuser x509Type: INVALID atlasdatabaseuser.atlas.mongodb.com/my-database-user created Update CRD with tightened validation… x509Type: + default: NONE + enum: + - NONE + - MANAGED + - CUSTOMER Change the username (which is valid), not x509Type (which is invalid)... apiVersion: atlas.mongodb.com/v1 kind: AtlasDatabaseUser ... username: theuser_CHANGED # this value is edited x509Type: INVALID FAIL The AtlasDatabaseUser "my-database-user" is invalid: spec.x509Type: Unsupported value: "INVALID": supported values: "NONE", "MANAGED", "CUSTOMER"
31. With CRD Ratcheting (1.30 and onwards) Create custom resource with an invalid field… apiVersion: atlas.mongodb.com/v1 kind: AtlasDatabaseUser ... username: theuser x509Type: INVALID atlasdatabaseuser.atlas.mongodb.com/my-database-user created Update CRD with tightened validation… x509Type: + default: NONE + enum: + - NONE + - MANAGED + - CUSTOMER Change the username (which is valid), not x509Type (which is invalid)... apiVersion: atlas.mongodb.com/v1 kind: AtlasDatabaseUser ... username: theuser_CHANGED # this value is edited x509Type: INVALID SUCCESS atlasdatabaseuser.atlas.mongodb.com/my-database-user configured
32. Questions?