Test and evolve your Kubernetes operators

We often hear people talking about Kubernetes (K8s) operators and if you ever used K8s you surely have deployed at least one. However, when the time comes to implement your first operator, it is sometimes challenging to keep in mind what matters and ensure the code we write can be testable, especially when it has to interact with multiple external services.

This talk will provide you with a solid foundation of:
* Best practices in writing operator code
* Admission/mutating webhooks and schema validation (including Common Expression Language)
* How to test operators (and how various mainstream operators do it)
* Custom Resource Definitions API versioning and how to upgrade and support multiple versions
* How to monitor your operator and operator-managed resources

https://community.cncf.io/events/details/cncf-linz-presents-cloud-native-linz-october-18-testing-k8s-operators-amp-opentelemetry-for-logs/

Transcript

1. Test and evolve your Kubernetes Operators Riccardo Capraro | 18.10.2023
2. Pre-requisites B E F O R E W E S T A R T If you are new to K8s operators, start by checking out these resources: What is an operator? | articles https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ https://www.redhat.com/en/topics/containers/what-is-a-kubernetes-operator What is an operator? | video https://youtu.be/i9V4oCa5f9I?feature=shared When do you need Custom Resources? https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#should-i- add-a-custom-resource-to-my-kubernetes-cluster
3. Your Navigator RICCARDO CAPRARO Cloud Infrastructure Engineer @SQUER M A I N F O C U S O N Design and implement code pipelines Infrastructure and cloud automation K8s-based solutions
4. What did I use K8s operators for? Autoscaling of K8s workloads based on custom metrics https://github.com/kubernetes/autoscaler/issues/4135 Scheduling of workloads on a K8s cluster based on latency/bandwidth between network regions in fog computing architectures (cloud+edge) Self-service platform on top of Kafka
5. Timeline (1/2) Intro (5m) What is a Kubernetes operator, what do we use it for, intro to OperatorSDK and kubebuilder Best practices (7m) How to write operators that won’t get stuck in infinite loops & more Custom Resource Definitions (7m) API versioning, schema validation, defaulting and deletion
6. Timeline (2/2) Testing (7m) Frameworks you can use to test your operator Monitoring (7m) How to use kube-state-metrics to monitor your CRs and deploy with confidence Q&A (7m)
7. B E F O R E W E S T A R T
8. B E F O R E W E S T A R T
9. B E F O R E W E S T A R T
10. 01 K8s operators | INTRO Custom Resource Definitions Best Practices
11. What is a K8s operator? A process that watches the K8s API for events on a limited set of resources, and takes actions based on those events, usually to manage external resources Once an event is triggered, the operator re- evaluates the state of the resource/cluster and decides what actions to perform Create/update/delete other K8s resources Create/update/delete external resources I N T R O
12. What is a K8s operator? A process that watches the K8s API for events on a limited set of resources, and takes actions based on those events, usually to manage external resources Once an event is triggered, the operator re- evaluates the state of the resource/cluster and decides what actions to perform Create/update/delete other K8s resources Create/update/delete external resources I N T R O
13. What is a K8s operator? I N T R O A process that watches the K8s API for events on a limited set of resources, and takes actions based on those events, usually to manage external resources Once an event is triggered, the operator re- evaluates the state of the resource/cluster and decides what actions to perform Create/update/delete other K8s resources Create/update/delete external resources
14. Why do we need operators? I N T R O Custom Resource Definitions are (arguably) one of the things that makes K8s successful, as they allow people to reuse the power of K8s (e.g. scheduling) while adding new logic Operators are just the business logic behind Custom Resources, allowing users to control other K8s and external resources declaratively via YAML files written using domain-specific language ?
15. Why do we need operators? I N T R O Custom Resource Definitions are (arguably) one of the things that makes K8s successful, as they allow people to reuse the power of K8s (e.g. scheduling) while adding new logic Operators are just the business logic behind Custom Resources, allowing users to control other K8s and external resources declaratively via YAML files, written using domain-specific language https://github.com/prometheus-operator/prometheus-operator
16. A framework to build operators: operator-sdk I N T R O Built on top of Kubebuilder Supports Go, Ansible, and Helm Why should you use a framework? Easy to deploy new Custom Resource Definitions API versions Easy to add new Custom Resources Easy to set up testing Lifefycle management from day 1 (e.g. OLM) https://sdk.operatorframework.io/build/ https://book.kubebuilder.io/ https://olm.operatorframework.io/
17. A framework to build operators: operator-sdk I N T R O Built on top of Kubebuilder Supports Go, Ansible, and Helm Why should you use a framework? Easy to deploy new Custom Resource Definitions API versions Easy to add new Custom Resources Easy to set up testing Lifefycle management from day 1 (e.g. OLM) https://sdk.operatorframework.io/build/ https://book.kubebuilder.io/ https://olm.operatorframework.io/
18. 02 K8s operators Best Practices Custom Resource Definitions
19. Where do you find these “best practices”? B E S T P R A C T I C E S https://sdk.operatorframework.io/docs/best- practices/ https://cloud.redhat.com/blog/kubernetes- operators-best-practices https://cloud.google.com/blog/products/contai ners-kubernetes/best-practices-for-building- kubernetes-operators-and-stateful-apps https://cloud.redhat.com/blog/7-best- practices-for-writing-kubernetes-operators-an- sre-perspective
20. What are the best practices? B E S T P R A C T I C E S Your CRDs are your contract. Supporting multiple API versions is hard. Make sure you evolve your APIs in a meaningful way! Perform validation as soon as possible (openAPIv3 schema validation, webhooks, operator code) to avoid replication and unnecessary code in your reconciliation loop Idempotency, idempotency, idempotency Make sure that you can recover from ANY error, especially when deleting resources
21. What are the best practices? B E S T P R A C T I C E S Your CRDs are your contract. Supporting multiple API versions is hard. Make sure you evolve your APIs in a meaningful way! Perform validation as soon as possible (openAPIv3 schema validation, webhooks, operator code) to avoid replication and unnecessary code in your reconciliation loop Idempotency, idempotency, idempotency Make sure that you can recover from ANY error, especially when deleting resources
22. What are the best practices? B E S T P R A C T I C E S Your CRDs are your contract. Supporting multiple API versions is hard. Make sure you evolve your APIs in a meaningful way! Perform validation as soon as possible (openAPIv3 schema validation, webhooks, operator code) to avoid replication and unnecessary code in your reconciliation loop Idempotency, idempotency, idempotency Make sure that you can recover from ANY error, especially when deleting resources
23. What are the best practices? B E S T P R A C T I C E S Your CRDs are your contract. Supporting multiple API versions is hard. Make sure you evolve your APIs in a meaningful way! Perform validation as soon as possible (openAPIv3 schema validation, webhooks, operator code) to avoid replication and unnecessary code in your reconciliation loop Idempotency, idempotency, idempotency Make sure that you can recover from ANY error, especially when deleting resources
24. What are the best practices? B E S T P R A C T I C E S Your CRDs are your contract. Supporting multiple API versions is hard. Make sure you evolve your APIs in a meaningful way! Perform validation as soon as possible (openAPIv3 schema validation, webhooks, operator code) to avoid replication and unnecessary code in your reconciliation loop Idempotency, idempotency, idempotency Make sure that you can recover from ANY error, especially when deleting resources
25. What are the best practices? B E S T P R A C T I C E S Your CRDs are your contract. Supporting multiple API versions is hard. Make sure you evolve your APIs in a meaningful way! Perform validation as soon as possible (openAPIv3 schema validation, webhooks, operator code) to avoid replication and unnecessary code in your reconciliation loop Idempotency, idempotency, idempotency Make sure that you can recover from ANY error, especially when deleting resources
26. What are the best practices? Failed reconciliations should not slow down other people resources: requeue failed resources with purpose Your users should be able to debug CRs configuration problems without your help: provide meaningful error messages (in the CR or via events) https://sdk.operatorframework.io/docs/building-operators/golang/tutorial/ B E S T P R A C T I C E S
27. What are the best practices? B E S T P R A C T I C E S Failed reconciliations should not slow down other people resources: requeue failed resources with purpose Your users should be able to debug CRs configuration problems without your help: provide meaningful error messages (in the CR or via events) https://sdk.operatorframework.io/docs/building-operators/golang/tutorial/ For a guide on Reconcilers, Clients, and interacting with resource Events, see the Client API doc. The following are a few possible return options for a Reconciler: • With the error: • Without an error: • Therefore, to stop the Reconcile, use: • Reconcile again after X time:
28. What are the best practices? Failed reconciliations should not slow down other people resources: requeue failed resources with purpose Your users should be able to debug CRs configuration problems without your help: provide meaningful error messages (in the CR or via events) B E S T P R A C T I C E S
29. 03 K8s operators Best Practices Custom Resource Definitions
31. CR: Your state machine C U S T O M R E S O U R C E D E F I N I T I O N S Think of your Custom Resource lifecycle as a state machine. Use different tools for different states OpenAPIv3 schemas and webhooks for validation and defaulting Operator for business logic Have clear error states and ensure the operator always knows in what state the resource is Is it updating a valid resource, is it trying to delete a resource whose deletion previously failed
32. CR: Your state machine C U S T O M R E S O U R C E D E F I N I T I O N S Think of your Custom Resource lifecycle as a state machine. Use different tools for different states OpenAPIv3 schemas and webhooks for validation and defaulting Operator for business logic Have clear error states and ensure the operator always knows in what state the resource is Is it updating a valid resource, is it trying to delete a resource whose deletion previously failed
33. CR: Your state machine C U S T O M R E S O U R C E D E F I N I T I O N S Have a clear separation between code paths: – Create – Update – Delete Create and update operations should strive to be idempotent: at times you might reuse the same logic for both: e.g., instead of having CreateUser() and UpdateUser() you just have SaveUser()
34. CR: Your state machine C U S T O M R E S O U R C E D E F I N I T I O N S Have a clear separation between code paths: – Create – Update – Delete Create and update operations should strive to be idempotent: at times you might reuse the same logic for both: e.g., instead of having CreateUser() and UpdateUser() you just have SaveUser()
35. CR stages: resource conditions How do we achieve CR state management in K8s and make it transparent to the user? My recommendation: use the standard status conditions: – Is my CR initialized? – Is my CR ready? Conditions provide a single place for your code to know about the state/phase of your CR, and also provide a single point for your users to check the status and errors. The alternative (K8s events) will be discussed later. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions C U S T O M R E S O U R C E D E F I N I T I O N S
36. CR stages: resource conditions How do we achieve CR state management in K8s and make it transparent to the user? My recommendation: use the standard status conditions: – Is my CR initialized? – Is my CR ready? Conditions provide a single place for your code to know about the state/phase of your CR, and also provide a single point for your users to check the status and errors. The alternative (K8s events) will be discussed later. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions C U S T O M R E S O U R C E D E F I N I T I O N S
37. CR stages: resource conditions How do we achieve CR state management in K8s and make it transparent to the user? My recommendation: use the standard status conditions: – Is my CR initialized? – Is my CR ready? Conditions provide a single place for your code to know about the state/phase of your CR, and also provide a single point for your users to check the status and errors. The alternative (K8s events) will be discussed later. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions C U S T O M R E S O U R C E D E F I N I T I O N S
38. CR stages: imperative commands You can also use webhooks to manage phase transition, e.g. requeuing failed resources when some annotations are added to the CR. Use imperative commands sparingly Do not create infinite reconciliation loops https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions C U S T O M R E S O U R C E D E F I N I T I O N S
39. CR stages: imperative commands You can also use webhooks to manage phase transition, e.g. requeing failed resources when some annotations are added to the CR. Use imperative commands sparingly Do not create infinite reconciliation loops https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions C U S T O M R E S O U R C E D E F I N I T I O N S
40. CR lifecycle: Validation Syntactic – OpenAPIv3 schema validation Semantic – Admission webhooks – Common Expression Language (beta K8s v1.25) – replace admission webhooks. Works also with kubebuilder! Some people recommend doing semantic validation at both the admission webhook and in the operator logic. You can also use phases to manage this (e.g. using conditions) https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions https://kubernetes.io/blog/2022/09/29/enforce-immutability-using-cel/ C U S T O M R E S O U R C E D E F I N I T I O N S
41. CR lifecycle: Validation Syntactic – OpenAPIv3 schema validation Semantic – Admission webhooks – Common Expression Language (beta K8s v1.25) – replace admission webhooks. Works also with kubebuilder! Some people recommend doing semantic validation at both the admission webhook and in the operator logic. You can also use phases to manage this (e.g. using conditions) https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions https://kubernetes.io/blog/2022/09/29/enforce-immutability-using-cel/ C U S T O M R E S O U R C E D E F I N I T I O N S
42. CR lifecycle: Validation Syntactic – OpenAPIv3 schema validation Semantic – Admission webhooks – Common Expression Language (beta K8s v1.25) – replace admission webhooks. Works also with kubebuilder! Some people recommend doing semantic validation at both the admission webhook and in the operator logic. You can also use phases to manage this (e.g. using conditions) https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions https://kubernetes.io/blog/2022/09/29/enforce-immutability-using-cel/ C U S T O M R E S O U R C E D E F I N I T I O N S
43. CR lifecycle: Validation Syntactic – OpenAPIv3 schema validation Semantic – Admission webhooks – Common Expression Language (beta K8s v1.25) – replace admission webhooks. Works also with kubebuilder! Some people recommend doing semantic validation at both the admission webhook and in the operator logic. You can also use phases to manage this (e.g. using conditions) C U S T O M R E S O U R C E D E F I N I T I O N S
44. CR lifecycle: Deletion C U S T O M R E S O U R C E D E F I N I T I O N S Use finalizers (add them during CR initialization) Ensure you can recover transient failures – make sure all delete operations in your code are idempotent Do not re-process failed CRs unless the error is recoverable – have different “Failure” states https://kubernetes.io/blog/2022/09/23/crd-validation-rules-beta/
45. CR lifecycle: Deletion C U S T O M R E S O U R C E D E F I N I T I O N S Use finalizers (add them during CR initialization) Ensure you can recover transient failures – make sure all delete operations in your code are idempotent Do not re-process failed CRs unless the error is recoverable – have different “Failure” states https://kubernetes.io/blog/2022/09/23/crd-validation-rules-beta/
46. CR lifecycle: Deletion C U S T O M R E S O U R C E D E F I N I T I O N S Use finalizers (add them during CR initialization) Ensure you can recover transient failures – make sure all delete operations in your code are idempotent Do not re-process failed CRs unless the error is recoverable – have different “Failure” states https://kubernetes.io/blog/2022/09/23/crd-validation-rules-beta/
47. CR lifecycle: Deletion C U S T O M R E S O U R C E D E F I N I T I O N S Example: two external resources A, B You delete A, deletion of B fails because of a network hiccup ”Bad”: DeleteA(), DeleteB() Better:TryDeleteA() or DeleteAIfExists() https://kubernetes.io/blog/2022/09/23/crd-validation-rules-beta/
48. CR lifecycle: error handling (events) C U S T O M R E S O U R C E D E F I N I T I O N S You can use K8s events Nice integration in ArgoCD You can also use external tools to trigger notifications based on this events (e.g., Grafana) Very nice description about how to do that at the end of this blog post https://cloud.redhat.com/blog/kubernetes- operators-best-practices https://www.cncf.io/blog/2023/03/13/how-to-use-kubernetes-events-for-effective-alerting-and-monitoring/ https://kubernetes.io/blog/2018/01/reporting-errors-using-kubernetes-events/
49. CR lifecycle: error handling (events) C U S T O M R E S O U R C E D E F I N I T I O N S You can use K8s events Nice integration in ArgoCD You can also use external tools to trigger notifications based on this events (e.g., Grafana) Very nice description about how to do that at the end of this blog post https://cloud.redhat.com/blog/kubernetes- operators-best-practices https://www.cncf.io/blog/2023/03/13/how-to-use-kubernetes-events-for-effective-alerting-and-monitoring/ https://kubernetes.io/blog/2018/01/reporting-errors-using-kubernetes-events/
50. CR lifecycle: error handling (events) C U S T O M R E S O U R C E D E F I N I T I O N S You can use K8s events Nice integration in ArgoCD You can also use external tools to trigger notifications based on this events (e.g., Grafana) Very nice description about how to do that at the end of this blog post https://cloud.redhat.com/blog/kubernetes- operators-best-practices https://www.cncf.io/blog/2023/03/13/how-to-use-kubernetes-events-for-effective-alerting-and-monitoring/ https://kubernetes.io/blog/2018/01/reporting-errors-using-kubernetes-events/
51. CR lifecycle: error handling (conditions) You can put error messages in a ”Ready” condition, as this is persisted for longer than regular K8s events (and still use events to notify your users) Conditions are persisted in the CR, events are transient https://www.cncf.io/blog/2023/03/13/how-to-use-kubernetes-events-for-effective-alerting- and-monitoring/ https://kubernetes.io/blog/2018/01/reporting-errors-using-kubernetes-events/ C U S T O M R E S O U R C E D E F I N I T I O N S
52. CR lifecycle: API versions C U S T O M R E S O U R C E D E F I N I T I O N S Know the difference between served/storage versions Serve multiple versions, encourage users to upgrade as soon as possible Only ONE storage version Use conversion webhooks sparingly Dropping/adding fields is unexpensive Renaming fields is expensive https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/
53. CR lifecycle: API versions C U S T O M R E S O U R C E D E F I N I T I O N S Know the difference between served/storage versions Serve multiple versions, encourage users to upgrade as soon as possible Only ONE storage version Use conversion webhooks sparingly Dropping/adding fields is unexpensive Renaming fields is expensive https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/
54. CR lifecycle: API versions C U S T O M R E S O U R C E D E F I N I T I O N S Know the difference between served/storage versions Serve multiple versions, encourage users to upgrade as soon as possible Only ONE storage version Use conversion webhooks sparingly Dropping/adding fields is unexpensive Renaming fields is expensive https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/
55. 04 Testing Monitoring Q&A
56. Recommendations T E S T I N G Move code to libraries that can be tested outside your operator Make sure you can attach a debugger to your tests Ensure you can unit test your functions (mock all external services) If possible, don’t rely on an existing cluster for testing Run integration tests against services only when needed
57. Recommendations T E S T I N G Move code to libraries that can be tested outside your operator Make sure you can attach a debugger to your tests Ensure you can unit test your functions (mock all external services) If possible, don’t rely on an existing cluster for testing Run integration tests against services only when needed
58. Recommendations T E S T I N G Move code to libraries that can be tested outside your operator Make sure you can attach a debugger to your tests Ensure you can unit test your functions (mock all external services) If possible, don’t rely on an existing cluster for testing Run integration tests against services only when needed
59. Recommendations T E S T I N G Move code to libraries that can be tested outside your operator Make sure you can attach a debugger to your tests Ensure you can unit test your functions (mock all external services) If possible, don’t rely on an existing cluster for testing Run integration tests against services only when needed
60. Recommendations T E S T I N G Move code to libraries that can be tested outside your operator Make sure you can attach a debugger to your tests Ensure you can unit test your functions (mock all external services) If possible, don’t rely on an existing cluster for testing Run integration tests against services only when needed
61. Recommendations T E S T I N G https://vektra.github.io/mockery/latest/ https://github.com/jarcoal/httpmock API mocking: mockery, httpmock, … Use domain-specific language: ginkgo + gomega Use envtest for integration tests Use an actual cluster (minikube, k3d, kind) only when necessary
62. Recommendations T E S T I N G https://onsi.github.io/ginkgo/ https://onsi.github.io/gomega/ API mocking: mockery, httpmock, … Use domain-specific language: ginkgo + gomega Use envtest for integration tests Use an actual cluster (minikube, k3d, kind) only when necessary
63. Recommendations T E S T I N G https://onsi.github.io/ginkgo/ https://onsi.github.io/gomega/ API mocking: mockery, httpmock, … Use domain-specific language: ginkgo + gomega Use envtest for integration tests Use an actual cluster (minikube, k3d, kind) only when necessary
64. Recommendations T E S T I N G https://onsi.github.io/ginkgo/ https://onsi.github.io/gomega/ API mocking: mockery, httpmock, … Use domain-specific language: ginkgo + gomega Use envtest for integration tests Use an actual cluster (minikube, k3d, kind) only when necessary
65. Recommendations T E S T I N G https://book.kubebuilder.io/reference/envtest.html API mocking: mockery, httpmock, … Use domain-specific language: ginkgo + gomega Use envtest for integration tests Use an actual cluster (minikube, k3d, kind) only when necessary
66. 05 Testing Monitoring Q&A
67. How do you monitor your CRs? M O N I T O R I N G Deploy an instance of kube-state-metrics that only monitors our CRs Easy to add/extend metrics Requires new pod to reload configurations – For now you can use something like https://github.com/stakater/Reloader – Active work to improve this https://github.com/kubernetes/kube-state- metrics/issues/1948 https://github.com/kubernetes/kube-state-metrics/blob/main/docs/customresourcestate-metrics.md
68. How do you monitor your CRs? M O N I T O R I N G Deploy an instance of kube-state-metrics that only monitors our CRs Easy to add/extend metrics Requires new pod to reload configurations – For now you can use something like https://github.com/stakater/Reloader – Active work to improve this https://github.com/kubernetes/kube-state- metrics/issues/1948 https://github.com/kubernetes/kube-state-metrics/blob/main/docs/customresourcestate-metrics.md
69. How do you monitor your CRs? M O N I T O R I N G Deploy an instance of kube-state-metrics that only monitors our CRs Easy to add/extend metrics Requires new pod to reload configurations – For now you can use something like https://github.com/stakater/Reloader – Active work to improve this https://github.com/kubernetes/kube-state- metrics/issues/1948 https://github.com/kubernetes/kube-state-metrics/blob/main/docs/customresourcestate-metrics.md
70. Observability, but Why? M O N I T O R I N G Good to provide insights to your users about the state of their CRs across namespaces (e.g. in Grafana) To deploy operator changes with confidence https://github.com/kubernetes/kube-state-metrics/blob/main/docs/customresourcestate-metrics.md
71. Observability, but Why? M O N I T O R I N G Good to provide insights to your users about the state of their CRs across namespaces (e.g. in Grafana) To deploy operator changes with confidence https://github.com/kubernetes/kube-state-metrics/blob/main/docs/customresourcestate-metrics.md
72. Quick live demo: setup minikube delete && minikube start --kubernetes-version=v1.23.0 --memory=6g -- bootstrapper=kubeadm --extra-config=kubelet.authentication-token-webhook=true --extra-config=kubelet.authorization-mode=Webhook --extra- config=scheduler.bind-address=0.0.0.0 --extra-config=controller-manager.bind- address=0.0.0.0 minikube addons disable metrics-server https://github.com/prometheus-operator/kube-prometheus/tree/v0.12.0 https://gist.github.com/squer-rcapraro/1b7de0f9b3d70026421ac28c0bdd4e0c M O N I T O R I N G
73. Quick live demo: kube-prometheus M O N I T O R I N G kubectl create namespace monitoring helm install kube-prometheus oci://registry-1.docker.io/bitnamicharts/kube- prometheus --version 8.18.0 --namespace monitoring https://github.com/prometheus-operator/kube-prometheus/tree/v0.12.0 https://gist.github.com/squer-rcapraro/1b7de0f9b3d70026421ac28c0bdd4e0c
74. Quick live demo: kube-state-metrics M O N I T O R I N G curl https://gist.githubusercontent.com/squer- rcapraro/1b7de0f9b3d70026421ac28c0bdd4e0c/raw/0cc4aa2cfb0ac0ac57bf500da1d7c2 7f22ea5551/gistfile1.txt | kubectl apply -f - minikube service -n monitoring prometheus-nodeport --url https://github.com/prometheus-operator/kube-prometheus/tree/v0.12.0 https://gist.github.com/squer-rcapraro/1b7de0f9b3d70026421ac28c0bdd4e0c
75. Quick live demo: cleanup M O N I T O R I N G minikube delete https://github.com/prometheus-operator/kube-prometheus/tree/v0.12.0 https://gist.github.com/squer-rcapraro/1b7de0f9b3d70026421ac28c0bdd4e0c
76. 06 Testing Monitoring Q&A
77. Questions?
78. Get in touch Riccardo Capraro Cloud Infrastructure Engineer
79. SQUER Solutions GmbH Marxergasse 17/1/33-34 1030 Wien office@squer.at www.squer.at 0676 / 849 963 200
80. Metacontroller: yet another operator framework B o n u s Addon to K8s that makes it easy to deploy custom controllers Provides Simple automation Reusable building blocks Complex orchestration You implement the webhooks, Metacontroller takes care of making it work https://metacontroller.github.io/metacontroller/intro.html
81. An example: Transient Role Bindings B o n u s Provide temporary “escalated” permissions to your users Use Open Policy Agent (OPA) and the Rego language to implement policies Let metacontroller handle the Custom Resources lifecycle https://cloud.google.com/anthos-config-management/docs/how-to/time-constrained-role-binding-kubernetes https://github.com/GoogleCloudPlatform/k8s-transient-role-binding https://www.openpolicyagent.org/
82. An example: Transient Role Bindings B o n u s Provide temporary “escalated” permissions to your users Use Open Policy Agent (OPA) and the Rego language to implement policies Let metacontroller handle the Custom Resources lifecycle https://cloud.google.com/anthos-config-management/docs/how-to/time-constrained-role-binding-kubernetes https://github.com/GoogleCloudPlatform/k8s-transient-role-binding https://www.openpolicyagent.org/
83. An example: Transient Role Bindings B o n u s Two custom resources: TransientRoleBinding TransientClusterRoleBinding https://cloud.google.com/anthos-config-management/docs/how-to/time-constrained-role-binding-kubernetes https://github.com/GoogleCloudPlatform/k8s-transient-role-binding https://www.openpolicyagent.org/