Replace Your Custom Kubernetes Controller with Kyverno Policies

To manage Kubernetes environments, custom controllers are often essential for enforcing policies, automating tasks, and ensuring compliance. While these controllers provide flexibility, they demand considerable development effort, ongoing maintenance and a profound understanding of Kubernetes. This is where Kyverno comes into play - a proprietary policy management tool for Kubernetes that simplifies the enforcement of cluster-wide rules through declarative policies, thereby eliminating the need for custom controller code from scratch.

In our talk, we will demonstrate how custom Kubernetes controllers can be replaced by Kyverno policies. We will address the challenges associated with maintaining custom controllers, highlight the advantages of Kyverno's declarative approach, and present real-world scenarios where Kyverno effectively manages tasks such as resource validation, mutation and generation. Examples will illustrate how Kyverno reduces operational overhead, accelerates implementation and enhances maintainability without sacrificing flexibility.

The presentation will provide insights into migrating from custom-written controllers to Kyverno, leveraging the capabilities of Kyverno on building a developer platform using Kubernetes.

Transcript

1. Kyverno policies Replace Your Custom Kubernetes Controller with Kyverno Policies
2. 2 Let‘s get to know each other Alfred Schmid Tech Lead, Cloud Platforms Kai René Koch Senior Software Engineer
3. 3 Short intro to Kyverno and Kubernetes controllers Today’s Agenda Typical issues with self-made controllers Advantages of Kyverno policies Real-world transformation approaches When to prefer policies or controllers? Further alternatives to self-made controllers/operators 01 02 03 04 05 06
4. 4 Short intro to Kyverno and controllers • Policy-as-code administration suite for Kubernetes • Policies can validate, mutate, generate, clone or clean up resources • Policies are provided via custom Kubernetes resources for declarative usage • Policies can utilize JMESpath and Common Expression Language (CEL) • Policy testing via Helm unit tests, Kyverno CLI or Kyverno Chainsaw (E2E) • Application and platform-specific sample policies available • Kubernetes controllers orchestrate a subset of Kubernetes resources • Controllers administrate existing resources • Operators additionally specify and maintain own custom resources • Can be used to logically abstract concepts (e.g. an App CRD which handles all required sub-resources) • Can be written in almost all languages which can access the Kubernetes API • Operator frameworks for Go, Java or Python simplify the creation Kyverno Kubernetes controllers Provides a simple and well-defined way to validate constraints and orchestrate resources Provide a flexible approach to orchestrate Kubernetes resources via the API
5. 5 Typical issues with self-made controllers Implementation effort dependent on language and framework usage, as well as the knowledge of them in the team Multiple controllers may differ in code architecture and used libraries, which could increase costs for maintenance or adaptation Easy to write initially, but often hard to provide a stable and robust implementation Maintenance and time required for bug fixes, new features and updates If multiple controllers do similar things, code (and maintenance) gets duplicated We want to lower our maintenance effort, reduce costs and risks and focus on our advanced controllers and other platform features
6. 6 Advantages of Kyverno policies Policies are declared with a well-defined common language Easy to extend and test The whole orchestration will be done via robust Kyverno operators Heavy lifting of updates and maintenance will be done upstream by Kyverno Namespaced and cluster- scoped policies to restrict impact and improve security Multi-purpose usage of Kyverno in Kubernetes platforms Resource maintenance done by Kyverno (creation, update, deletion) Already present in many Kubernetes environments Kyverno policies offer a solid way to (partly) replace our small controllers and avoid new ones (or make them smaller)
7. 7 Real-world transformation approaches General approach • Identify controller to replace (or parts of it) • Check if code is suitable for a policy, e.g.: • Simple copy/generate tasks on few resources? • Creation of resources based on conditions? • Cleanup tasks? • Cluster or namespaced policy/policies? • Use scaffold of similar existing policies or sample policies • Adapt Kyverno RBAC for custom resources • Define context, preconditions, exclusions and matches • Write rules based on controller task • Test policy/policies (e.g. in a local test cluster) • Deprecate controller
8. 8 Real-world transformation approaches Examples
9. 9 When to prefer policies or controllers? Kyverno • Resource generation, cloning and cleanup • Combining and mutating of resource properties • Conditional creation/deletion of objects Kubernetes controller • Many resources need to be orchestrated • Complex calculations are required • Advanced abstraction layers should be provided Kyverno policies are often a valid alternative to controllers or can work together with them
10. 10 Further alternatives to self-made controllers/operators • Kubernetes policy engine alternatives: OPA, jsPolicy • Crossplane compositions for Crossplane-managed resources • Kro.run
11. 11 Summary Successfully replaced several small controllers with Kyverno policies Less code and reduced maintenance effort Reduced platform complexity Kyverno handles proper creation, modification and cleanup of resources More focus on platform improvement and complex operators Easy to start with simple policies (step by step replacement) Policies can also replace parts of existing controllers New controllers can be simplified or completely replaced by Kyverno policies
12. 12 Sources and additional information • Kyverno https://kyverno.io/ • Kubernetes https://kubernetes.io/ • Kubernetes policy engine comparison https://opensource.com/article/23/2/kubernetes-policy-engines • Crossplane Compositions https://docs.crossplane.io/latest/concepts/compositions/ • Kro https://kro.run/ • Examples and slides: https://github.com/steadforce/cloudland-2025
13. 13 Let’s talk technology! Visit us on www.steadforce.com! Come to our booth for further discussions, live cluster demos and the Kubernetes Escape Room game Steadforce GmbH Westendstraße 193 80686 Munich Germany +49 89 51727 0 contact@steadforce.com