Crossplane Providers and OLM Repackaging Openshift Commons Briefing Krish Chowdhary Red Hat 1

2 - SWE @ Red Hat Office of the CTO - Provider-AWS Maintainer - Provider-In-Cluster Maintainer - Kubernetes SIG-Storage Member - Twitter: @krishchow_ - Github: krishchow Krish Chowdhary

Intro - What is a Crossplane provider? - Where are providers? - When should I use or create a provider? - Why would I need to create a provider? - Who creates & maintains providers? - What are the main components? - OLM Repackaging 3 Agenda

4 What?

What is Crossplane 5 Let’s quickly go over Crossplane Crossplane allows you to manage the entire lifecycle, you can update and delete external resources Observing Managing Specific Providers expose information for the resources through Kubernetes fields on the CRs and Events. Provisioning The Crossplane project allows you to provision cloud resources from within your Kubernetes Cluster Context

Context “ Providers are packages that enable Crossplane to provision infrastructure on an external service. They bring CRDs (i.e. managed resources) that map one-to-one to external infrastructure resources, as well as controllers to manage the life-cycle of those resources. “ 6 - Crossplane Docs

Context 7 ▸ Providers are similar to Operators, but you utilize the Crossplane Operator for installing and management ▸ Providers use a lot of the same tooling as Operators, such as kube-builder, controller-runtime and controller-tools. For this reason, it is commonly considered as an opinionated form of an Operator. ▸ The main conceptual difference is that providers are designed to reference some external* resource ▸ For a developer, the nice thing is that you don’t need to worry about RBAC, deployment and bootstrapping since tooling exists What are providers, really? * This is kind of a lie, providers can really manage any resource, and there are providers designed to handle in cluster resources as well!

8 Where?

Where are providers located? Crossplane providers are open source, and available on Github under the Crossplane or Crossplane-Contrib organizations. Some examples are: Provider-SQL - Repo Orchestrates SQL servers by creating Users, Grants, Roles, etc. Provider-Helm - Repo Manage and deploy Helm charts using custom resources Provider-AWS - Repo Provision resources on AWS such as databases (RDS) or k8s clusters (EKS) 9 Where

10 When?

When When does it make sense to create a provider? 11 When should a provider be created rather than using an existing one? Providers are designed to interface with external APIs, and manage the CRUD lifecycle for these resources. Consume External Resources There is an emphasis on high fidelity for managed resources, you should create a provider if your resources are well defined and granular. High Fidelity Resources If you want to use the powerful composition engine, the best way* is to create a provider which manages your resources Utilize Compositions * Technically you could utilize any resource within a composition, but the recommended approach is utilize a provider: via 1730

12 Why?

Crossplane is a CNCF Sandbox project, the project and many providers are open-source, anyone can contribute. ▸ Shared development and maintenance of common resources ▸ Cloud vendors can expose their APIs in Kubernetes through a common interface ▸ Streamlined development and consumption, Crossplane handles a lot of the messy parts. Why 13 Why should you create or contribute to a provider?

Context ▸ Proxy Issues: We will see this shortly, but integral to the Crossplane Operator is pulling an OCI image with metadata. This can cause issues with cluster running behind a proxy. ▸ Credential sharing: Similar to above, if you are utilizing a private container registry, you will need to separately supply credentials to Crossplane and the Kubelet. ▸ Resource limitations: You may not always want to install the Crossplane Operator for security or resource limitations. With OLM repackaging you can choose to just install the providers you need, and you can directly provision resources. 14 Why might we want to repackage a provider?

15 Who?

Who is creating and maintaining providers? Who 16 There are many contributors from organizations like Upbound, Red Hat, Squiz and Accenture Vendors Organizations Engineers from vendors like Alibaba, IBM, AWS and Equinix contribute to the development of their providers Community The Crossplane operator related providers are open source, many contributors are active members of the community

17 Provider & OLM Components

- One of the most popular Crossplane providers, with support for dozens of AWS resources - Also, a great example of provider to dissect :) 18 Components Provider Primer - AWS

- A common sample Operator used in guides and examples - Also, a great example of the standard structure of an OLM Operator 19 Components OLM Primer - Memcached

Components 20 Memcached Operator Structure Core Components ▸ api/ - contains the API definitions for the Memcached Operator Crossplane Provider Structure ▸ apis/ - analogous to the “api” folder in Memcached Operator. ▸ controllers/ - contains the logic and tests for the controllers/reconcilers (CRUD) ▸ pkg/ - controllers and clients used by the provider, similar to the “controller” folder. ▸ config/ + bundle/ - manifests for the deployment of the operator and the related RBAC, as well as CRDs from the API ▸ package/ + cluster/ - contain the Dockerfiles used, and the CRDs generated from the API

TL;DR - Mainly semantic differences at this point, however the last point is key. 21 Components

Components 22 Memcached Operator Structure Core Components ▸ api/ - contains the API definitions for the Memcached Operator Crossplane Provider Structure ▸ apis/ - analogous to the “api” folder in Memcached Operator. ▸ controllers/ - contains the logic and tests for the controllers/reconcilers (CRUD) ▸ pkg/ - controllers and clients used by the provider, similar to the “controller” folder. ▸ config/ + bundle/ - manifests for the deployment of the operator and the related RBAC, as well as CRDs from the API ▸ package/ + cluster/ - contain the Dockerfiles used, and the CRDs generated from the API How do I actually deploy a provider? There’s no Deployment, ClusterRole or anything else that I can use!

- One of the most popular ways to install a Crossplane provider 23 Aside on Deployment Provider CRD apiVersion: pkg.crossplane.io/v1 kind: Provider metadata: name: provider-aws spec: package: crossplane/provider-aws:alpha

- Installed the same way as providers, but the configuration exposes XRDs - Users install providers through the use of dependencies. 24 Aside on Deployment Configuration CRD apiVersion: meta.pkg.crossplane.io/v1 kind: Configuration metadata: name: my-favorite-configuration spec: crossplane: version: ">=v1.0.0" dependsOn: - provider: crossplane/provider-aws version: ">=v0.14.0"

25 The Delta

Where are our missing components related to deploying the provider? 26 Components

Components 27 1. https://github.com/tokibi/undocker - What does our package actually contain? - With a tool like undocker we can actually browse the contents, without digging through the build process1. - What we find is interesting, the image we reference contains only CRDs, metadata, and a reference to the actual controller within a YAML file The Provider Package $ docker pull crossplane/provider-aws:v0.18.1 v0.18.1: Pulling from crossplane/provider-aws $ docker save crossplane/provider-aws:v0.18.1 | undocker -o provider-aws $ tree provider-aws provider-aws └── package.yaml $ tail provider-aws/package.yaml -n 143 apiVersion: meta.pkg.crossplane.io/v1alpha1 kind: Provider metadata: name: provider-aws ... spec: controller: image: crossplane/provider-aws-controller:v0.18.1

Components 28 For brevity, we are not covering the Crossplane Operator in detail, relevant code for this logic can be found here with the relevant controller focusing on the provider revisions It follows that there are two separate steps, with our metadata image, and then the actual controller. The rest of the puzzle The main Crossplane Operator parses the package.yaml, installs CRDs and configures RBAC for the provider at runtime The Crossplane Operator then also creates the deployment with the tag from spec.controller.image in the package.yaml

Components 29 For brevity, we are not covering the Crossplane Operator in detail, relevant code for this logic can be found here with the relevant controller focusing on the provider revisions It follows that there are two separate steps, with our metadata image, and then the actual controller. The rest of the puzzle The main Crossplane Operator parses the package.yaml, installs CRDs and configures RBAC for the provider at runtime The Crossplane Operator then also creates the deployment with the tag from spec.controller.image in the package.yaml This is notable because OLM opts to define RBAC and deployment manifests at build time

30 OLM Repackaging

- Dockerfile - Makefile - PROJECT.boilerplate.txt - config/ - gen_rbac.sh - This script creates an `rbac.go` file, which contains kubebuilder annotations that generate the RBAC - gen_project.sh - This script generates our top-level PROJECT file (metadata) 31 Aside on Deployment Configuration CRD $ git clone [email protected]:redhat-et/olm-repackage.git $ tree olm-repackage . ├── Dockerfile ├── Makefile ├── PROJECT.boilerplate.txt ├── README.md ├── config/ │ ├── certmanager/ │ ├── crd/ # contains generated CRDs │ ├── default/ │ ├── manager/ # contains deployment │ ├── manifests/ # ClusterServiceVersion (csv) │ ├── prometheus/ │ ├── rbac/ # contains generated RBAC │ ├── samples/ │ └── scorecard/ ├── gen_project.sh └── gen_rbac.sh Again for brevity some resources in sub-folders have been omitted, the contents of the config directory mirrors contains all of the required manifests

Repackaging 32 Not included: Go-lang, Docker/Podman 1. https://github.com/mikefarah/yq 2. https://sdk.operatorframework.io/docs/building-operators/golang/installation/ 3. Kustomize and controller-gen are installed by targets in the Makefile Kustomize is a tool used to customize Kubernetes resources through configuration patches The Operator-SDK CLI tool is used to handle the generation and validation of deployment manifests Operator-SDK2 controller-gen is a tool used to generate CRDs, RBAC and code through marker comments in Go controller-gen3 yq is a tool similar to jq, except it allows you to query fields in a YAML document yq1 Kustomize3 Repackage Tooling

33 Setup all of our dependencies, these were all seen on the previous slide. Make sure to also to have credentials setup for a container registry (like Quay.io) Repackaging Provider-OLM Repackaging Part 1 - Setup Clone your target repository - for this example, we will be using the provider-aws Clone the olm-repackage repository, then copy the contents into the root of the provider repository.

Repackaging 34

35 Lastly, we want to build our bundle. Although the newly created Operator has been pushed, we still need to create the bundle, which adds all of our manifests and metadata to the single package which is installed by the Operator Lifecycle Manager (OLM) Repackaging Provider-OLM Repackaging Part 2 - Build Add the appropriate environment variables, and build the Operator. This also runs all of our code gen, and afterwards we can push it up to our container registry.

Repackaging 36

37 Running the Provider-AWS Operator

linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHat 38 Linkedin: krishchow Twitter: @krishchow_ Github: krishchow Or just ping me on Slack :) Thank you