VSHN – The DevOps Company Wednesday April 27th, 2022 Adrian Kosmaczewski, Developer Relations, VSHN The New GitLab Kubernetes Agent Switzerland GitLab Meetup Welcome to the third Switzerland GitLab Meetup. Today we’re going to talk about a fundamental piece of machinery for GitOps engineers working with GitLab: the new GitLab Kubernetes Agent, replacing the old certificate-based integration, to be discontinued soon. Speaker notes 1

VSHN – The DevOps Company  The integration of GitLab and Kubernetes is a critical one. More and more developers are using it to automatically deploy their applications to Kubernetes clusters from the confort of their Git client, in a typical GitOps fashion: git push and a few minutes later, your app is published. Speaker notes 2

VSHN – The DevOps Company To integrate GitLab and Kubernetes, since GitLab 10.4 (released in 2018) we were using certificates; that is, you had to give the URL of the Kubernetes API of your cluster, provide a few secrets, and boom, your GitLab can talk to your Kubernetes. But this is all about to change right now. Speaker notes 3

VSHN – The DevOps Company The New GitLab Kubernetes Agent Let us talk today about the new way of integrating GitLab and Kubernetes, using the brand new GitLab Kubernetes Agent, which completely replaces the old certificate-based integration. Speaker notes 4

VSHN – The DevOps Company (Source: about.gitlab.com/blog) The deprecation notice of the certificate-based mechanism, and its replacement with the new GitLab Kubernetes Agent, was announced in a blog post in November last year. Speaker notes 5

VSHN – The DevOps Company (Source: about.gitlab.com/blog) The list of breaking changes in GitLab 15.0 contains a special section called "Configure" with precisely a section explaining how the SaaS certificate-based integration with Kubernetes will be phased out completely. This is huge news for both SaaS and self-hosted users of GitLab. Speaker notes 6

VSHN – The DevOps Company 1. It requires direct access to the K8s API 2. Many valuable features required giving cluster admin rights to GitLab 3. New trend: pull based deployments & CI/CD (Argo CD, anyone?) 4. Polling… polling… polling. Issues with Certificates Why would GitLab perform such a groundbreaking change? It turns out that the old certificate-based mechanism had some shortcomings: It required complete and direct access to the Kubernetes API, which, well, can be considered a security risk by some organizations. To access many advanced features, you had to provide GitLab with cluster admin rights. Again, not something that would put a smile on a security engineer. There’s a new trend in the CI/CD space, which is "pull-based deployments", similar to what Argo CD is offering, and GitLab wants to be a part of that market too. In general, the certificate-based mechanism relies in polling, which as we all know, is not the greatest of ideas in Computer Science. Speaker notes 7

VSHN – The DevOps Company Deprecated since version 14.5 GitLab.com: removed in version 15.0 Self-hosted: disabled in 15.0, removed in 15.6 Kubernetes version: minimum 1.20 (released December 2020, supported until February 2022) Deprecation & Requirements How and when will the old certificate mechanism be replaced by the GitLab Kubernetes Agent? The certificate mechanism has been already marked as deprecated since version 14.5, and will be completely removed in GitLab SaaS in version 15.0. For self- hosted users, it will be disabled in 15.0 and completely removed in 15.6, so you have some time to migrate to the agent. Finally, to be able to use the GitLab Kubernetes Agent, you must use a version of Kubernetes equal or higher to 1.20. Speaker notes 8

VSHN – The DevOps Company  Not all certificate-based features will be available with Agents! Important! A very important thing to know is that this is effectively a breaking change. Many features available with the certificate-based mechanism will not be available with the new GitLab Kubernetes Agent! For example, GitLab Managed Apps (GMA) were deprecated in GitLab 14.0, and the agent for Kubernetes does not support them. Speaker notes 9

VSHN – The DevOps Company Clusters configured through code Based on regular K8s RBAC rules Scaling is trivial: one agent, one environment Agent connections can be shared to other groups and projects Also supports push-based "classic" CI/CD deployments Agent supports pull-based GitOps deployments Bidirectional communication opens door to new features Advantages However, not all is bad news, since the new Agent brings some quite interesting advantages to the table: You can configure your clusters using code, including a configuration file in your projects It uses regular Kubernetes RBAC rules There is a one-to-one relationship between agents and environments Agent connections can be shared among projects Agents supports the classic CI/CD experience offered by GitLab today But it also support GitOps pull-based deployments And it offers a bidirectional communication, opening the door to new features in the future. Speaker notes 10

VSHN – The DevOps Company Integrate a cluster, located behind a firewall or NAT, with GitLab Access API endpoints in a cluster in real time Enable real-time features by pushing information about events happening in a cluster Enable a cache of Kubernetes objects through informers, kept up- to-date with very low latency …in general, removing polling Major Goals of the Agent To reach such goals, GitLab built the Kubernetes Agent with some very specific ideas in mind: It should be possible to integrate clusters behind firewalls or NATs It should be possible to access API endpoints in real time It should be possible to enable real-time features, getting live information about what’s going on in the cluster. It should be possible to cache information about Kubernetes objects in GitLab, kept updated with very low latency And in general, we should get rid of polling altogether! Speaker notes 11

VSHN – The DevOps Company Kubernetes cluster GitLab gRPC bidirectional streaming gRPC gRPC REST API agentk kas GitLab RoR Gitaly Architecture (Source: gitlab.com/gitlab-org) This diagram, taken from the technical architecture of the agent, shows the main architecture ideas of the agent, and how it works. GitLab RoR is the main GitLab application. It uses gRPC to talk to kas. agentk is the GitLab Agent. It keeps a connection established to a kas instance, waiting for requests to process. It may also actively send information about things happening in the cluster. kas is the GitLab Agent Server, and is responsible for: Accepting requests from agentk. Authentication of requests from agentk by querying GitLab RoR. Fetching agent’s configuration from a corresponding Git repository by querying Gitaly. Matching incoming requests from GitLab RoR with existing connections from the right agentk, forwarding requests to it and forwarding responses back. (Optional) Sending notifications through ActionCable for events received from agentk. Polling manifest repositories for GitOps support by communicating with Gitaly. Speaker notes 12

VSHN – The DevOps Company 1. Minimize network traffic 2. The client-server relationship is inverted 3. GitLab will add more logic into kas rather than into agentk Design Principles These major design principles of the agent are driven by two major requirements: 1. This is because the connection must be initiated from inside the Kubernetes cluster to bypass any firewall or NAT the cluster may be located behind 2. agentk should be kept streamlined and small to minimize the need for upgrades. Speaker notes 13

VSHN – The DevOps Company agentk and kas are written in Go instead of Ruby Each agentk connects to an instance of kas Connections are kept open Multiple agents per K8s cluster allowed… … each running under (potentially) a separate ServiceAccount … with different privileges! Each kas instance tracks the agents connected to it in a Redis instance Agent names follow RFC 1123 (alphanumeric, –, 63 chars max) Interesting Tidbits Would you like to know more about the GitLab Kubernetes Agent? Here go some interesting bits of information for nerds. Speaker notes 14

VSHN – The DevOps Company kas Kubernetes 3 Kubernetes 2 Kubernetes 1 register connected agents register connected agents lookup agent gRPC gRPC gRPC gRPC gRPC to any kas kas 1 kas 2 kas 3 agentk 3, Pod1 agentk 2, Pod1 agentk 1, Pod1 agentk 1, Pod2 GitLab Rails Redis (Source: gitlab.com/gitlab-org) And this other diagram also shows how the agentk and kas components work together. Speaker notes 15

VSHN – The DevOps Company GitOps Workflow: pull "Classic" CI/CD Workflow: push New Workflows So, as we mentioned previously, the new GitLab Kubernetes Agent opens up the door to pull-based deployments, similar to what Argo CD provides, but fully integrated in GitLab. Speaker notes 16

VSHN – The DevOps Company  Start with the CI/CD Workflow! Agent not exposed to the internet Does not require full cluster-admin access Migrating to the Agent Now the big question: should you migrate to the new GitLab Kubernetes Agent? Well, unless you are using self-hosted GitLab, you won’t have much of a choice, and even so, the writing’s on the wall. You might want to start using the "Classic" CI/CD push-based workflow, which offers a few advantages to the old certificate-based mechanism. Speaker notes 17

VSHN – The DevOps Company 1. In your GitLab project create a file: .gitlab/agents//config.yaml Contains GitOps Workflow configuration Empty for CI/CD Workflow! 2. Install the GitLab agent in your cluster Using helm 3. Update your .gitlab-ci.yml by selecting the proper context 4. Run your pipeline! How to Use? How to use the GitLab Kubernetes Agent? It is a very short and simple process: Create a configuration file in your GitLab project and push it to the repo. Install the agent in your cluster Update your .gitlab-ci.yml file And run your pipeline! Speaker notes 18

VSHN – The DevOps Company SaaS GitLab Exoscale SKS Cluster (Kubernetes 1.23.5) "Classic" push-based CI/CD workflow Demo! In this demo I’m going to show you how to connect a project located in SaaS GitLab to an Exoscale SKS Cluster, running Kubernetes 1.23.5. I will use the "Classic" push-based workflow, which is the simplest possible approach to get started. Speaker notes 19

VSHN – The DevOps Company  So as I’ve said previously, the GitLab to Kubernetes connectivity has changed completely… Speaker notes 20

VSHN – The DevOps Company   … featuring now a full back and forth communication mechanism, based on a small piece of software running in your cluster. Speaker notes 21

VSHN – The DevOps Company Official Documentation Announcement Blog Post Breaking changes in GitLab 15.0 More Information docs.gitlab.com/ee/user/clusters/agent/ about.gitlab.com/blog/2021/11/15/deprecating-the-cert-based- kubernetes-integration/ about.gitlab.com/blog/2022/04/18/gitlab-releases-15-breaking- changes/#configure If you would like to learn more about the GitLab Kubernetes Agent, here are some links for you. Speaker notes 22

VSHN – The DevOps Company Discussion about APPUiO Cloud Agent-less GitLab integration with OpenShift About APPUiO Cloud github.com/appuio/appuio-cloud-community/discussions/41 www.vshn.ch/en/blog/agent-less-gitlab-integration-with- openshift/ In the context of our product APPUiO Cloud, which is an "OpenShift Projects as a Service" offering, we are currently evaluating options regarding the new GitLab Kubernetes Agent. In the meantime, my colleague Christian Cremer has written an article with a simple hack to get your GitLab instance of choice connected to your Kubernetes clusters in the simplest way possible, and I strongly recommend that you check it out. Speaker notes 23

VSHN – The DevOps Company Adrian Kosmaczewski, Developer Relations, VSHN – VSHN AG – Neugasse 10 – CH-8005 Zürich – +41 44 545 53 00 – – Thanks! [email protected] vshn.ch [email protected] Thanks a lot for your attention! Do not hesitate to contact VSHN if you want to start using the GitLab Kubernetes Agent in your own projects. Speaker notes 24