Navigating the CI/CD Landscape in the K8S Era Andreas Evers CTO KOR Financial

Abstract With Kubernetes, everything from deployment to monitoring to scaling is more standardized and inherently easier to automate. This presents the possibility to achieve a more effective and comprehensive Continuous Integration (CI) and Continuous Delivery (CD) experience. We can incorporate the practice of infrastructure-as-code, and take advantage of a flourishing ecosystem of tools to improve and fully automate our application deployment strategies. In this session, we’ll tour the landscape of new Kubernetes-centric tooling, including Tekton, Kustomize, and ArgoCD. We’ll also explore how GitOps can be leveraged for CI/CD.

KOR is a Fintech created by a global distributed team build up out of expert in their field. Our engineering team is centered in the EU region. Product is driven by industry expert from the US and Leadership is spread between UK, US and EU. ~ We are hiring ~ People Powered Disruption

Mission: Avoid another financial crisis by mapping out risk in financial markets

Selena Gomez explains “CDOs”, one of the financial products that were the cause of the 2008 crash

No content

We are building Real-time insights based on regulation to make financial products like CDO, transparent to the market and regulators.

Architecture For Scale Cloud native Event driven architecture Fully hosted on Amazon Web Services ~12 Petabytes per year Event Sourcing / Stream Processing Kafka as a Database

package Path To Production change prod code test deploy test deploy promote rollback test strategic rollout reviews pull request feature branch declared state maintain test kubernetes infrastructure as code self-healing extensible api git source of truth version control pull/merge request flows triggers buildpacks build images autonomous rebuilds patch OS at scale gitops declarative desired state continuous operations tekton pipelined workflows

10 ● CNCF Sandbox project initiated by Pivotal and Heroku in 2018 ○ Contributors include Google, Microsoft, and others ● Well-defined platform-to-buildpack contract to take source code and output OCI images ● Focus on how to orchestrate builds, not how to run an app ● Unified standard to enable ecosystem of platforms and modular buildpacks ● Optimize for fast rebuilds, minimal data transfer, and efficient OS rebasing Cloud Native Buildpacks (CNB)

pack kpack Spring Boot Tekton Tanzu Build Service az acr pack Google Cloud Run Button builder image buildpacks pla$orm run image * Platform Ecosystem Buildpacks Ecosystem Cloud Native Buildpacks (CNB)

12 kpack: Buildpacks as a Service ● Automatically rebuild images based on changes in inputs ● Efficiently rebase OS layers with a CVE hotfix (new run-image) ● Consistent, repeatable, transparent and secure builds -- at scale Hosted, Centralized, Automated

13 Primarily a Package Manager (Charts), used for configuration customization as well Parameterized kubernetes resources + values.yml files for different environments Lots of {{-if / else}} switches The Kubernetes Package Manager Helm ● Excellent for deploying OTS software ● Unreadable GoLang templating (example) ● Mandatory semver ● Not great for CD ● Non-declarative by default

14 Does one thing well Cleanly separates static resources (fixed), the resource changes (overlay), and the config that combines them (kustomize) Kubernetes native configuration management Kustomize ● ConfigMap lifecycle mgmt ● Part of kubectl ● Simplicity ● Transformers ● Not every part of the resources are “selectable” for changes

15 #1. The entire system described declaratively. Kubernetes is just one example of many modern cloud native tools that are “declarative” and that can be treated as code. Declarative means that configuration is guaranteed by a set of facts instead of by a set of instructions. With your application’s declarations versioned in Git, you have a single source of truth. Your apps can then be easily deployed and rolled back to and from Kubernetes. And even more importantly, when disaster strikes, your cluster’s infrastructure can also be dependably and quickly reproduced. #2. The canonical desired system state versioned in Git. With the declaration of your system stored in a version control system, and serving as your canonical source of truth, you have a single place from which everything is derived and driven. This trivializes rollbacks; where you can use a `Git revert` to go back to your previous application state. With Git’s excellent security guarantees, you can also use your SSH key to sign commits that enforce strong security guarantees about the authorship and provenance of your code. #3. Approved changes that can be automatically applied to the system. Once you have the declared state kept in Git, the next step is to allow any changes to that state to be automatically applied to your system. What's significant about this is that you don't need cluster credentials to make a change to your system. With GitOps, there is a segregated environment of which the state definition lives outside. This allows you to separate what you do and how you're going to do it. #4. Software agents to ensure correctness and alert on divergence. Once the state of your system is declared and kept under version control, software agents can inform you whenever reality doesn’t match your expectations. The use of agents also ensures that your entire system is self-healing. And by self-healing, we don’t just mean when nodes or pods fail—those are handled by Kubernetes—but in a broader sense, like in the case of human error. In this case, software agents act as the feedback and control loop for your operations. Four Principles of GitOps GitOps

16 INSTRUCTIONS 1. From city center, turn onto Route 4 north. 2. At the traffic circle, take the first exit 3. At the first light, turn right. 4. Take the first available right turn. 5. At the traffic circle, start in the left lane and take the third exit. 6. The library is the first building on that road. How to get to the library? Imperative vs. Declarative FACTS ● The library is located at 10 Community Way East Greenbush, NY 12061 United States uninvolved instructor, smart engine can recover from any starting point smart instructor, dumb engine must begin at starting point

17 IMPERATIVE ● kubectl create … ● kubectl run … ● kubectl expose … ● kubectl edit … ● kubectl patch ... ● kubectl label … ● kubectl scale … ● kubectl annotate … How to get to the desired state? Imperative vs. Declarative DECLARATIVE ● kubectl apply … Controllers are the smart engines that converge running resources to desired state ✓ Great for creating reproducible deployments ✓ Great for creating yamls, troubleshooting, learning, experimentation

18 #1. The entire system described declaratively. Kubernetes is just one example of many modern cloud native tools that are “declarative” and that can be treated as code. Declarative means that configuration is guaranteed by a set of facts instead of by a set of instructions. With your application’s declarations versioned in Git, you have a single source of truth. Your apps can then be easily deployed and rolled back to and from Kubernetes. And even more importantly, when disaster strikes, your cluster’s infrastructure can also be dependably and quickly reproduced. #2. The canonical desired system state versioned in Git. With the declaration of your system stored in a version control system, and serving as your canonical source of truth, you have a single place from which everything is derived and driven. This trivializes rollbacks; where you can use a `Git revert` to go back to your previous application state. With Git’s excellent security guarantees, you can also use your SSH key to sign commits that enforce strong security guarantees about the authorship and provenance of your code. #3. Approved changes that can be automatically applied to the system. Once you have the declared state kept in Git, the next step is to allow any changes to that state to be automatically applied to your system. What's significant about this is that you don't need cluster credentials to make a change to your system. With GitOps, there is a segregated environment of which the state definition lives outside. This allows you to separate what you do and how you're going to do it. #4. Software agents to ensure correctness and alert on divergence. Once the state of your system is declared and kept under version control, software agents can inform you whenever reality doesn’t match your expectations. The use of agents also ensures that your entire system is self-healing. And by self- healing, we don’t just mean when nodes or pods fail—those are handled by Kubernetes—but in a broader sense, like in the case of human error. In this case, software agents act as the feedback and control loop for your operations. Four Principles of GitOps GitOps

19 ➔ Code vs configuration ➔ Separation of access, cleaner audit logs ➔ Enables modular pipelines (“elbow joints”) ➔ Trigger action based on different life cycles ➔ Makes sense for coupled deployments comprised out of multiple repositories Separate repositories

20 #1. The entire system described declaratively. Kubernetes is just one example of many modern cloud native tools that are “declarative” and that can be treated as code. Declarative means that configuration is guaranteed by a set of facts instead of by a set of instructions. With your application’s declarations versioned in Git, you have a single source of truth. Your apps can then be easily deployed and rolled back to and from Kubernetes. And even more importantly, when disaster strikes, your cluster’s infrastructure can also be dependably and quickly reproduced. #2. The canonical desired system state versioned in Git. With the declaration of your system stored in a version control system, and serving as your canonical source of truth, you have a single place from which everything is derived and driven. This trivializes rollbacks; where you can use a `Git revert` to go back to your previous application state. With Git’s excellent security guarantees, you can also use your SSH key to sign commits that enforce strong security guarantees about the authorship and provenance of your code. #3. Approved changes that can be automatically applied to the system. Once you have the declared state kept in Git, the next step is to allow any changes to that state to be automatically applied to your system. What's significant about this is that you don't need cluster credentials to make a change to your system. With GitOps, there is a segregated environment of which the state definition lives outside. This allows you to separate what you do and how you're going to do it. #4. Software agents to ensure correctness and alert on divergence. Once the state of your system is declared and kept under version control, software agents can inform you whenever reality doesn’t match your expectations. The use of agents also ensures that your entire system is self-healing. And by self-healing, we don’t just mean when nodes or pods fail—those are handled by Kubernetes—but in a broader sense, like in the case of human error. In this case, software agents act as the feedback and control loop for your operations. Principles of GitOps GitOps

21 #1. The entire system described declaratively. Kubernetes is just one example of many modern cloud native tools that are “declarative” and that can be treated as code. Declarative means that configuration is guaranteed by a set of facts instead of by a set of instructions. With your application’s declarations versioned in Git, you have a single source of truth. Your apps can then be easily deployed and rolled back to and from Kubernetes. And even more importantly, when disaster strikes, your cluster’s infrastructure can also be dependably and quickly reproduced. #2. The canonical desired system state versioned in Git. With the declaration of your system stored in a version control system, and serving as your canonical source of truth, you have a single place from which everything is derived and driven. This trivializes rollbacks; where you can use a `Git revert` to go back to your previous application state. With Git’s excellent security guarantees, you can also use your SSH key to sign commits that enforce strong security guarantees about the authorship and provenance of your code. #3. Approved changes that can be automatically applied to the system. Once you have the declared state kept in Git, the next step is to allow any changes to that state to be automatically applied to your system. What's significant about this is that you don't need cluster credentials to make a change to your system. With GitOps, there is a segregated environment of which the state definition lives outside. This allows you to separate what you do and how you're going to do it. #4. Software agents to ensure correctness and alert on divergence. Once the state of your system is declared and kept under version control, software agents can inform you whenever reality doesn’t match your expectations. The use of agents also ensures that your entire system is self-healing. And by self- healing, we don’t just mean when nodes or pods fail—those are handled by Kubernetes—but in a broader sense, like in the case of human error. In this case, software agents act as the feedback and control loop for your operations. Principles of GitOps GitOps

22 #1. The entire system described declaratively. Kubernetes is just one example of many modern cloud native tools that are “declarative” and that can be treated as code. Declarative means that configuration is guaranteed by a set of facts instead of by a set of instructions. With your application’s declarations versioned in Git, you have a single source of truth. Your apps can then be easily deployed and rolled back to and from Kubernetes. And even more importantly, when disaster strikes, your cluster’s infrastructure can also be dependably and quickly reproduced. #2. The canonical desired system state versioned in Git. With the declaration of your system stored in a version control system, and serving as your canonical source of truth, you have a single place from which everything is derived and driven. This trivializes rollbacks; where you can use a `Git revert` to go back to your previous application state. With Git’s excellent security guarantees, you can also use your SSH key to sign commits that enforce strong security guarantees about the authorship and provenance of your code. #3. Approved changes that can be automatically applied to the system. Once you have the declared state kept in Git, the next step is to allow any changes to that state to be automatically applied to your system. What's significant about this is that you don't need cluster credentials to make a change to your system. With GitOps, there is a segregated environment of which the state definition lives outside. This allows you to separate what you do and how you're going to do it. #4. Software agents to ensure correctness and alert on divergence. Once the state of your system is declared and kept under version control, software agents can inform you whenever reality doesn’t match your expectations. The use of agents also ensures that your entire system is self-healing. And by self-healing, we don’t just mean when nodes or pods fail—those are handled by Kubernetes—but in a broader sense, like in the case of human error. In this case, software agents act as the feedback and control loop for your operations. Principles of GitOps GitOps

23 Concept of Application (set of Kubernetes resources) Uses CRDs to store this information Focused on enterprise application deployments Provides UI with enterprise authentication capabilities ArgoCD

24 Doesn’t have the Application context, Flux is used primarily to bootstrap clusters where the application layer hasn’t necessarily been deployed yet Can scan image repositories for new versions and update the cluster automatically Intentionally doesn’t provide a UI Weave Flux

25 GitOps Promotions ✅ One repo for all pipelines ⛔ Three separate pipelines in source repo ⛔ Possibly no notification when deploy doesn't complete

26 GitOps Promotions

27 GitOps Promotions ✅ One repo for all pipelines ✅ One pipeline to rule them all ✅ Guaranteed pipeline failure when deploy doesn't complete ⛔ The PRs keep the source pipeline execution waiting until merged

28 GitOps Promotions

29 GitOps Promotions ⛔ Three separate repos for all pipelines ⛔ Three separate pipelines in three repos ✅ Guaranteed pipeline failure when deploy doesn't complete

30 GitOps Promotions

31 GitOps Promotions ✅ Only one deployment repo to track ⛔ Three separate pipelines ✅ Guaranteed pipeline failure when deploy doesn't complete ⛔ No RBAC controls

32 GitOps Promotions

33 Jenkins Pipeline - Stage - Step Concourse Resource - Job - Task Spinnaker Stage - Step - Task CircleCI Workflow - Job - Step Workflow Automation GitLab CI Stage - Job Travis Stage - Job ⬤ ⬤ ⬤

34 Jenkins Pipeline - Stage - Step Concourse Resource - Job - Task Spinnaker Stage - Step - Task CircleCI Workflow - Job - Step Workflow Automation GitLab CI Stage - Job Travis Stage - Job Tekton Pipeline - Task - Step ⬤ ⬤ ⬤

35 Generic workflow engine utilizing CRDs, happens to be really good at CD with Kubernetes Standardization of pipelines native to Kubernetes Provides other building blocks than pipelines as well Kubernetes-native Workflow Engine Tekton

36 Designed to run on and target Kubernetes Quickstarts & buildpacks Uses Prow (Lighthouse), Draft and Tekton internally Supports promotion flows with on-demand environments Hides Tekton’s complexity, but provides its flexibility Jenkins Kubernetes edition Jenkins X

37 Keep it simple, stupid People over Processes over Tools

38 Keep it simple, stupid People over Processes over Tools

39 Powerful turn-key multi-cloud deployment tool Advanced deployment strategies including automated canary analysis Keeps track of state of your deployments and uses it to drive pipelines Offers composite UI across clouds, clusters, regions and deployment platforms Complex to set up and a platform on its own to maintain Lacks declarative configuration of pipelines Deployment platform for fast and safe deployments Spinnaker

40 GitLab is a comprehensive SDLC platform, containing a Git server, an image registry, issue tracker, application monitoring, security scanning, … The differentiating feature is Auto DevOps It automatically sets up the complete pipeline and necessary integrations ● detects the language of the code automatically builds ● tests and measures code quality ● scans for potential vulnerabilities, security flaws and licensing issues ● packages ● instruments (for monitoring in real-time) and deploys the application It provides an opinionated pipeline out-of-the-box Jenkins Kubernetes edition GitLab CI

41 Avoids boilerplate code which is the same for 80% of the projects by abstracting it. Examples of opinionated solutions: ● Spring Boot provides an opinionated interpretation of the Spring framework abstracted in libraries ● Cloud Native Buildpacks provide an opinionated way of building and containerizing your application ● Spinnaker provides opinionated advanced deployment strategies ● Jenkins X quickstart uses Draft packs to provide an opinionated way of bootstrapping your pipeline ● GitLab’s Auto DevOps provides an opinionated full-featured end-to-end pipeline, including support for Cloud Native Buildpacks Characteristics Opinionated Pipelines

package Path To Production change prod code test deploy test deploy promote rollback test strategic rollout reviews pull request feature branch declared state maintain test

Thanks Andreas Evers andreasevers www.KORfinancial.com