VSHN – The DevOps Company Adrian Kosmaczewski Managing Kubernetes Clusters as Cattle with Project Syn GIDS Live – April 28th 2021 Thanks everyone for attending this session, and thanks Wurreka for inviting me to GIDS Live 2021. My name is Adrian Kosmaczewski, I am Developer Relations at VSHN, the DevOps company, and I speak to you from the beautiful city of Zürich in Switzerland. My local time is 08:30 in the morning, and I welcome you to this session from wherever in the world you happen to be right now. Speaker notes 1

VSHN – The DevOps Company • Pronounced ˈvɪʒn – like "vision" • The DevOps Company • Founded 2014, 40 VSHNeers located in Zürich • Switzerland’s leading DevOps, Docker & Kubernetes partner • 24/7 support • ISO 27001 certi ed • ISAE 3402 Report Type 1 veri ed • First Swiss Kubernetes Certi ed Service Provider Speaking about my team, just a few words about VSHN; that’s how you pronounce the name, and we’re "The DevOps Company". We’ve been in Zurich since 2014, we’re 40 engineers (this is actually an important data point I’m going to refer to later) and we’re Switzerland’s leading DevOps, Docker & Kubernetes partner, offering 24/7 support to our customers. We’ve got a few certifications, and most importantly, we were the First Swiss Kubernetes Certified Service Provider back in 2016. Speaker notes 2

VSHN – The DevOps Company We’re partners of many companies very active in the Cloud Native space, you might recognize some of the logos on this slide. Speaker notes 3

VSHN – The DevOps Company And last but not least, we run our own PaaS offering called "APPUiO", and even more important, we’ve created our own suite of tools to manage Kubernetes services called "Project Syn", which is exactly what I’m going to talk about today. Speaker notes 4

VSHN – The DevOps Company Join Us! vshn.ch/en/jobs handbook.vshn.ch Oh, and did I mention that we’re hiring? Check out our Handbook to know more about us. Speaker notes 5

VSHN – The DevOps Company • The Evolution of Cloud Services • Introduction to Project Syn • Project Syn in Action: Live Demo! • Questions & Answers Agenda Today I’m going to very quickly review the evolution of the cloud in the past 25 years, and where we at VSHN think it is heading right now. Project Syn embodies our thoughts and ideas about that, and I’m going to explain to you what it is, how to use it, and how useful it can be. Finally, I’m going to show you a small demo of Project Syn in action. At the end of the session I’d gladly answer some questions from you. Speaker notes 6

VSHN – The DevOps Company 2021 – 25 = 1996 Trip Back in Time Let us take a trip back in time: 25 years in the past. Speaker notes 7

VSHN – The DevOps Company 1996 Back in 1996, the World Wide Web was starting to grow dramatically. Speaker notes 8

VSHN – The DevOps Company • Perl • Java • VBScript • Objective-C (yes, WebObjects anyone?) • JavaScript (yes, Netscape Application Server!) • Lisp (only if you’re Paul Graham) Cloud Native == Web Applications We were also building "Cloud Native" applications; we just used to call them "Web Applications." They were quite monolithic; most of them built with Perl, VBScript, Java, maybe Objective-C if you were a lucky developer working with an early version of WebObjects, and even JavaScript if you were using Netscape’s own application server. Unless you were Paul Graham, in which case you were writing your web app in Lisp. Speaker notes 9

VSHN – The DevOps Company user Internet web db Servers as Pets Our servers had names. Remember that? What a weird concept nowadays! Those servers could bear boring names like "web", "db", or "assets". Speaker notes 10

VSHN – The DevOps Company user Internet Alderaan Hoth Servers as Pets If you were in a more imaginative environment, you would use names borrowed from Star Wars… Speaker notes 11

VSHN – The DevOps Company user Internet Aragorn Gandalf Servers as Pets … or The Lord of the Rings. Speaker notes 12

VSHN – The DevOps Company FTP We connected to those servers using some kind of remote desktop technology, or actually, way more often using FTP, or rcp if you were using BSD. If you were on the cutting edge, you might have used rsync, which was first released that year. Windows 95 users would happily use WS_FTP to handle those file tranfers. That is right: file transfers! We would sync the local folder in our local laptop with the folder in our server, and hope that we weren’t overwriting the changes from some other colleague. Speaker notes 13

VSHN – The DevOps Company CVS $ cvs checkout src/infrastructure $ cvs add new_component.c $ cvs commit -m "Added new component" $ cvs release src/infrastructure Some super advanced teams were using CVS to store changes – this is before Git, before Mercurial, before Subversion: CVS! Speaker notes 14

VSHN – The DevOps Company Visual SourceSafe Other, less lucky teams were using Microsoft SourceSafe. They were really less lucky. Speaker notes 15

VSHN – The DevOps Company • Apache • IIS • Oracle • SQL Server Clusters? No! There were no clusters; you just had one Apache or IIS web server, and one database server running Oracle or SQL Server, and maybe another file server serving as CDN, and everybody was happy. Speaker notes 16

VSHN – The DevOps Company At best, you had a server rack, and hopefully it didn’t look like this. (Photo by Martijn Baudoin on Unsplash: ) Speaker notes unsplash.com/@martijnbaudoin 17

VSHN – The DevOps Company Slogan The Age of the Hardware Server™®© Deployment Speed Once every few months (if any) 1996 This is the age of the hardware server, and your deployment speed is once every few months, if you are lucky, that is. Speaker notes 18

VSHN – The DevOps Company 2006 Let’s fast forward 10 years, to 2006. Speaker notes 19

VSHN – The DevOps Company CVS → Subversion $ svn checkout http://svn.server.com/svn/project_repo $ cd project_repo/trunk/ $ svn status $ svn add new_component.c $ svn commit -m "Added new component" By that time, your CVS repository might as well have been updated to Subversion. You had to adapt your application to work in Firefox and Internet Explorer 6, which seems stuck in time. You use Prototype or jQuery to load some XML data live on the browser using this neat new technology called AJAX. Speaker notes 20

VSHN – The DevOps Company If your operations team was following the latest trends, your hardware servers might be virtualized using VMWare or some other technology. If you wanted to be in the cutting edge, maybe even running as a virtual EC2 instance on the beta EC2 program, which was launched in August of that year. The truth is that probably those virtual machines still had names, and you still connected to them directly; of course not with FTP, but most probably with scp or rsync. Speaker notes 21

VSHN – The DevOps Company Web 2.0 The spread of Web 2.0 technologies such as Ruby on Rails and Django meant that your applications required much larger runtimes to work; installing all the infrastructure required for those frameworks required lots of ssh connections and testing, to make sure your applications would run correctly. Speaker notes 22

VSHN – The DevOps Company namespace :deploy do desc 'Runs any rake task, cap deploy:rake task=db:rollback' task rake: [:set_rails_env] do on release_roles([:db]) do within release_path do with rails_env: fetch(:rails_env) do execute :rake, ENV['task'] end end end end end $ cap production deploy One of the offsprings of Ruby on Rails was a tool called "Capistrano" born in 2006, and for those who remember, it was absolutely ground-breaking: it allowed developers to specify the configuration of their application environment in a Ruby script, and then have the tool automatically sync the environment running the Ruby script on your server. But of course, Capistrano sill required you to give names to your servers. Speaker notes 23

VSHN – The DevOps Company user Internet lb web1 web2 db First Cluster This way you could have several instances of the application, all configured exactly the same, and then a quick NGINX instance could load balance requests among those instances. Neat, huh? You have got yourself your first cluster. Speaker notes 24

VSHN – The DevOps Company Much more interesting than Capistrano was the release of Puppet in 2005 (also written in Ruby, clearly the most popular language back then!) Speaker notes 25

VSHN – The DevOps Company case $operatingsystem { centos, redhat: { $service_name = 'ntpd' } debian, ubuntu: { $service_name = 'ntp' } } package { 'ntp': ensure => installed, } service { 'ntp': name => $service_name, ensure => running, enable => true, subscribe => File['ntp.conf'], } file { 'ntp.conf': path => '/etc/ntp.conf', ensure => file, require => Package['ntp'], source => "puppet:///modules/ntp/ntp.conf", } Puppet was different in many ways: the configuration files were not Ruby code, just pure text, which made them more generic, and accessible to non-developers. Puppet and Capistrano kicked-off the "configuration as code" movement that swept off the world of IT operations ever since. Speaker notes 26

VSHN – The DevOps Company Slogan The Age of Virtual Machines™®© Deployment Speed Once every few weeks 2006 This is the age of the virtual machine, and your deployment speed is once every few weeks. Speaker notes 27

VSHN – The DevOps Company 2016 Fast forward 10 more years to the future! Speaker notes 28

VSHN – The DevOps Company So many have happened in the meantime: GitHub, Stack Overflow, Google App Engine, the Go programming language, and Microsoft Azure were all relesed in annus mirabilis 2008. Speaker notes 29

VSHN – The DevOps Company Red Hat OpenShift released in 2011 and Docker released in 2013. Speaker notes 30

VSHN – The DevOps Company GitLab, Kubernetes and Terraform released in 2014. Speaker notes 31

VSHN – The DevOps Company Subversion → Git $ git clone [email protected]:akosma/component-fortune.git $ cd component-fortune $ git status $ git add new_component.c $ git commit -m "Added new component" $ git push Your good old Subversion server has been ported to Git, and this repository is probably hosted on GitHub or GitLab. Speaker notes 32

VSHN – The DevOps Company Your apps are following the principles laid out in the Twelve-Factor app manifesto. Speaker notes 33

VSHN – The DevOps Company The new cool thing in 2016 is running your apps in containers (for example, using Docker Compose or Docker Swarm) or even better, in Kubernetes! This nifty container orchestrator was poised to take over the world of operations completely. Speaker notes 34

VSHN – The DevOps Company The world of IT is divided in IaaS, PaaS, and SaaS. Speaker notes 35

VSHN – The DevOps Company Couple that with the release of "The Phoenix Project" book, and the age of DevOps was finally here. Speaker notes 36

VSHN – The DevOps Company Kubernetes Cluster namespace-1 namespace-2 ingress svc1 pod1 pod2 db1 svc2 pod3 pod4 pod5 queue1 db2 user Internet Pets → Cattle Very important, nobody gives names to your servers anymore. They have turned from pets into cattle. And your Kubernetes cluster is happily serving your users, and you have different namespaces for each of the applications you run in it. Your pods come and go, and only very rarely you need to know their names (unless for example if you need to kubectl logs into one of them, or something like that) Speaker notes 37

VSHN – The DevOps Company Clusters as Pets! No notes on this slide. Speaker notes 38

VSHN – The DevOps Company Slogan The Age of Containers™®© Deployment Speed Once every few days 2016 This is the age of the container, and your deployment speed is once every few days. Speaker notes 39

VSHN – The DevOps Company 2021 One last time jump, and we’re back in the present time. Speaker notes 40

VSHN – The DevOps Company You are using GitHub actions or GitLab CI/CD pipelines to talk to your Kubernetes clusters, themselves configured through Terraform or Crossplane. Speaker notes 41

VSHN – The DevOps Company You don’t build your containers with Docker anymore, but with Podman, and you don’t store images in Docker Hub but in Quay.io or GitLab. Speaker notes 42

VSHN – The DevOps Company Thanks to AKS, EKS, GKE, Exoscale SKS, Red Hat OpenShift, and Rancher you have now various Kubernetes clusters; some for development, others for testing and quality control, and even some in the edge, running inside Raspberry Pi computers. You are running thousands of pods in hundreds of deployments, and keeping everything synchronized becomes a nightmare. IBM bought Red Hat and SUSE bought Rancher. Kubernetes is big business, measured in the hundreds of billions of dollars. And because of the current state of the world, your developers are all working from home or from Trinidad and Tobago, developing their apps and testing them locally in Minikube or K3s instances. Speaker notes 43

VSHN – The DevOps Company Clusters as Pets Cattle! But you still synchronize changes across Kubernetes clusters manually, and you even give names to those clusters. That’s cute, but as history shows, that doesn’t scale. Time for a change. Speaker notes 44

VSHN – The DevOps Company Slogan The Age of Clusters™®© Deployment Speed Once every few hours 2021 This is the age of the cluster, and your deployment speed is once every few hours. Speaker notes 45

VSHN – The DevOps Company Kubernetes Has Won We at VSHN believe that Kubernetes has won. It is the de facto platform for Cloud Native applications, and the size and dynamics of its ecosystem are just exploding as we speak. These days, teams need more than just one Kubernetes cluster. A single namespace is just not enough. Teams need lots of clusters, for various reasons: testing, compliance, development, etc. And they need to bring them up and down very quickly, and to ensure lots of specific configurations and apps are synchronized across those pieces of infrastructure. Operators need to make sure that their clusters are compliant in terms of security, features, configuration, payloads, and so much more. Speaker notes 46

VSHN – The DevOps Company This is where Project Syn comes into play. Remember the beginning of this talk, when I mentioned that VSHN has only 40 VSHNeers? Well, only 30 of them are actual engineers, and they manage almost a 100'000 services in several thousand clusters spread all over the world, in almost all cloud providers you can imagine. This scale of work is simply unimaginable without strong principles and tight collaboration patterns. We built Project Syn to allow us to manage such a huge amount of services, providing 24/7 support to all of our clients. Speaker notes 47

VSHN – The DevOps Company user Internet cluster1 cluster2 cluster3 All of this means that at VSHN we breathe DevOps every day. All of our infrastructure, as well as that from our customers, is fully configured as code and stored in Git repos. Every change is tracked through GitHub pull requests and GitLab merge requests, so that many pairs of eyeballs could make sure that we are doing the right thing, and that we are doing the thing right. This also allows us to roll back changes in all security and confidence. Speaker notes 48

VSHN – The DevOps Company user Internet cluster1 cluster2 cluster3 cluster4 cluster5 cluster6 cluster7 And Project Syn extends this notion to support multiple Kubernetes clusters, anywhere in the world. Speaker notes 49

VSHN – The DevOps Company What is Project Syn? Project Syn is a set of tools helping to securely manage a eet of Kubernetes clusters. It brings a hierarchical con guration management based on GitOps principles, reusable components and an inventory of information about all Kubernetes clusters. Project Syn is a series of tools used to manage Kubernetes-at-large, or as I like to call it, "Kubernetes as Cattle". Speaker notes 50

VSHN – The DevOps Company • Multi-Cluster and Multi-Tenant Management • Full GitOps Operations • Tooling Bootstrapping • Con guration Management • Reusable Components • Default Toolset • Automatically Maintained Components • Secret Management • Service Provisioning Features Project Syn provides the following features. Speaker notes 51

VSHN – The DevOps Company • Lieutenant Operator • Lieutenant API • Steward • Commodore Main Components These are the main components of Project Syn: • Lieutenant Operator: Manages the CRDs Cluster, Tenant, GitRepo. Implements the business logic for handling all aspects around Git repositories for clusters and tenants. • Lieutenant API: REST API to expose the functionality of the Lieutenant Operator and to enforce some defaults. • Steward: In-cluster agent which bootstraps, configures and starts Argo CD (GitOps operator) and takes care of keeping the Syn cluster tools up and running. • Commodore: Configuration generator leveraging a hierarchical configuration model. Uses Kapitan under the hood, and can be extended with components. Each one of these projects is available in GitHub. Speaker notes 52

VSHN – The DevOps Company Git repo created & configured by Lieutenant Operator Cluster information SSH keys Project Syn Tool Lieutenant Operator Deployment Reconciliation Renovate Get cluster, tenant and Git repo information Cluster catalog Tenant and cluster configuration Project Syn enabled Kubernetes cluster Configuration Git repositories Common configuration Update Push compiled catalog Lieutenant API Commodore Legend Git repository Commodore Components Steward Argo CD Vault Update This diagram shows how all the pieces come together. Speaker notes 53

VSHN – The DevOps Company • Extension points • Written in Jsonnet • 50+ already published in GitHub • Commodore Component Hub in the works Commodore Components Commodore Components are the extension point of Project Syn. They allow you to customize and extend the number and variety of tasks that you can perform with Project Syn. Using Commodore Components, you can customize and apply changes to all of your clusters, ensuring conformity, coherence, alignment, and drastically reducing the workload for DevOps engineers. There are also lots of Commodore Components ready to use on GitHub; at the time of this writing, there are 50 components tagged commodore-component in the platform. Backup, security, networking, storage, you name it: there’s a Commodore Component for that! Commodore Components are written in a configuration language called Jsonnet, which extends JSON and compiles to other configuration languages such as TOML or YAML. Jsonnet is very simple to learn, and at first sight it looks like JSON with variables and reusability. We are planning the creation of a "Commodore Components Hub" to federate all of the documentation of all Commodore Components in a central location. Stay tuned for news! Speaker notes 54

VSHN – The DevOps Company • Rancher Fleet • CAOS ORBOS Similar Tools Of course, Project Syn is not alone in the race. Our two major competitors are excellent products, but even if they overlap a little, they have slightly different target use cases. The main goal of Fleet is to deploy applications to remote clusters, called the Fleet agents, whereas Project Syn focuses on generating configuration prepared for each cluster, depending on the properties of the destination cluster. Fleet implements its own GitOps model, rather than reusing existing tools like Flux or Argo CD like Project Syn does. Contrary to Project Syn, ORBOS focuses on the full lifecycle of Kubernetes clusters, whereas Project Syn doesn’t provide support for provisioning and managing Kubernetes Clusters themselves. Instead, Project Syn delegates cluster lifecycle management to specialized tooling and focuses on managing the content on a Kubernetes cluster. Speaker notes 55

VSHN – The DevOps Company 1. Minikube 2. K3s 3. Kind 4. K9s Demo Time! Show, don’t tell! This demo will show Project Syn in action. I have three Kubernetes clusters running in my own laptop in this precise moment: 1. Minikube, running the Lieutenant API and Operator & Steward 1. The Lieutenant API is exposed to the outer world using ngrok 2. K3s, running Steward 3. Kind, running Steward 1. Steward is monitoring Git repositories with all the configuration. I am using my own GitLab.com account to store those Git repositories. Speaker notes 56

VSHN – The DevOps Company In this demo, I will use Project Syn to deploy a simple application to both clusters at once. This application is specified as a Commodore Component. By compiling and pushing the catalog after adding the application, we will see it appear like magic in our clusters. We can remove it as simply, just by removing the reference to it, pushing our changes to Git, and recompiling our catalog. The Jsonnet files that make up my component are very simple, and they actually represent the structure of the YAML required to deploy my application on a cluster. They represent a Kubernetes namespace, a deployment, and a service. Once they are applied onto a cluster, the application they deploy is contained in a Docker image called "fortune-cookie-service". This Docker image encapsulates a very simple application written in Python, which returns a funny "fortune cookie of the day" type of message, and a random number. This app can be called from any web browser, or on the command line, using curl, as if it were a simple API. Speaker notes 57

VSHN – The DevOps Company (Link only valid during the GIDS Live 2021 session!) Try it now! fortune.ngrok.io If you are a live attendee of GIDS Live 2021, try opening this link on your browser, and you’ll get a funny fortune cookie of the day, courtesy of GIDS and VSHN! Speaker notes 58

VSHN – The DevOps Company • Main website: • GitHub: • Chat: • Code of Conduct: • Syn Design Documents (SDDs): • Getting started: • Commodore Components tutorial: Call to Action syn.tools github.com/projectsyn community.appuio.ch/channel/projectsyn syn.tools/syn/about/code_of_conduct.html syn.tools/syn/SDDs/index.html syn.tools/syn/tutorials/getting-started.html syn.tools/tuto/index.html Project Syn is 100% open source, using the BSD 3- Clause "New" or "Revised" License. We even make our decision process open to everyone, through our Syn Design Documents or SDDs, which you can read online at the official website of Project Syn, . The demo I have shown is available online, so please check it out and try it out. It contains all the scripts required for you to verify the minimum requirements, and then to spin up your clusters with the least possible amount of work. Speaker notes syn.tools 59

VSHN – The DevOps Company • Lieutenant API: 0.5.0 • Lieutenant Operator: 0.5.2 • Commodore: 0.6.0 • Steward: 0.3.1 Current Versions We’re actively working on Project Syn as we speak. The current versions of our tools at the time of this talk are the following: • Lieutenant API: 0.5.0 • Lieutenant Operator: 0.5.2 • Commodore: 0.6.0 • Steward: 0.3.1 In spite of the "0.something" version numbers, Project Syn is already in production in VSHN since 2020, and we are actively using it every day to deliver services to our customers. Please join our chat channel if you decide to use Project Syn in your infrastructure, and let us know how you are using it, and if you have any feedback we’d be glad to hear about it. Speaker notes 60

VSHN – The DevOps Company Get the slides here: Adrian Kosmaczewski – VSHN AG – Neugasse 10 – CH-8005 Zürich – +41 44 545 53 00 – – Thanks! bit.ly/gids-cattle [email protected] vshn.ch [email protected] I’d like to thank you all for having watched this presentation, and please do not hesitate to contact us if you have any questions about Project Syn. Speaker notes 61