Kubernetes' Architecture Deep Dive - Umeå May 2019

A very improved version building on top of: https://speakerdeck.com/luxas/kubernetes-architecture-fundamentals
Online slides: https://docs.google.com/presentation/d/1rtiCqBZrPkwSiXQE8g1t2Z3IrBvgyxdUgsiHu2GCy4I/edit

I did this presentation at Umeå University for the WASP (https://wasp-sweden.org) PhD programme. The session was not recorded.
Location: Umeå University, Dept. Of Computing Science, Umeå, Sweden

Transcript

1. 1 Kubernetes’ Architecture Deep Dive Lucas Käldström - CNCF Ambassador 7th of May 2019 - Umeå Image credit: @ashleymcnamara
2. 2 $ whoami Lucas Käldström, High School Student, 19 years old CNCF Ambassador, Certified Kubernetes Administrator and Kubernetes SIG Lead KubeCon Speaker in Berlin, Austin, Copenhagen, Shanghai & Seattle Kubernetes approver and subproject owner, active in the community for ~3 years. Got kubeadm to GA. Driving luxas labs which currently performs contracting for Weaveworks A guy that has never attended a computing class
3. An intro to CNCF Cloud Native Computing Foundation helps us all succeed
4. 4 = OPEN SOURCE CLOUD COMPUTING FOR APPLICATIONS
5. 5 Cloud Native Computing Foundation • Nonprofit, part of the Linux Foundation; founded Dec 2015 • Platinum members: Incubating Service Mesh Storage Service Discovery Distributed Tracing Software Update Spec Security Graduated Package Management Orchestration Networking API Service Proxy Logging Remote Procedure Call Distributed Tracing API Container Runtime Container Runtime Messaging Monitoring Registry Storage Key/Value Store Source
6. 6 CNCF Project Maturities INNOVATORS “TECHIES” EARLY MAJORITY “PRAGMATISTS” LAGGARDS “SKEPTICS” “THE CHASM” LATE MAJORITY “CONSERVATIVES” SANDBOX GRADUATED INCUBATING EARLY ADOPTERS “VISIONARIES” SANDBOX Identity Spec Identity Policy Tooling Metrics Spec Key/Value Store Monitoring Serverless Packaging Spec Container Security Image Distribution Nodeless Source
8. 8 Cloud Native Trail Map Trail Map: l.cncf.io Source
9. 9 CNCF Cloud Native Definition v1.0 Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach. These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil. The Cloud Native Computing Foundation seeks to drive adoption of this paradigm by fostering and sustaining an ecosystem of open source, vendor-neutral projects. We democratize state-of-the-art patterns to make these innovations accessible for everyone. Source
10. 10 Online, Proctored Kubernetes Exams • Certified Kubernetes Administrator (CKA) – Over 3,500 registrations already – https://www.cncf.io/certification/expert/cka/ • Certified Kubernetes Application Developer (CKAD) – Certifies that users can design, build, configure, and expose cloud native applications for Kubernetes – https://www.cncf.io/certification/expert/cka/ckad/ • Both tests – Tests consist of a set of scenarios to resolve from the command line over 3 hours; there is no multiple choice – Each exam is $300 – Quarterly exam updates to match K8s releases` Source
11. 11 Certified Kubernetes Conformance • CNCF runs a software conformance program for Kubernetes – Implementations run conformance tests and upload results – New mark and more flexible use of Kubernetes trademark for conformant implementations – cncf.io/ck Source
12. 12 75 Certified Kubernetes Partners Source
13. 13 KubeCon + CloudNativeCon • Europe 2019 (sponsorships and CFP open) – Barcelona: May 20-23, 2019 • China 2019 (sponsorships open) – Shanghai: June 24-26, 2019 • North America 2019 (sponsorships open) – San Diego: November 18-21, 2019 Source
14. 14 KubeCon + CloudNativeCon Attendance First CNCF-organized event Source
15. Kubernetes on a high-level Kubernetes lets you efficiently declaratively manage your apps at any scale
16. 16 Most importantly: What does “Kubernetes” mean? = Greek for “pilot” or “helmsman of a ship”
17. 17 What is Kubernetes? = A Production-Grade Container Orchestration System • A project that was spun out of Google as an open source container orchestration platform (~2 billion containers/week). • Built from the lessons learned in the experiences of developing and running Google’s Borg and Omega. • Designed from the ground-up as a loosely coupled collection of components centered around deploying, maintaining and scaling workloads.
18. 18 What Does Kubernetes do? • Known as the linux kernel of distributed systems. • Abstracts away the underlying hardware of the nodes and provides a uniform interface for workloads to be both deployed and consume the shared pool of resources. • Works as an engine for resolving state by converging the actual and the desired state of the system.
19. 19 Kubernetes is self-healing Kubernetes will ALWAYS try and steer the cluster to its desired state. • Me: “I want 3 healthy instances of Redis to always be running.” • Kubernetes: “Okay, I’ll ensure there are always 3 instances up and running.” • Kubernetes: “Oh look, one has died. I’m going to attempt to spin up a new one.”
20. 20 What can Kubernetes REALLY do? • Autoscale Workloads • Blue/Green Deployments • Fire off Jobs and scheduled CronJobs • Manage Stateless and Stateful Applications • Built-in Service Discovery • ~Easily integrate and support 3rd party apps~
21. 21 Most Importantly... Use the SAME API across bare metal and EVERY cloud provider!!!
22. 22 Kubernetes’ incredible velocity (last 365 days!) 32 000+ human commits 15 000+ contributors 51 000+ opened Pull Requests 73 000+ opened issues 88 000+ Kubernetes professionals 35 000+ Kubernetes jobs 55 000+ users on Slack 50 000+ edX course enrolls Source 5 Source 4 Last updated: 09.01.2019 Source 2 318 000+ Github comments Source 1 Source 3
23. 23 Kubernetes is a “platform-platform”
24. Kubernetes’ Architecture
25. 25 Nodes Control Plane Kubernetes’ high-level component architecture Node 3 OS Container Runtime Kubelet Networking Node 2 OS Container Runtime Kubelet Networking Node 1 OS Container Runtime Kubelet Networking API Server (REST API) Controller Manager (Controller Loops) Scheduler (Bind Pod to Node) etcd (key-value DB, SSOT) User Legend: CNI CRI OCI Protobuf gRPC JSON
26. 26 kube-apiserver, the heart of the cluster • Provides a forward facing REST interface into the Kubernetes control plane and datastore. • All clients and other applications interact with Kubernetes strictly through the API Server. • Acts as the gatekeeper to the cluster by handling authentication and authorization, request validation, mutation, and admission control in addition to being the front-end to the backing datastore.
27. 27 etcd, the key-value datastore • etcd acts as the cluster datastore. • A standalone incubating CNCF project • Purpose in relation to Kubernetes is to provide a strong, consistent and highly available key-value store for persisting all cluster state. • Uses “Raft Consensus” among a quorum of systems to create a fault-tolerant consistent “view” of the cluster.
28. 28 kube-controller-manager, the reconciliator • Serves as the primary daemon that manages all core components’ reconcilation loops. • Handles a lot of the business logic of Kubernetes. • Monitors the cluster state via the API Server and steers the cluster towards the desired state. • List of core controllers
29. 29 kube-scheduler, the placement engine • Verbose policy-rich engine that evaluates workload requirements and attempts to place it on a matching resource. • The default scheduler uses the “binpacking” mode. • Workload Requirements can include: general hardware requirements, affinity/anti-affinity, labels, and other various custom resource requirements. • Is swappable, you can create your own scheduler
30. 30 kubelet, the node agent • Acts as the node agent responsible for managing the lifecycle of every pod on its host. • Kubelet understands JSON/YAML container manifests that it can read from several sources: ◦ Watching the API server (the primary mode) ◦ A directory with files ◦ A HTTP Endpoint ◦ HTTP Server mode accepting container manifests over a simple API.
31. 31 Container Runtime, the executor • A container runtime is a CRI (Container Runtime Interface) compatible application that executes and manages containers. ◦ Docker (default, built into the kubelet atm) ◦ containerd ◦ cri-o ◦ rkt ◦ Kata Containers (formerly clear and hyper) ◦ Virtlet (VM CRI compatible runtime)
32. 32 Container Network Interface (CNI) • Pod networking within Kubernetes is plumbed via the Container Network Interface (CNI). • Functions as an interface between the container runtime and a network implementation plugin. • CNCF Project • Uses a simple JSON Schema.
33. 33 Kubernetes Networking • Pod Network (third-party implementation) ◦ Cluster-wide network used for pod-to-pod communication managed by a CNI (Container Network Interface) plugin. • Service Network (kube-proxy) ◦ Cluster-wide range of Virtual IPs managed by kube-proxy for service discovery.
34. 34 kube-proxy, the Service proxier • Manages the network rules for Services on each node. • Performs connection forwarding or load balancing for Kubernetes Services. • Available Proxy Modes: ◦ ipvs (default if supported) ◦ iptables (default fallback) ◦ userspace (legacy)
35. 35 Third-party CNI Plugins for Pod Networking • Amazon ECS • Calico • Cillium • Contiv • Contrail • Flannel • GCE • kube-router • Multus • OpenVSwitch • Romana • Weave Net
36. 36 Cluster DNS, today CoreDNS • Provides Cluster Wide DNS for Kubernetes Services. ◦ CoreDNS (current default) ◦ kube-dns (default pre-1.13) • Resolves `{name}.{namespace}.svc.cluster.local` queries to the Service Virtual IPs.
37. 37 The Kubernetes Dashboard A limited, general purpose web front end for the Kubernetes Cluster.
38. 38 Kubernetes is extensible (!) • CNI (Container Network Interface) - write your own network plugin for the cluster • CRI (Container Runtime Interface) - write your own container runtime for Kubernetes • CSI (Container Storage Interface) - write your own persistent storage plugin • Device Plugins - register custom devices present on your Node • Pluggable cloud providers - write your own extension that integrates with your cloud • CustomResourceDefinitions - create your own Kubernetes Resources easily • API Aggregation - proxy a new API group from the core API server to your extension
39. Kubernetes Applied Dive into how to use Kubernetes in practice
40. 40 Create a cluster with kubeadm 1. Provision a Linux machine 2. Install kubeadm from package repos or by hand: 3. Bootstrap the Control Plane Node easily: 4. Install a 3rd-party Pod Network Provider (e.g. Weave Net): 5. Repeat step 1 & 2 on any Node to join to the cluster: curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - echo “deb http://apt.kubernetes.io/ kubernetes-xenial main” > /etc/apt/sources.list.d/k8s.list apt-get update && apt-get install -y kubeadm docker.io kubeadm init kubectl apply -f https://git.io/weave-kube-1.6 kubeadm join --token <token> <master-ip>:6443
41. 41 kubeadm Master 1 Master N Node 1 Node N kubeadm kubeadm kubeadm kubeadm Cloud Provider Load Balancers Monitoring Logging Cluster API Spec Cluster API Cluster API Implementation Addons Kubernetes API Bootstrapping Machines Infrastructure = The official tool to bootstrap a minimum viable, best-practice Kubernetes cluster Layer 2 kubeadm Layer 3 Addon Operators Layer 1 Cluster API
42. 42 end-to-end solution Master 1 Master N Node 1 Node N kubeadm kubeadm kubeadm kubeadm Cloud Provider Load Balancers Monitoring Logging Cluster API Spec Cluster API Cluster API Implementation Addons Kubernetes API Bootstrapping Machines Infrastructure kubeadm vs an “end-to-end solution” kubeadm is built to be part of a higher-level solution
43. 43 75 Certified Kubernetes Partners Source
44. 44 The core primitive: A Pod • The basic, atomically deployable unit. • Consists of one or many co-located containers. • Represents a single instance of an application. apiVersion: v1 kind: Pod metadata: name: nginx namespace: default labels: app: nginx spec: containers: - image: nginx:1.13.9 name: nginx ports: - name: http containerPort: 80
45. 45 The core primitive: A Pod apiVersion: v1 kind: Pod metadata: name: nginx namespace: default labels: app: nginx spec: containers: - image: nginx:1.13.9 name: nginx ports: - name: http containerPort: 80 • Has its own random internal IP address • Containers share network & volumes • Is non-mutable, hence mortal; dies on Node shutdown or rolling upgrades
46. 46 A replicated, upgradeable set of Pods: A Deployment apiVersion: apps/v1 kind: Deployment metadata: labels: app: nginx name: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: nginx:1.13.9 name: nginx ports: - name: http containerPort: 80 Pod Template • Manages Pods in a declarative & upgradable manner • Creates $replicas Pods from a template • Is mutable, performs a rolling upgrade of the Pods when changed
47. 47 Various possible Deployment upgrade strategies The built-in Deployment behavior The other strategies can be implemented fairly easily by talking to the API. Picture source: Kubernetes effect by Bilgin Ibryam
48. 48 Access your replicated Pods via a Service apiVersion: v1 kind: Service metadata: name: nginx namespace: default labels: app: nginx spec: type: ClusterIP ports: - name: http port: 80 targetPort: 80 selector: app: nginx Pod Selector • Provides a stable, immortal front-end IP address for a set of Pods. • Loadbalances traffic to any Pod matching the key-value label selector (here: app=nginx)
49. 49 Access your replicated Pods via a Service apiVersion: v1 kind: Service metadata: name: nginx namespace: default labels: app: nginx spec: type: ClusterIP ports: - name: http port: 80 targetPort: 80 selector: app: nginx Pod Selector • Accessible in the cluster through its IP address or via its internal DNS name (here: nginx.default.svc.cluster.local) • Can expose multiple ports
50. 50 Expose your Service to the world with an Ingress apiVersion: extensions/v1beta1 kind: Ingress metadata: name: nginx namespace: default labels: app: nginx spec: rules: - host: nginx.kubernetesfinland.com http: paths: - path: / backend: serviceName: nginx servicePort: 80 Service Reference • Expose your Service to the outer world with Ingress • Declare in a Kubernetes-native way how to proxy traffic to the Service
51. 51 Expose your Service to the world with an Ingress apiVersion: extensions/v1beta1 kind: Ingress metadata: name: nginx namespace: default labels: app: nginx spec: rules: - host: nginx.kubernetesfinland.com http: paths: - path: / backend: serviceName: nginx servicePort: 80 Service Reference • The Ingress Implementation is 3rd-party; e.g. Traefik or nginx • It’s recommended to run the implementation itself in Kubernetes
52. 52 Isolate your stuff in a Namespace Internet nginx.demo.kubernetesfinland.com Traefik as Ingress Controller Namespace: default nginx Ingress Rule nginx Service nginx Pod 1 nginx Pod 2 nginx Pod 3 nginx Deployment • A logical isolation method for resources; used for grouping and policy • Role Based Access Control (RBAC), and other Policies operate based on Namespaces
53. 53 What happens when you create a Deployment? Note that the API server (and the backing key-value DB) are involved in virtually any operation in the cluster. The “business logic” components (the controller-manager and scheduler) acts when something interesting happens. Finally, the Kubelet receives the message it should run a Pod; and delegates that task to a container runtime (like docker) via a daemon shim implementing the Container Runtime Interface (CRI).
54. Kubernetes’ Community Navigate the community with this map
55. 55 How do I kick the tires with Kubernetes? Play with Kubernetes right away in your browser! Create a single-node cluster on your workstation with minikube Create a production-ready cluster on any machines with kubeadm Test out Kubernetes easily locally with kind (Kubernetes IN Docker) Explore Kubernetes Certified Service Providers’ solutions
56. 56 Everything is done in Special Interest Groups Special Interest Groups (SIGs) manage Kubernetes’ various components and features. All code in the Kubernetes Github organization should be owned by one or more SIGs; with directory-level granularity. SIGs have regular (often weekly) video meetings where the attendees discuss design decisions, new features, bugs, testing, onboarding or whatever else that is relevant to the group. Attending these meetings is the best way to get to know the project Image source
57. 57 Where is the community doing work? 1. On Slack: https://kubernetes.slack.com & http://slack.k8s.io 2. On Github: https://github.com/kubernetes (in ~150 repositories) 3. In weekly Special Interest Group (SIG), Working Group (WG) and Subproject meetings Check out the community repo: https://github.com/kubernetes/community Youtube archive of ALL meetings available in the Kubernetes Youtube channel 4. In the Kubernetes Community Meeting every Thursday 6pm - 7pm UTC https://github.com/kubernetes/community/blob/master/events/community-meeting.md 5. On the Discuss Kubernetes site: https://discuss.kubernetes.io/ 6. In Google Groups in #kubernetes-*, e.g. https://groups.google.com/forum/#!forum/kubernetes-dev 7. At meetups worldwide (~170 meetups / ~40 countries): https://www.meetup.com/pro/cncf/ 8. In the community Office Hours and Meet Our Contributors - Ask Us Anything series
58. 58 How to start contributing? 1. Read as much of https://github.com/kubernetes/community as you can!! 2. Read the CONTRIBUTORS GUIDE 3. Find a SIG you’re interested to contribute to (e.g. SIG Cluster Lifecycle) 4. Create a Github account if you haven’t done so and sign up to the Kubernetes Slack: slack.k8s.io 5. Join the #sig-foo channels in Slack you’re interested in and #kubernetes-dev 6. Sign up to the #kubernetes-dev, #kubernetes-announce and #kubernetes-sig-foo mailing lists 7. Watch the New Contributor Series on Youtube from a New Contributor Summit 8. Check out the Codebase Tour in the “Meet Our Contributors” Youtube series 9. Learn how to use Kubernetes in more detail in this Free Kubernetes Course on edx.org 10. Join a weekly SIG meeting using Zoom and let them know you’d like to contribute :)
59. 59 Kubernetes An Introduction Kubernetes v1.12 11/2018 CC-BY 4.0 Next step: Check out this AWESOME 182-slide Kubernetes workshop presentation, made by Bob Killen and Jeffrey Sica from the University of Michigan.
60. 60 Various excellent blog posts Core Kubernetes: Jazz Improv over Orchestration - Joe Beda, 30th of May 2017 Kubernetes deep dive: API Server, Part 1 - Michael Hausenblas & Stefan Schimanski, 28th April 2017 Kubernetes deep dive: API Server, Part 2 - Michael & Stefan, 21st July 2017 Kubernetes deep dive: API Server, Part 3 - Michael & Stefan, 15th August 2017 Reasons Kubernetes is cool - Julia Evans, 5th October 2017 How Kubernetes certificate authorities work - Julia Evans, 5th August 2017 Operating a Kubernetes network - Julia Evans, 10th October 2017
61. 61 Join the community in Barcelona, Shanghai and San Diego!
62. Thank you! @luxas on Github @luxas on Kubernetes’ Slack @kubernetesonarm on Twitter lucas@luxaslabs.com