K3sup 之極度快速 Set up K3s cluster Presenter: Samina Fu 2019/10/18, Taipei, SDN x Cloud Native Meetup #21

Samina Fu CNTUG co-organizer TGmeetup & CDNJS project member Interested in Cloud, Network & open source GitHub, Telegram: @sufuf3 Twitter: @sufuf3149 2

Outline • What is K3s • K3s architecture & Setup a cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary 3

• What is K3s • K3s architecture & Setup a cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary What is K3s 4

5

• Lightweight Kubernetes • Certified Kubernetes distribution, Open source project • Designed for production workloads • Great for (5 less than k8s) – Edge, IoT, CI, ARM – Situations where a PhD in k8s clusterology is infeasible • Minimum System Requirements • Linux 3.10+ • 512 MB of ram per server • 75 MB of ram per node • 200 MB of disk space • x86_64, ARMv7, ARM64 6

• What is K3s • K3s architecture & Setup a cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary K3s architecture 7

K3s architecture 8

• What is K3s • K3s architecture & Setup a cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary Setup a K3s cluster way github.com/sufuf3/k3s-lab 9

K3s Server (Master) $ cat << 'EOF' | tee -a install-k3s-server.sh #!/bin/sh IPADDR="$(ip a show enp0s8 | grep "inet " | awk '{print $2}' | cut -d / -f1)" curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v0.9.1 INSTALL_K3S_EXEC="--docker --node-ip=${IPADDR} --flannel-iface=enp0s8 --write-kubeconfig-mode 644 --no-deploy=servicelb --no-deploy=traefik" sh - systemctl status k3s --no-pager echo "export K3S_MASTER_IP_ADDRESS=${IPADDR}" echo "export NODE_TOKEN=$(sudo cat /var/lib/rancher/k3s/server/node-token)" EOF $ sh install-k3s-server.sh Dependency: Docker 10

$ export NODE_TOKEN= $ export K3S_MASTER_IP_ADDRESS= $ cat << 'EOF' | tee -a install-k3s-node.sh #!/bin/sh NODE_TOKEN=$1 K3S_MASTER_IP_ADDRESS=$2 echo "${K3S_MASTER_IP_ADDRESS} master" | sudo tee -a /etc/hosts IPADDR=$(ip a show enp0s8 | grep "inet " | awk '{print $2}' | cut -d / -f1) curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v0.9.1 INSTALL_K3S_EXEC="--docker --node-ip=${IPADDR} --flannel-iface=enp0s8" K3S_URL=https://${K3S_MASTER_IP_ADDRESS}:6443 K3S_TOKEN=${NODE_TOKEN} sh - systemctl status k3s-agent --no-pager EOF $ sh install-k3s-node.sh ${NODE_TOKEN} ${K3S_MASTER_IP_ADDRESS} K3s Agent (Node) Dependency: Docker 11

Verification and Uninstall $ kubectl get no -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node1 Ready worker 2m1s v1.15.4-k3s.1 192.168.0.201 Ubuntu 18.04.3 LTS 4.15.0-64-generic docker://18.6.3 master Ready master 8m35s v1.15.4-k3s.1 192.168.0.200 Ubuntu 18.04.3 LTS 4.15.0-64-generic docker://18.6.3 $ kubectl get componentstatus NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok $ kubectl run mynginx --image=nginx --replicas=1 --port=80 deployment.apps/mynginx created $ kubectl expose deployment mynginx --port 80 service/mynginx exposed $ kubectl get deploy,po,svc -l run=mynginx NAME READY UP-TO-DATE AVAILABLE AGE deployment.extensions/mynginx 1/1 1 1 10m NAME READY STATUS RESTARTS AGE pod/mynginx-568f57494d-wns86 1/1 Running 0 4m16s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mynginx ClusterIP 10.43.70.19 80/TCP 10m 12

• What is K3s • K3s architecture & Setup a cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary How K3s reduces the size of Kubernetes 13

How K3s reduces the size of Kubernetes 14 • Removing extra features • Eliminating external dependencies • Reducing the number of binaries required at runtime • Reducing the complexity of installation From: Accelerating Edge Computing with Arm and Rancher k3s Lightweight Kubernetes

15

• Dropping alpha APIs (ref) $ k3s kubectl api-versions admissionregistration.k8s.io/v1beta1 apiextensions.k8s.io/v1beta1 apiregistration.k8s.io/v1 apiregistration.k8s.io/v1beta1 apps/v1 apps/v1beta1 apps/v1beta2 authentication.k8s.io/v1 authentication.k8s.io/v1beta1 authorization.k8s.io/v1 authorization.k8s.io/v1beta1 autoscaling/v1 autoscaling/v2beta1 autoscaling/v2beta2 … Removing old and non-essential code 16

• Removing all non-default admission controllers, in-tree cloud providers, and storage drivers Add by admin: https://kubernetes-csi.github.io/docs/drivers.html • Removing over 1 million lines of code Removing old and non-essential code (Cont.) 17

Consolidating the packaging of running processes • For conserve memory: – Combined the processes that typically run on a Kubernetes management server into a single process – Combined the Kubelet, kube-proxy and flannel agent that run on a worker node into a single process 18

Using containerd instead of Docker as the runtime container engine • Cut the runtime footprint significantly • Removing functionality – e.g. libnetwork, swarm, Docker storage drivers and other plugins 19

Kubernetes $ sudo tree -lph /var/lib/etcd/member/ /var/lib/etcd/member/ ├── [drwx------ 4.0K] snap │ ├── [-rw-r--r-- 7.3K] 0000000000000003-0000000000256345.snap │ ├... │ ├── [-rw-r--r-- 7.3K] 0000000000000003-000000000025ff89.snap │ └── [-rw------- 16M] db └── [drwx------ 4.0K] wal ├── [-rw------- 61M] 000000000000001e-00000000002140ee.wal ├── [-rw------- 61M] 000000000000001f-00000000002258ea.wal ├... ├── [-rw------- 61M] 0000000000000022-0000000000259b03.wal └── [-rw------- 61M] 0.tmp $ sudo file /var/lib/etcd/member/wal/000000000000001e-00000000002140ee.wal /var/lib/etcd/member/wal/000000000000001e-00000000002140ee.wal: data Introducing SQLite as an optional datastore in addition to etcd 20 etcd is a Key-value storage

K3s $ tree -lph /var/lib/rancher/k3s/server/db/ /var/lib/rancher/k3s/server/db/ ├── [-rw-r--r-- 440K] state.db ├── [-rw-r--r-- 32K] state.db-shm └── [-rw-r--r-- 4.0M] state.db-wal $ file /var/lib/rancher/k3s/server/db/state.db-wal /var/lib/rancher/k3s/server/db/state.db-wal: SQLite Write-Ahead Log, version 3007000 Introducing SQLite as an optional datastore in addition to etcd (Cont.) 21

K3s $ journalctl -u k3s.service Oct 09 14:01:09 master k3s[1067]: time="2019-10-09T14:01:09.988580431Z" level=info msg="Fetching bootstrap data from etcd" Oct 09 14:01:10 master k3s[1067]: time="2019-10-09T14:01:10.050587488Z" level=info msg="Running kube-apiserver ... --etcd-servers=unix://kine.sock ... --storage-backend=etcd3 … $ ls /var/lib/rancher/k3s/server/ cred db kine.sock manifests node-token static tls Introducing SQLite as an optional datastore in addition to etcd (Cont.) 22

K3s pkg/daemons/control/server.go#L291-L293 pkg/daemons/control/bootstrap.go#L20-L58 module:http://github.com/rancher/kine/pkg/client => http://github.com/ibuildthecloud/kine v0.1.0 schema = []string{ `CREATE TABLE IF NOT EXISTS kine ( id INTEGER primary key autoincrement, name INTEGER, created INTEGER, deleted INTEGER, create_revision INTEGER, prev_revision INTEGER, lease INTEGER, value BLOB, old_value BLOB )`, `CREATE INDEX IF NOT EXISTS kine_name_index ON kine (name)`, `CREATE UNIQUE INDEX IF NOT EXISTS kine_name_prev_revision_uindex ON kine (name, prev_revision)`, } Introducing SQLite as an optional datastore in addition to etcd (Cont.) 23

• Service load balancer (serviceLB) – exposes kubernetes services – Use traefik • k8s endpoint (Can’t be disabled) – Load balancer (tcpproxy) • Provide connectivity for HA scenarios – Reverse Tunnel connection • For master node's api-servers to communicate to agent node's kubelet & containerd Tunnel Proxy and ServiceLB 24

• What is K3s • K3s architecture & Setup a cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary What is K3sup 25

• Light-weight utility to get from zero to KUBECONFIG with k3s • Need: ssh access, k3sup binary, kubectl k3sup (said 'ketchup') 26

• What is K3s • K3s architecture & Setup a cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary Setup a k3s cluster via k3sup 27 github.com/sufuf3/k3sup-lab

1. Prepare hosts – k3sup host: 1 node for install k3sup – k3s cluster: N nodes for setting a k3s cluster 2. Make sure k3sup host can use SSH public-key authentication to connect to N nodes 3. On k3sup host: Install k3sup and setup k3s cluster (Next Page) Setup k3s cluster steps via k3sup 28

1. Prepare hosts, 1 node for install k3sup, N nodes for setting a k3s cluster 2. Make sure k3sup host can use SSH public-key authentication to connect to N nodes 3. Install k3sup and setup k3s cluster 29 $ export SERVER_IP="192.168.0.200" $ export NODE1_IP="192.168.0.202" $ export NODE2_IP="192.168.0.203" $ export KUBECONFIG=`pwd`/kubeconfig $ curl -sLS https://get.k3sup.dev | sh $ sudo install k3sup /usr/local/bin/ $ curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.14.6/bin/linux/amd64/ kubectl $ chmod +x ./kubectl && sudo mv ./kubectl /usr/local/bin/kubectl $ k3sup install --ip ${SERVER_IP} --user vagrant --k3s-version v0.9.1 --k3s-extra-args '--flannel-iface enp0s8' $ k3sup join --ip ${NODE1_IP} --server-ip ${SERVER_IP} --user vagrant --k3s-version v0.9.1 --k3s-extra-args '--flannel-iface enp0s8' $ k3sup join --ip ${NODE2_IP} --server-ip ${SERVER_IP} --user vagrant --k3s-version v0.9.1 --k3s-extra-args '--flannel-iface enp0s8'

• What is K3s • K3s architecture & Setup a cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary K3s experience on ARM planform 30

Architecture • CPU: 4x Arm Cortex-A53 @ 1.5 GHz • Memory: 4 GB • Architecture: ARMv8 (AArch64) 31 Host: K3s Master VM: Node VM: Node

Point for Attention • Checkout necessary modules are fine in Linux kernel version before compile – Cgroup issue in some Linux kernel version (If you use docker) – Modify the kernel config file & all needed modules are turn on • Don't use K3s v0.9.0 in ARM planform – https://github.com/rancher/k3s/issues/828 32

Summary • K3s is a lightweight, easy to operate package which is ideal for accelerating and delivering solutions at the Edge – Kubernetes is operationally challenging for the edge case • K3s design & implementation overview – Removing extra features – Eliminating external dependencies – Reducing the number of binaries required at runtime – Reducing the complexity of installation • K3s cluster can setup via k3s script or k3sup • 33

Special Event share: Hacktoberfest To qualify for the official limited edition Hacktoberfest shirt, you must register and make 4 pull requests (PRs) between October 1-31 (in any time zone). PRs can be made to any public repo on GitHub 34

for listening! Thank You 35

Reference • K3s docs site - https://rancher.com/docs/k3s/latest/en/ • K3s GitHub page - https://github.com/rancher/k3s • K3s landing page - https://k3s.io/ • https://rancher.com/press/2019-02-26-press-release-rancher-labs-intr oduces-lightweight-distribution-kubernetes-simplify/ • https://info.rancher.com/hubfs/eBooks,%20reports,%20and%20white papers/ARM%20White%20Paper,-V3%20(2).pdf?hsCtaTracking=347 54c8a-d543-4347-b1b5-38b4f4261192%7C6a6807a2-575e-4aa2-bd 70-73c3f3ff518a 36