K3sup 之極度快速 Set up K3s cluster

Transcript

1. K3sup 之極度快速 Set up K3s cluster Presenter: Samina Fu 2019/10/18,

   Taipei, SDN x Cloud Native Meetup #21

2. Samina Fu CNTUG co-organizer TGmeetup & CDNJS project member Interested

   in Cloud, Network & open source GitHub, Telegram: @sufuf3 Twitter: @sufuf3149 2

3. Outline • What is K3s • K3s architecture & Setup

   a cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary 3

4. • What is K3s • K3s architecture & Setup a

   cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary What is K3s 4

5. 5

6. • Lightweight Kubernetes • Certified Kubernetes distribution, Open source project

   • Designed for production workloads • Great for (5 less than k8s) – Edge, IoT, CI, ARM – Situations where a PhD in k8s clusterology is infeasible • Minimum System Requirements • Linux 3.10+ • 512 MB of ram per server • 75 MB of ram per node • 200 MB of disk space • x86_64, ARMv7, ARM64 6

7. • What is K3s • K3s architecture & Setup a

   cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary K3s architecture 7

8. K3s architecture 8

9. • What is K3s • K3s architecture & Setup a

   cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary Setup a K3s cluster way github.com/sufuf3/k3s-lab 9

10. K3s Server (Master) $ cat << 'EOF' | tee -a

    install-k3s-server.sh #!/bin/sh IPADDR="$(ip a show enp0s8 | grep "inet " | awk '{print $2}' | cut -d / -f1)" curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v0.9.1 INSTALL_K3S_EXEC="--docker --node-ip=${IPADDR} --flannel-iface=enp0s8 --write-kubeconfig-mode 644 --no-deploy=servicelb --no-deploy=traefik" sh - systemctl status k3s --no-pager echo "export K3S_MASTER_IP_ADDRESS=${IPADDR}" echo "export NODE_TOKEN=$(sudo cat /var/lib/rancher/k3s/server/node-token)" EOF $ sh install-k3s-server.sh Dependency: Docker 10

11. $ export NODE_TOKEN= $ export K3S_MASTER_IP_ADDRESS= $ cat << 'EOF'

    | tee -a install-k3s-node.sh #!/bin/sh NODE_TOKEN=$1 K3S_MASTER_IP_ADDRESS=$2 echo "${K3S_MASTER_IP_ADDRESS} master" | sudo tee -a /etc/hosts IPADDR=$(ip a show enp0s8 | grep "inet " | awk '{print $2}' | cut -d / -f1) curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v0.9.1 INSTALL_K3S_EXEC="--docker --node-ip=${IPADDR} --flannel-iface=enp0s8" K3S_URL=https://${K3S_MASTER_IP_ADDRESS}:6443 K3S_TOKEN=${NODE_TOKEN} sh - systemctl status k3s-agent --no-pager EOF $ sh install-k3s-node.sh ${NODE_TOKEN} ${K3S_MASTER_IP_ADDRESS} K3s Agent (Node) Dependency: Docker 11

12. Verification and Uninstall $ kubectl get no -o wide NAME

    STATUS ROLES AGE VERSION INTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node1 Ready worker 2m1s v1.15.4-k3s.1 192.168.0.201 Ubuntu 18.04.3 LTS 4.15.0-64-generic docker://18.6.3 master Ready master 8m35s v1.15.4-k3s.1 192.168.0.200 Ubuntu 18.04.3 LTS 4.15.0-64-generic docker://18.6.3 $ kubectl get componentstatus NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok $ kubectl run mynginx --image=nginx --replicas=1 --port=80 deployment.apps/mynginx created $ kubectl expose deployment mynginx --port 80 service/mynginx exposed $ kubectl get deploy,po,svc -l run=mynginx NAME READY UP-TO-DATE AVAILABLE AGE deployment.extensions/mynginx 1/1 1 1 10m NAME READY STATUS RESTARTS AGE pod/mynginx-568f57494d-wns86 1/1 Running 0 4m16s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mynginx ClusterIP 10.43.70.19 <none> 80/TCP 10m 12

13. • What is K3s • K3s architecture & Setup a

    cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary How K3s reduces the size of Kubernetes 13

14. How K3s reduces the size of Kubernetes 14 • Removing

    extra features • Eliminating external dependencies • Reducing the number of binaries required at runtime • Reducing the complexity of installation From: Accelerating Edge Computing with Arm and Rancher k3s Lightweight Kubernetes

15. 15

16. • Dropping alpha APIs (ref) $ k3s kubectl api-versions admissionregistration.k8s.io/v1beta1

    apiextensions.k8s.io/v1beta1 apiregistration.k8s.io/v1 apiregistration.k8s.io/v1beta1 apps/v1 apps/v1beta1 apps/v1beta2 authentication.k8s.io/v1 authentication.k8s.io/v1beta1 authorization.k8s.io/v1 authorization.k8s.io/v1beta1 autoscaling/v1 autoscaling/v2beta1 autoscaling/v2beta2 … Removing old and non-essential code 16

17. • Removing all non-default admission controllers, in-tree cloud providers, and

    storage drivers Add by admin: https://kubernetes-csi.github.io/docs/drivers.html • Removing over 1 million lines of code Removing old and non-essential code (Cont.) 17

18. Consolidating the packaging of running processes • For conserve memory:

    – Combined the processes that typically run on a Kubernetes management server into a single process – Combined the Kubelet, kube-proxy and flannel agent that run on a worker node into a single process 18

19. Using containerd instead of Docker as the runtime container engine

    • Cut the runtime footprint significantly • Removing functionality – e.g. libnetwork, swarm, Docker storage drivers and other plugins 19

20. Kubernetes $ sudo tree -lph /var/lib/etcd/member/ /var/lib/etcd/member/ ├── [drwx------ 4.0K]

    snap │ ├── [-rw-r--r-- 7.3K] 0000000000000003-0000000000256345.snap │ ├... │ ├── [-rw-r--r-- 7.3K] 0000000000000003-000000000025ff89.snap │ └── [-rw------- 16M] db └── [drwx------ 4.0K] wal ├── [-rw------- 61M] 000000000000001e-00000000002140ee.wal ├── [-rw------- 61M] 000000000000001f-00000000002258ea.wal ├... ├── [-rw------- 61M] 0000000000000022-0000000000259b03.wal └── [-rw------- 61M] 0.tmp $ sudo file /var/lib/etcd/member/wal/000000000000001e-00000000002140ee.wal /var/lib/etcd/member/wal/000000000000001e-00000000002140ee.wal: data Introducing SQLite as an optional datastore in addition to etcd 20 etcd is a Key-value storage

21. K3s $ tree -lph /var/lib/rancher/k3s/server/db/ /var/lib/rancher/k3s/server/db/ ├── [-rw-r--r-- 440K] state.db

    ├── [-rw-r--r-- 32K] state.db-shm └── [-rw-r--r-- 4.0M] state.db-wal $ file /var/lib/rancher/k3s/server/db/state.db-wal /var/lib/rancher/k3s/server/db/state.db-wal: SQLite Write-Ahead Log, version 3007000 Introducing SQLite as an optional datastore in addition to etcd (Cont.) 21

22. K3s $ journalctl -u k3s.service Oct 09 14:01:09 master k3s[1067]:

    time="2019-10-09T14:01:09.988580431Z" level=info msg="Fetching bootstrap data from etcd" Oct 09 14:01:10 master k3s[1067]: time="2019-10-09T14:01:10.050587488Z" level=info msg="Running kube-apiserver ... --etcd-servers=unix://kine.sock ... --storage-backend=etcd3 … $ ls /var/lib/rancher/k3s/server/ cred db kine.sock manifests node-token static tls Introducing SQLite as an optional datastore in addition to etcd (Cont.) 22

23. K3s pkg/daemons/control/server.go#L291-L293 pkg/daemons/control/bootstrap.go#L20-L58 module:http://github.com/rancher/kine/pkg/client => http://github.com/ibuildthecloud/kine v0.1.0 schema = []string{

    `CREATE TABLE IF NOT EXISTS kine ( id INTEGER primary key autoincrement, name INTEGER, created INTEGER, deleted INTEGER, create_revision INTEGER, prev_revision INTEGER, lease INTEGER, value BLOB, old_value BLOB )`, `CREATE INDEX IF NOT EXISTS kine_name_index ON kine (name)`, `CREATE UNIQUE INDEX IF NOT EXISTS kine_name_prev_revision_uindex ON kine (name, prev_revision)`, } Introducing SQLite as an optional datastore in addition to etcd (Cont.) 23

24. • Service load balancer (serviceLB) – exposes kubernetes services –

    Use traefik • k8s endpoint (Can’t be disabled) – Load balancer (tcpproxy) • Provide connectivity for HA scenarios – Reverse Tunnel connection • For master node's api-servers to communicate to agent node's kubelet & containerd Tunnel Proxy and ServiceLB 24

25. • What is K3s • K3s architecture & Setup a

    cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary What is K3sup 25

26. • Light-weight utility to get from zero to KUBECONFIG with

    k3s • Need: ssh access, k3sup binary, kubectl k3sup (said 'ketchup') 26

27. • What is K3s • K3s architecture & Setup a

    cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary Setup a k3s cluster via k3sup 27 github.com/sufuf3/k3sup-lab

28. 1. Prepare hosts – k3sup host: 1 node for install

    k3sup – k3s cluster: N nodes for setting a k3s cluster 2. Make sure k3sup host can use SSH public-key authentication to connect to N nodes 3. On k3sup host: Install k3sup and setup k3s cluster (Next Page) Setup k3s cluster steps via k3sup 28

29. 1. Prepare hosts, 1 node for install k3sup, N nodes

    for setting a k3s cluster 2. Make sure k3sup host can use SSH public-key authentication to connect to N nodes 3. Install k3sup and setup k3s cluster 29 $ export SERVER_IP="192.168.0.200" $ export NODE1_IP="192.168.0.202" $ export NODE2_IP="192.168.0.203" $ export KUBECONFIG=`pwd`/kubeconfig $ curl -sLS https://get.k3sup.dev | sh $ sudo install k3sup /usr/local/bin/ $ curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.14.6/bin/linux/amd64/ kubectl $ chmod +x ./kubectl && sudo mv ./kubectl /usr/local/bin/kubectl $ k3sup install --ip ${SERVER_IP} --user vagrant --k3s-version v0.9.1 --k3s-extra-args '--flannel-iface enp0s8' $ k3sup join --ip ${NODE1_IP} --server-ip ${SERVER_IP} --user vagrant --k3s-version v0.9.1 --k3s-extra-args '--flannel-iface enp0s8' $ k3sup join --ip ${NODE2_IP} --server-ip ${SERVER_IP} --user vagrant --k3s-version v0.9.1 --k3s-extra-args '--flannel-iface enp0s8'

30. • What is K3s • K3s architecture & Setup a

    cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary K3s experience on ARM planform 30

31. Architecture • CPU: 4x Arm Cortex-A53 @ 1.5 GHz •

    Memory: 4 GB • Architecture: ARMv8 (AArch64) 31 Host: K3s Master VM: Node VM: Node

32. Point for Attention • Checkout necessary modules are fine in

    Linux kernel version before compile – Cgroup issue in some Linux kernel version (If you use docker) – Modify the kernel config file & all needed modules are turn on • Don't use K3s v0.9.0 in ARM planform – https://github.com/rancher/k3s/issues/828 32

33. Summary • K3s is a lightweight, easy to operate package

    which is ideal for accelerating and delivering solutions at the Edge – Kubernetes is operationally challenging for the edge case • K3s design & implementation overview – Removing extra features – Eliminating external dependencies – Reducing the number of binaries required at runtime – Reducing the complexity of installation • K3s cluster can setup via k3s script or k3sup • 33

34. Special Event share: Hacktoberfest To qualify for the official limited

    edition Hacktoberfest shirt, you must register and make 4 pull requests (PRs) between October 1-31 (in any time zone). PRs can be made to any public repo on GitHub 34

35. for listening! Thank You 35

36. Reference • K3s docs site - https://rancher.com/docs/k3s/latest/en/ • K3s GitHub

    page - https://github.com/rancher/k3s • K3s landing page - https://k3s.io/ • https://rancher.com/press/2019-02-26-press-release-rancher-labs-intr oduces-lightweight-distribution-kubernetes-simplify/ • https://info.rancher.com/hubfs/eBooks,%20reports,%20and%20white papers/ARM%20White%20Paper,-V3%20(2).pdf?hsCtaTracking=347 54c8a-d543-4347-b1b5-38b4f4261192%7C6a6807a2-575e-4aa2-bd 70-73c3f3ff518a 36