Upgrade to Pro — share decks privately, control downloads, hide ads and more …

K3sup 之極度快速 Set up K3s cluster

K3sup 之極度快速 Set up K3s cluster

- What is K3s
- K3s architecture & Setup a cluster way
- How K3s reduces the size of Kubernetes
- What is K3sup and usage
- K3s experience on ARM planform
- Summary

Avatar for Samina (Shan Jung Fu)

Samina (Shan Jung Fu)

October 18, 2019
Tweet

More Decks by Samina (Shan Jung Fu)

Other Decks in Technology

Transcript

  1. Samina Fu CNTUG co-organizer TGmeetup & CDNJS project member Interested

    in Cloud, Network & open source GitHub, Telegram: @sufuf3 Twitter: @sufuf3149 2
  2. Outline • What is K3s • K3s architecture & Setup

    a cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary 3
  3. • What is K3s • K3s architecture & Setup a

    cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary What is K3s 4
  4. 5

  5. • Lightweight Kubernetes • Certified Kubernetes distribution, Open source project

    • Designed for production workloads • Great for (5 less than k8s) – Edge, IoT, CI, ARM – Situations where a PhD in k8s clusterology is infeasible • Minimum System Requirements • Linux 3.10+ • 512 MB of ram per server • 75 MB of ram per node • 200 MB of disk space • x86_64, ARMv7, ARM64 6
  6. • What is K3s • K3s architecture & Setup a

    cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary K3s architecture 7
  7. • What is K3s • K3s architecture & Setup a

    cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary Setup a K3s cluster way github.com/sufuf3/k3s-lab 9
  8. K3s Server (Master) $ cat << 'EOF' | tee -a

    install-k3s-server.sh #!/bin/sh IPADDR="$(ip a show enp0s8 | grep "inet " | awk '{print $2}' | cut -d / -f1)" curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v0.9.1 INSTALL_K3S_EXEC="--docker --node-ip=${IPADDR} --flannel-iface=enp0s8 --write-kubeconfig-mode 644 --no-deploy=servicelb --no-deploy=traefik" sh - systemctl status k3s --no-pager echo "export K3S_MASTER_IP_ADDRESS=${IPADDR}" echo "export NODE_TOKEN=$(sudo cat /var/lib/rancher/k3s/server/node-token)" EOF $ sh install-k3s-server.sh Dependency: Docker 10
  9. $ export NODE_TOKEN= $ export K3S_MASTER_IP_ADDRESS= $ cat << 'EOF'

    | tee -a install-k3s-node.sh #!/bin/sh NODE_TOKEN=$1 K3S_MASTER_IP_ADDRESS=$2 echo "${K3S_MASTER_IP_ADDRESS} master" | sudo tee -a /etc/hosts IPADDR=$(ip a show enp0s8 | grep "inet " | awk '{print $2}' | cut -d / -f1) curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v0.9.1 INSTALL_K3S_EXEC="--docker --node-ip=${IPADDR} --flannel-iface=enp0s8" K3S_URL=https://${K3S_MASTER_IP_ADDRESS}:6443 K3S_TOKEN=${NODE_TOKEN} sh - systemctl status k3s-agent --no-pager EOF $ sh install-k3s-node.sh ${NODE_TOKEN} ${K3S_MASTER_IP_ADDRESS} K3s Agent (Node) Dependency: Docker 11
  10. Verification and Uninstall $ kubectl get no -o wide NAME

    STATUS ROLES AGE VERSION INTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node1 Ready worker 2m1s v1.15.4-k3s.1 192.168.0.201 Ubuntu 18.04.3 LTS 4.15.0-64-generic docker://18.6.3 master Ready master 8m35s v1.15.4-k3s.1 192.168.0.200 Ubuntu 18.04.3 LTS 4.15.0-64-generic docker://18.6.3 $ kubectl get componentstatus NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok $ kubectl run mynginx --image=nginx --replicas=1 --port=80 deployment.apps/mynginx created $ kubectl expose deployment mynginx --port 80 service/mynginx exposed $ kubectl get deploy,po,svc -l run=mynginx NAME READY UP-TO-DATE AVAILABLE AGE deployment.extensions/mynginx 1/1 1 1 10m NAME READY STATUS RESTARTS AGE pod/mynginx-568f57494d-wns86 1/1 Running 0 4m16s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mynginx ClusterIP 10.43.70.19 <none> 80/TCP 10m 12
  11. • What is K3s • K3s architecture & Setup a

    cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary How K3s reduces the size of Kubernetes 13
  12. How K3s reduces the size of Kubernetes 14 • Removing

    extra features • Eliminating external dependencies • Reducing the number of binaries required at runtime • Reducing the complexity of installation From: Accelerating Edge Computing with Arm and Rancher k3s Lightweight Kubernetes
  13. 15

  14. • Dropping alpha APIs (ref) $ k3s kubectl api-versions admissionregistration.k8s.io/v1beta1

    apiextensions.k8s.io/v1beta1 apiregistration.k8s.io/v1 apiregistration.k8s.io/v1beta1 apps/v1 apps/v1beta1 apps/v1beta2 authentication.k8s.io/v1 authentication.k8s.io/v1beta1 authorization.k8s.io/v1 authorization.k8s.io/v1beta1 autoscaling/v1 autoscaling/v2beta1 autoscaling/v2beta2 … Removing old and non-essential code 16
  15. • Removing all non-default admission controllers, in-tree cloud providers, and

    storage drivers Add by admin: https://kubernetes-csi.github.io/docs/drivers.html • Removing over 1 million lines of code Removing old and non-essential code (Cont.) 17
  16. Consolidating the packaging of running processes • For conserve memory:

    – Combined the processes that typically run on a Kubernetes management server into a single process – Combined the Kubelet, kube-proxy and flannel agent that run on a worker node into a single process 18
  17. Using containerd instead of Docker as the runtime container engine

    • Cut the runtime footprint significantly • Removing functionality – e.g. libnetwork, swarm, Docker storage drivers and other plugins 19
  18. Kubernetes $ sudo tree -lph /var/lib/etcd/member/ /var/lib/etcd/member/ ├── [drwx------ 4.0K]

    snap │ ├── [-rw-r--r-- 7.3K] 0000000000000003-0000000000256345.snap │ ├... │ ├── [-rw-r--r-- 7.3K] 0000000000000003-000000000025ff89.snap │ └── [-rw------- 16M] db └── [drwx------ 4.0K] wal ├── [-rw------- 61M] 000000000000001e-00000000002140ee.wal ├── [-rw------- 61M] 000000000000001f-00000000002258ea.wal ├... ├── [-rw------- 61M] 0000000000000022-0000000000259b03.wal └── [-rw------- 61M] 0.tmp $ sudo file /var/lib/etcd/member/wal/000000000000001e-00000000002140ee.wal /var/lib/etcd/member/wal/000000000000001e-00000000002140ee.wal: data Introducing SQLite as an optional datastore in addition to etcd 20 etcd is a Key-value storage
  19. K3s $ tree -lph /var/lib/rancher/k3s/server/db/ /var/lib/rancher/k3s/server/db/ ├── [-rw-r--r-- 440K] state.db

    ├── [-rw-r--r-- 32K] state.db-shm └── [-rw-r--r-- 4.0M] state.db-wal $ file /var/lib/rancher/k3s/server/db/state.db-wal /var/lib/rancher/k3s/server/db/state.db-wal: SQLite Write-Ahead Log, version 3007000 Introducing SQLite as an optional datastore in addition to etcd (Cont.) 21
  20. K3s $ journalctl -u k3s.service Oct 09 14:01:09 master k3s[1067]:

    time="2019-10-09T14:01:09.988580431Z" level=info msg="Fetching bootstrap data from etcd" Oct 09 14:01:10 master k3s[1067]: time="2019-10-09T14:01:10.050587488Z" level=info msg="Running kube-apiserver ... --etcd-servers=unix://kine.sock ... --storage-backend=etcd3 … $ ls /var/lib/rancher/k3s/server/ cred db kine.sock manifests node-token static tls Introducing SQLite as an optional datastore in addition to etcd (Cont.) 22
  21. K3s pkg/daemons/control/server.go#L291-L293 pkg/daemons/control/bootstrap.go#L20-L58 module:http://github.com/rancher/kine/pkg/client => http://github.com/ibuildthecloud/kine v0.1.0 schema = []string{

    `CREATE TABLE IF NOT EXISTS kine ( id INTEGER primary key autoincrement, name INTEGER, created INTEGER, deleted INTEGER, create_revision INTEGER, prev_revision INTEGER, lease INTEGER, value BLOB, old_value BLOB )`, `CREATE INDEX IF NOT EXISTS kine_name_index ON kine (name)`, `CREATE UNIQUE INDEX IF NOT EXISTS kine_name_prev_revision_uindex ON kine (name, prev_revision)`, } Introducing SQLite as an optional datastore in addition to etcd (Cont.) 23
  22. • Service load balancer (serviceLB) – exposes kubernetes services –

    Use traefik • k8s endpoint (Can’t be disabled) – Load balancer (tcpproxy) • Provide connectivity for HA scenarios – Reverse Tunnel connection • For master node's api-servers to communicate to agent node's kubelet & containerd Tunnel Proxy and ServiceLB 24
  23. • What is K3s • K3s architecture & Setup a

    cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary What is K3sup 25
  24. • Light-weight utility to get from zero to KUBECONFIG with

    k3s • Need: ssh access, k3sup binary, kubectl k3sup (said 'ketchup') 26
  25. • What is K3s • K3s architecture & Setup a

    cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary Setup a k3s cluster via k3sup 27 github.com/sufuf3/k3sup-lab
  26. 1. Prepare hosts – k3sup host: 1 node for install

    k3sup – k3s cluster: N nodes for setting a k3s cluster 2. Make sure k3sup host can use SSH public-key authentication to connect to N nodes 3. On k3sup host: Install k3sup and setup k3s cluster (Next Page) Setup k3s cluster steps via k3sup 28
  27. 1. Prepare hosts, 1 node for install k3sup, N nodes

    for setting a k3s cluster 2. Make sure k3sup host can use SSH public-key authentication to connect to N nodes 3. Install k3sup and setup k3s cluster 29 $ export SERVER_IP="192.168.0.200" $ export NODE1_IP="192.168.0.202" $ export NODE2_IP="192.168.0.203" $ export KUBECONFIG=`pwd`/kubeconfig $ curl -sLS https://get.k3sup.dev | sh $ sudo install k3sup /usr/local/bin/ $ curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.14.6/bin/linux/amd64/ kubectl $ chmod +x ./kubectl && sudo mv ./kubectl /usr/local/bin/kubectl $ k3sup install --ip ${SERVER_IP} --user vagrant --k3s-version v0.9.1 --k3s-extra-args '--flannel-iface enp0s8' $ k3sup join --ip ${NODE1_IP} --server-ip ${SERVER_IP} --user vagrant --k3s-version v0.9.1 --k3s-extra-args '--flannel-iface enp0s8' $ k3sup join --ip ${NODE2_IP} --server-ip ${SERVER_IP} --user vagrant --k3s-version v0.9.1 --k3s-extra-args '--flannel-iface enp0s8'
  28. • What is K3s • K3s architecture & Setup a

    cluster way • How K3s reduces the size of Kubernetes • What is K3sup and usage • K3s experience on ARM planform • Summary K3s experience on ARM planform 30
  29. Architecture • CPU: 4x Arm Cortex-A53 @ 1.5 GHz •

    Memory: 4 GB • Architecture: ARMv8 (AArch64) 31 Host: K3s Master VM: Node VM: Node
  30. Point for Attention • Checkout necessary modules are fine in

    Linux kernel version before compile – Cgroup issue in some Linux kernel version (If you use docker) – Modify the kernel config file & all needed modules are turn on • Don't use K3s v0.9.0 in ARM planform – https://github.com/rancher/k3s/issues/828 32
  31. Summary • K3s is a lightweight, easy to operate package

    which is ideal for accelerating and delivering solutions at the Edge – Kubernetes is operationally challenging for the edge case • K3s design & implementation overview – Removing extra features – Eliminating external dependencies – Reducing the number of binaries required at runtime – Reducing the complexity of installation • K3s cluster can setup via k3s script or k3sup • 33
  32. Special Event share: Hacktoberfest To qualify for the official limited

    edition Hacktoberfest shirt, you must register and make 4 pull requests (PRs) between October 1-31 (in any time zone). PRs can be made to any public repo on GitHub 34
  33. Reference • K3s docs site - https://rancher.com/docs/k3s/latest/en/ • K3s GitHub

    page - https://github.com/rancher/k3s • K3s landing page - https://k3s.io/ • https://rancher.com/press/2019-02-26-press-release-rancher-labs-intr oduces-lightweight-distribution-kubernetes-simplify/ • https://info.rancher.com/hubfs/eBooks,%20reports,%20and%20white papers/ARM%20White%20Paper,-V3%20(2).pdf?hsCtaTracking=347 54c8a-d543-4347-b1b5-38b4f4261192%7C6a6807a2-575e-4aa2-bd 70-73c3f3ff518a 36