rancher/system-toolsを試してみた

Transcript

1. rancher/system-tools Λࢼͯ͠Έͨ Masaki-Nakayama

2. rancher/system-toolsͱ͸ʁ IUUQTHJUIVCDPNSBODIFSSBODIFSJTTVFT

3. rancher/system-toolsͱ͸ʁ IUUQTHJUIVCDPNSBODIFSTZTUFNUPPMT

4. rancher/system-toolsͱ͸ʁ

5. rancher/system-toolsͱ͸ʁ remove Ϋϥελ͔Βrancherؔ࿈ϦιʔεΛ࡟আ logs Ϋϥελ͔ΒϩάΛऩूͯ͠อଘ stats Ϋϥελ͔Β౷ܭ৘ใΛϦΞϧλΠϜͰऩू

6. લ४උ 1. DockerΠϯετʔϧࡁͷLinuxαʔόʔΛ༻ҙ 2. k8sͷconfigΛίϐʔʢkubeconfig.ymlͳͲͰอଘʣ 3. git͔Βsystem-toolsͷιʔεΛclone 4. makeίϚϯυͰϏϧυͯ͠system-toolsͷόΠφϦΛ࡞੒ ※releases͔ΒόΠφϦΛμ΢ϯϩʔυͯ͠΋ྑ͍

7. ؀ڥ αʔόʔɿ rancherαʔόʔ x1ʢrancher v2.1.0ʣ Ϋϥελʔϊʔυ x1ʢetcd, control plane, workerʣ

   ※ࠓճ͸KAGOYA CLOUD /2ͷVMΠϯελϯεΛ࢖༻ OS: Ubuntu 16.04 ※Կ͔͋Ε͹ιʔεΛ͍ͬͯ͡όΠφϦԽ͢ΔͨΊ ※makeͰͷόΠφϦԽ͸macͰ͸ಈ͖·ͤΜͰͨ͠ Docker: 18.06.1-ce

8. ͜Μͳײ͡Ͱ࡞ͬͯ͋Γ·͢

9. system-tools remove # system-tools remove -c kubeconfig.yml Are you sure

   you want to remove Rancher Management Plane in Namespace [cattle-system] [y/n]: y INFO[0002] Removing Rancher management plane in namespace: [cattle-system] INFO[0002] Getting conenction configuration INFO[0002] Removing Cattle deployment INFO[0002] Removed Cattle deployment succuessfully INFO[0002] Removing ClusterRoleBindings INFO[0002] deleting cluster role binding [cattle-admin-binding].. INFO[0002] deleting cluster role binding [clusterrolebinding-5c7rb].. INFO[0002] deleting cluster role binding [clusterrolebinding-5fwvp].. INFO[0004] Removing ClusterRoleBindings INFO[0005] deleting cluster role binding [clusterrolebinding-5fwvp].. INFO[0005] deleting cluster role binding [clusterrolebinding-cbz99].. INFO[0005] deleting cluster role binding [clusterrolebinding-sj6ps].. INFO[0005] deleting cluster role binding [clusterrolebinding-v8zwl].. INFO[0005] deleting cluster role binding [clusterrolebinding-vtflr].. INFO[0005] deleting cluster role binding [clusterrolebinding-zvdh6].. INFO[0005] deleting cluster role binding [globaladmin-u-p9d29].. INFO[0005] deleting cluster role binding [globaladmin-u-zz9xm].. INFO[0005] deleting cluster role binding [globaladmin-user-mbvd4].. INFO[0005] Successfully removed ClusterRoleBindings INFO[0005] Removing ClusterRoles INFO[0007] Removing ClusterRoles INFO[0009] Removing ClusterRoles FATA[0011] clusterroles.rbac.authorization.k8s.io is forbidden: User "user-mbvd4" cannot list clusterroles.rbac.authorization.k8s.io at the cluster scope

10. system-tools remove ׬ྃͰ͖·ͤΜͰͨ͠ɾɾɾ ʢremove.goͷgetCattleClusterRoleBindingsList Ͱࣦഊ͍ͯ͠ΔͬΆ͍ɾɾɾʣ ଞʹ΋ࢼͨ͠ํ͋Ε͹ڭ͍͑ͯͩ͘͞ɾɾ

11. system-tools remove ຊདྷͰ͋Ε͹ɺk8sϊʔυ͔Βrancherؔ࿈ʢ໊લۭؒ: cattle-systemͳͲʣͷϦιʔεΛશͯ࡟আͯ͘͠ΕΔ • ClusterRoles • ClusterRoleBindings • Namespaces

    • Projects • Users • Clusters • CRD

12. system-tools remove rancherΛར༻͍͕ͯͨ͠ࢭΊΔࡍɺ७ਮͳ k8sΫϥελʔͱͯͦ͠ͷ··ӡ༻Λଓ͚ͨ ͍৔߹ʹҠߦΛߟ͑ͳͯ͘΋ྑ͍ rancherΛ७ਮʹk8sΠϯετʔϥʔͱͯ͠࢖ ͑ͦ͏

13. system-tools logs # system-tools logs -c kubeconfig.yml INFO[0000] deploying log

    collection DaemonSet [log-collector].. INFO[0000] starting log collection.. INFO[0000] fetching logs from node [v133-18-205-207].. INFO[0002] fetching logs from node [v133-18-175-238].. INFO[0002] removing log collection DaemonSet [log-collector].. INFO[0002] log collection DaemonSet [log-collector] removed successfully.. # ls cluster-logs.tar system-tools # tar xvf cluster-logs.tar && ls cluster-logs.tar system-tools node1 node1# ls -l total 13720 -rw-r----- 1 root root 1537598 Nov 16 01:39 etcd.log -rw-r----- 1 root root 9966735 Nov 16 01:39 kube-apiserver.log -rw-r----- 1 root root 740657 Nov 16 01:39 kube-controller-manager.log -rw-r----- 1 root root 1746861 Nov 16 01:39 kubelet.log -rw-r----- 1 root root 21867 Nov 16 01:39 kube-proxy.log -rw-r----- 1 root root 23845 Nov 16 01:39 kube-scheduler.log

14. system-tools logs log-collectorͱ͍͏DaemonSetΛσϓϩΠͯ͠ɺϩάऩूͱग़ ྗ͕ऴΘΕ͹࡟আ͍ͯ͠ΔͷͰطଘ؀ڥΛԚ͞ͳ͍ ϩάͷத਎Λݟͨײ͡Ͱ͸ɺ/var/lib/docker/containers/{ίϯς φID}/{ίϯςφID}-json.logΛҾͬ͜ൈ͍͍ͯΔͬΆ͍ ૉͷdockerͷϩάͩͱԿͷϩά͔ͱͯ΋Θ͔Γʹ͍͘ͷͰͦΕ ͚ͩͰ΋ॿ͔Δ ֎෦αʔόʔ͔ΒطଘΫϥελʔʹ΄΅ӨڹΛ༩͑ͣϊʔυ͝ͱ ͷlogΛऩू͢Δͱ͍ͬͨӡ༻΋૊Ίͦ͏

15. system-tools logs ͨͩɺԼهͷΑ͏ʹࣦഊ͢Δ͜ͱ΋͋Δ FATA[0000] error executing command on pod [cattle-system/log-

    collector-q6rtt]: unable to upgrade connection: container not found ("log-collector") ΋͔ͨ͠͠ΒlogͷfetchΛ଴ͨͣͯ͠ DaemonSetΛ࡟আ͍ͯ͠Δͷ͔΋͠Εͳ͍…

16. system-tools stats # system-tools stats -c kubeconfig.yml ɾɾɾɾ INFO[0015] node

    stats for [v133-18-205-207].. Linux 4.4.0-34-generic (stats-collector-mfgtn) 11/15/18 _x86_64_ (2 CPU) 14:32:31 CPU %user %nice %system %iowait %steal %idle 14:32:32 all 4.10 0.00 6.67 12.31 0.00 76.92 14:32:31 kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 14:32:32 157740 233888 1890504 92.30 6136 239116 4688380 228.90 1621768 119652 808 14:32:31 MBfsfree MBfsused %fsused %ufsused Ifree Iused %Iused FILESYSTEM 14:32:32 22494 7615 25.29 30.11 1713379 252701 12.85 /dev/vda1 14:32:32 22494 7615 25.29 30.11 1713379 252701 12.85 /dev/vda1 14:32:32 22494 7615 25.29 30.11 1713379 252701 12.85 /dev/vda1 14:32:32 22494 7615 25.29 30.11 1713379 252701 12.85 /dev/vda1 Average: CPU %user %nice %system %iowait %steal %idle Average: all 4.10 0.00 6.67 12.31 0.00 76.92 Average: kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty Average: 157740 233888 1890504 92.30 6136 239116 4688380 228.90 1621768 119652 808 Summary: MBfsfree MBfsused %fsused %ufsused Ifree Iused %Iused FILESYSTEM Summary: 22494 7615 25.29 30.11 1713379 252701 12.85 /dev/vda1 Summary: 22494 7615 25.29 30.11 1713379 252701 12.85 /dev/vda1 Summary: 22494 7615 25.29 30.11 1713379 252701 12.85 /dev/vda1 Summary: 22494 7615 25.29 30.11 1713379 252701 12.85 /dev/vda1 ɾɾɾɾ

17. system-tools stats stats-collectorͱ͍͏DaemonSetΛσϓϩΠͯ͠statsΛग़ ྗ͠ଓ͚Δ →Ϣʔβʔૢ࡞ʹΑΔதஅͰDaemonSet࡟আ ֎෦͔Βk8sΫϥελͷ౷ܭ৘ใͷऩू͕Ͱ͖Δ ग़ྗΛՃ޻͢Ε͹rancherͷϞχλʔ͕࡞Εͦ͏ͳՄೳੑΛײ͡ Δ

18. ·ͱΊ ΋͔ͨ͠͠ΒRancherͷόʔδϣϯʹΑͬͯ͸ಈ͔͘ ΋͠Εͳ͍ʢ্ख͍ͬͨ͘Βͦͷ͏ͪQiitaʹͰ΋ॻ͖ ·͢ʣ ҆ఆ͢Ε͹rancherӡ༻໘ͰͷϝϦοτ͕ظ଴Ͱ͖ͦ͏ ίϚϯυ͕૿͍͖͑ͯͦ͏ͳظ଴