#Kubernetes #Provisioning
카카오에서는 쿠버네티스를 서비스하기 위해 DKOS를 사용하고 있습니다.
본 세션에서는 DKOS가 쿠버네티스를 서비스 하기 위해, 고려하였던 프로비저닝 툴들과 선택을 시작으로, 시행착오를 거쳐, 프로비저닝 툴을 걷어내기까지의 여정을 공유 드리고자 합니다.
감사합니다
발표자 : rookie.jeon 카카오에서 클라우드 플랫폼을 개발하고 있는 루키입니다.
୍ rookie.jeonয়Copyright 2022. Kakao Corp. All rights reserved. Redistribution or public display is not permitted without written permission from Kakao.Kubernetes provisioningif(kakao)2022۽࠺ ోҗ ݅թࠗఠ যө . . .
View Slide
য়ীࢲ ߡ֎౭झܳ ࢲ࠺झ ೞח ߑधߡ֎౭झ ۽࠺ ో ࢶఖೞӝ ו՛ ۽࠺ ో(Kubespray) ইए۽࠺ ో ب হ ߡ֎౭झ ࢲ࠺झ ೞӝܻ ߂ ݃ޖܻ
DKOSۆ ޖੋо?DKOSDatacenter of Kakao Operating System- Multi- Managed
DKOSۆ ޖੋо?CNILogLB VolumeDNS IngressManaged podKubernetes clusterUserDKOS
DKOSীࢲ যڃ সਸ ࣻ೯೧ঠ ೡө?1. VM ࢤࢿ (OpenStack ӝ߈ ࢎղ IaaS ۽ંੋ krane ࢎਊ)2. kernel / OS ۄఠ ࢸ3. ஶపց ۠ఋ ҳ୷4. ߡ֎౭झ ҳ୷5. managed pod ߓನ
Provisioning tool۽࠺ ో ਃࢿ
Kubeadm“рױ”kubeadmۆ ޖੋо?
kubeadmۆ ޖੋо?DKOSKubeadmউࢿрױೣױગ ਗ ߧਤ(য় ߡ֎౭झ ࢸ݅ ਗ)ѐ߹ স ࣻ೯NodeA - ZA - ZA - Z
Kubespray“և ߧਤ زച + নೠ ࢶఖ ઁҕ”Kubesprayۆ ޖੋо?
Kubesprayۆ ޖੋо?Kubesprayև ߧਤ زച ਗনೠ ࢶఖ ઁҕױੌ ࣻ೯ਸ ాೠ ࣻ ֢٘ ઁয оמউࢿDKOSNodeaccess info+ ansible script
Kubespray ఖ!!Kubespray ఖ
ݥ!!“Cluster APIח Ҋ۰ೞ ঋওաਃ?”ܲ ో ࠺Ү উೞաਃ?
Kubespray ఖKubespray ఖ
Kubespray ఖ??Kubespray ఖ
ࠁউ ઓࢿࢸ दр ਬ ࠁࣻKubespray ইए
ࠁউSSHKubespray ইए - ࠁউ
Kubespray ইए - ઓࢿ20.04 20.04 20.0418.04 18.04 18.0416.04 16.04 16.04[email protected]AnsiblePythonPkg
Kubespray ইए - ֢٘ ୶о दрস ࢸݺ ࣻ೯ दр ࢎ স оמ ৈࠗDownload binary ‘container runtime, k8s, CNI . . .’ प೯ ੌ 300s Image ղ оמSet container runtime & k8s container runtime & K8s ࢸ ߂ ҳز 90s ੌࠗ ܻ оמGenerate etc cert etcd cert ࢤࢿ 30s DKOS ܻ оמGather facts anisible – inventory ֢٘ ࠁ ࣻ 20s XSet OS proxy, NSS, hostname . . . ࢸ 10s Image ղ оמ450s – (Image ղ + DKOS ܻ) = 150s୨ ࢸ दр: 450s
ਬ ࠁࣻ300 ~ 400 Ansible TASKKubespray ইए - ਬ ࠁࣻ
Ҋ ݆ҳ݅…ࢶఖ ӝ۽
ܲ ో ѐߊCluster API A - Zࢶఖ ӝ۽
Cluster API“ࢿמ गী ೠ ޙ”cluster-API Ѩష
۽࠺ ో ઁѢProvisioning tool
DKOSীࢲ যڃ সਸ ࣻ೯೧ঠ ೡө?1. VM ࢤࢿ (OpenStack ӝ߈ IaaS ۽ંੋ krane ࢎਊ)2. kernel / OS ۄఠ ࢸ3. ஶపց ۠ఋ ҳ୷4. ߡ֎౭झ ҳ୷5. managed pod ߓನ
DKOSীࢲ যڃ সਸ ࣻ೯೧ঠ ೡө?1. VM ࢤࢿ (OpenStack ӝ߈ IaaS ۽ંੋ krane ࢎਊ)2. kernel / OS ۄఠ ࢸ3. ஶపց ۠ఋ ҳ୷4. ߡ֎౭झ ҳ୷5. managed pod ߓನControlplane1 - 5 ױ҅ ೯֢٘ ୶о Dataplane1 - 4 ױ҅ ೯֢٘ ୶о ഝߊࢸ दрী х
DKOSীࢲ যڃ সਸ ࣻ೯೧ঠ ೡө?1. VM ࢤࢿ (OpenStack ӝ߈ IaaS ۽ંੋ krane ࢎਊ)2. kernel / OS ۄఠ ࢸ3. ஶపց ۠ఋ ҳ୷4. ߡ֎౭झ ҳ୷5. managed pod ߓನ ؘఠ+ز ؘఠ
DKOSীࢲ যڃ সਸ ࣻ೯೧ঠ ೡө?1. VM ࢤࢿ (OpenStack ӝ߈ IaaS ۽ંੋ krane ࢎਊ)2. kernel / OS ۄఠ ࢸ3. ஶపց ۠ఋ ҳ୷4. ߡ֎౭झ ҳ୷5. managed pod ߓನ ؘఠ(Image)+ز ؘఠ(Cloud-init)
DKOSImage RepoCloud-initData centerKraneؘఠ ۨੋ ۽࠺ - द
DKOSImage RepoCloud-initData centerKranePre-processingimageؘఠ ۨੋ ۽࠺ - द
DKOSImage RepoCloud-initData centerKraneDataPlaneCreate VMWith scriptؘఠ ۨੋ ۽࠺ - द
DKOSImage RepoCloud-initData centerKraneDataPlaneCreate VMWith scriptUpload scriptؘఠ ۨੋ ۽࠺ - द
DKOSImage RepoCloud-initData centerKraneDataPlaneCreate VMWith script Create VMؘఠ ۨੋ ۽࠺ - दPre-processingimage
DKOSImage RepoCloud-initData centerKrane1. Request cloud-init script2. Get Cert & Config from scriptDataPlaneؘఠ ۨੋ ۽࠺ - द
DKOSImage RepoCloud-initData centerKrane ConfigCert1. Request cloud-init script2. Get Cert & Config from scriptDataPlaneؘఠ ۨੋ ۽࠺ - द
DKOSImage RepoCloud-initData centerKrane ConfigCert ControlPlaneJoinDataPlaneؘఠ ۨੋ ۽࠺ - द
DKOSImage RepoCloud-initData centerKrane ConfigCert ControlPlaneJoinDataPlaneؘఠ ۨੋ ۽࠺ - दDone!!
DKOSImage RepoCloud-initData centerKrane2. Create VMWith script 3. Create VM3. Upload scriptConfigCert6. JoinControlPlane4. Request cloud-init script5. Get Cert & Config from script1. Pre-processingimageDataPlaneؘఠ ۨੋ ۽࠺ - द
যڌѱ?“ઑੋ җ”ؘఠ ۨੋ ஶ܀ ۨੋ ઑੋ җ
DataPlane ControlPlanekube-apiserverkube-schedulercontroller-manageretcdKubeletKube-Proxyؘఠ ۨੋ ஶ܀ ۨੋ ઑੋ җ - द
DataPlane ControlPlaneKubeletKube-Proxyؘఠ ۨੋ ஶ܀ ۨੋ ઑੋ җ - दkube-apiserverkube-schedulercontroller-manageretcd
DataPlane ControlPlaneKubeletKube-Proxyؘఠ ۨੋ ஶ܀ ۨੋ ઑੋ җ - दCertkube-apiserverkube-schedulercontroller-manageretcd
DataPlane ControlPlaneKubeletKube-Proxyؘఠ ۨੋ ஶ܀ ۨੋ ઑੋ җ - दToken1. Create Tokenkube-apiserverkube-schedulercontroller-manageretcd
DataPlane ControlPlaneKubeletKube-Proxyؘఠ ۨੋ ஶ܀ ۨੋ ઑੋ җ - दTokenTokenConfig1. Create Token2. Create configkube-apiserverkube-schedulercontroller-manageretcd
DataPlane ControlPlaneKubeletKube-Proxyؘఠ ۨੋ ஶ܀ ۨੋ ઑੋ җ - दTokenTokenConfigrequestcert kube-apiserverkube-schedulercontroller-manageretcd
DataPlane ControlPlaneKubeletKube-Proxyؘఠ ۨੋ ஶ܀ ۨੋ ઑੋ җ - दTokensendcertCertkube-apiserverkube-schedulercontroller-manageretcd
DataPlane ControlPlaneKubeletKube-Proxyؘఠ ۨੋ ஶ܀ ۨੋ ઑੋ җ - दTokenCertjoinkube-apiserverkube-schedulercontroller-manageretcd
DKOSImage RepoCloud-initData centerKrane2. Create VMWith script 3. Create VM3. Upload scriptConfigCert6. JoinControlPlane4. Request cloud-init script5. Get Cert & Config from script1. Pre-processingimageDataPlaneۄ٘ ী ׳غח ੌ
DKOSImage RepoCloud-initData centerKrane2. Create VMWith script 3. Create VM3. Upload scriptConfigCert6. JoinControlPlane4. Request cloud-init script5. Get Cert & Config from script1. Pre-processingimageTokenConfigDataPlaneۄ٘ ী ׳غח ੌ
ܻ - ۽࠺ ో ࢶఖ ߓ҃DKOSDatacenter of Kakao Operating System Provisioning tool+
ܻ - Kubespray ইएSSH450s 300 taskdependency
ܻ - ۽࠺ ో ઁѢProvisioning tool
ܻ - ۽࠺ ో হ ߡ֎౭झ ࢲ࠺झೠ ѾҗSSH450s 300 taskcloud-init20sઓࢿ Xਬ ࠁࣻ Xdependency
नࣘউ ܻ҃ - ۽࠺ ో হ ߡ֎౭झ ࢲ࠺झೠ Ѿҗ
NEXTControlPlane provisioning+ managed pod deployখਵ۽ ೧ঠೡ ੌ
E.O.D