Kubernetes provisioning

#Kubernetes #Provisioning

카카오에서는 쿠버네티스를 서비스하기 위해 DKOS를 사용하고 있습니다.

본 세션에서는 DKOS가 쿠버네티스를 서비스 하기 위해, 고려하였던 프로비저닝 툴들과 선택을 시작으로, 시행착오를 거쳐,
프로비저닝 툴을 걷어내기까지의 여정을 공유 드리고자 합니다.

감사합니다

발표자 : rookie.jeon
카카오에서 클라우드 플랫폼을 개발하고 있는 루키입니다.

Transcript

1. ੹޹୍ rookie.jeon
   ஠஠য়
   Copyright 2022. Kakao Corp. All rights reserved. Redistribution or public display is not permitted without written permission from Kakao.
   Kubernetes provisioning
   if(kakao)2022
   ೐۽࠺੷׬ ోҗ੄ ݅թࠗఠ ೻য૗ө૑ . . .
   

   View Slide

2. ஠஠য়ীࢲ ௢ߡ֎౭झܳ ࢲ࠺झ ೞח ߑध
   ௢ߡ֎౭झ ೐۽࠺੷׬ ో ࢶఖೞӝ
   ਍৔઺ ו՛ ೐۽࠺੷׬ ో(Kubespray)੄ ইए਍੼
   ೐۽࠺੷׬ ో੄ ب਑ হ੉ ௢ߡ֎౭झ ࢲ࠺झ ೞӝ
   ੿ܻ ߂ ݃ޖܻ
   

   View Slide

3. ஠஠য়ীࢲ ௢ߡ֎౭झܳ ࢲ࠺झ ೞח ߑध
   ௢ߡ֎౭झ ೐۽࠺੷׬ ో ࢶఖೞӝ
   ਍৔઺ ו՛ ೐۽࠺੷׬ ో(Kubespray)੄ ইए਍੼
   ೐۽࠺੷׬ ో੄ ب਑ হ੉ ௢ߡ֎౭झ ࢲ࠺झ ೞӝ
   ੿ܻ ߂ ݃ޖܻ
   

   View Slide

4. DKOSۆ ޖ঺ੋо?
   DKOS
   Datacenter of Kakao Operating System
   - Multi
   - Managed
   

   View Slide

5. DKOSۆ ޖ঺ੋо?
   CNI
   Log
   LB Volume
   DNS Ingress
   Managed pod
   Kubernetes cluster
   User
   DKOS
   

   View Slide

6. DKOSۆ ޖ঺ੋо?
   CNI
   Log
   LB Volume
   DNS Ingress
   Managed pod
   Kubernetes cluster
   User
   DKOS
   

   View Slide

7. DKOSۆ ޖ঺ੋо?
   CNI
   Log
   LB Volume
   DNS Ingress
   Managed pod
   Kubernetes cluster
   User
   DKOS
   

   View Slide

8. DKOSۆ ޖ঺ੋо?
   CNI
   Log
   LB Volume
   DNS Ingress
   Managed pod
   Kubernetes cluster
   User
   DKOS
   

   View Slide

9. DKOSীࢲ যڃ ੘সਸ ࣻ೯೧ঠ ೡө?
   1. VM ࢤࢿ (OpenStack ӝ߈੄ ࢎղ IaaS ೐۽ં౟ੋ krane ࢎਊ)
   2. kernel / OS ౵ۄ޷ఠ ࢸ੿
   3. ஶప੉ց ۠ఋ੐ ҳ୷
   4. ௢ߡ֎౭झ ҳ୷
   5. managed pod ߓನ
   

   View Slide

10. Provisioning tool
    ೐۽࠺੷׬ ో੄ ೙ਃࢿ
    

    View Slide

11. ஠஠য়ীࢲ ௢ߡ֎౭झܳ ࢲ࠺झ ೞח ߑध
    ௢ߡ֎౭झ ೐۽࠺੷׬ ో ࢶఖೞӝ
    ਍৔઺ ו՛ ೐۽࠺੷׬ ో(Kubespray)੄ ইए਍੼
    ೐۽࠺੷׬ ో੄ ب਑ হ੉ ௢ߡ֎౭झ ࢲ࠺झ ೞӝ
    ੿ܻ ߂ ݃ޖܻ
    

    View Slide

12. Kubeadm
    “рױ”
    kubeadm੉ۆ ޖ঺ੋо?
    

    View Slide

13. kubeadm੉ۆ ޖ঺ੋо?
    DKOS
    Kubeadm
    ੢੼
    উ੿ࢿ
    рױೣ
    ױ੼
    ગ਷ ૑ਗ ߧਤ
    (য়૒ ௢ߡ֎౭झ ࢸ஖݅ ૑ਗ)
    ѐ߹ ੘স ࣻ೯
    Node
    A - Z
    A - Z
    A - Z
    

    View Slide

14. Kubespray
    “և਷ ߧਤ੄ ੗زച + ׮নೠ ࢶఖ૑ ઁҕ”
    Kubesprayۆ ޖ঺ੋо?
    

    View Slide

15. Kubesprayۆ ޖ঺ੋо?
    Kubespray
    ੢੼
    և਷ ߧਤ੄ ੗زച ૑ਗ
    ׮নೠ ࢶఖ૑ ઁҕ
    ױੌ ࣻ೯ਸ ాೠ ׮ࣻ ֢٘ ઁয оמ
    উ੿ࢿ
    DKOS
    Node
    access info
    + ansible script
    

    View Slide

16. Kubespray ଻ఖ!!
    Kubespray ଻ఖ
    

    View Slide

17. ݥ୾!!
    “Cluster APIח Ҋ۰ೞ૑ ঋওաਃ?”
    ׮ܲ ో਷ ࠺Ү উೞաਃ?
    

    View Slide

18. Kubespray ଻ఖ!!
    Kubespray ଻ఖ
    

    View Slide

19. Kubespray ଻ఖ
    Kubespray ଻ఖ
    

    View Slide

20. Kubespray ଻ఖ??
    Kubespray ଻ఖ
    

    View Slide

21. ஠஠য়ীࢲ ௢ߡ֎౭झܳ ࢲ࠺झ ೞח ߑध
    ௢ߡ֎౭झ ೐۽࠺੷׬ ో ࢶఖೞӝ
    ਍৔઺ ו՛ ೐۽࠺੷׬ ో(Kubespray)੄ ইए਍੼
    ೐۽࠺੷׬ ో੄ ب਑ হ੉ ௢ߡ֎౭झ ࢲ࠺झ ೞӝ
    ੿ܻ ߂ ݃ޖܻ
    

    View Slide

22. ࠁউ ੄ઓࢿ
    ࢸ஖ दр ਬ૑ ࠁࣻ
    Kubespray ইए਍੼
    

    View Slide

23. ࠁউ
    SSH
    Kubespray ইए਍੼ - ࠁউ
    

    View Slide

24. Kubespray ইए਍੼ - ੄ઓࢿ
    20.04 20.04 20.04
    18.04 18.04 18.04
    16.04 16.04 16.04
    Kubespray@v1.19
    Ansible
    Python
    Pkg
    

    View Slide

25. Kubespray ইए਍੼ - ֢٘ ୶о दр
    ੘স ࢸݺ ࣻ೯ दр ࢎ੹ ੘স оמ ৈࠗ
    Download binary ‘container runtime, k8s, CNI . . .’ प೯ ౵ੌ ׮਍ 300s Image ղ੢ оמ
    Set container runtime & k8s container runtime & K8s ࢸ੿ ߂ ҳز 90s ੌࠗ ୊ܻ оמ
    Generate etc cert etcd cert ࢤࢿ 30s DKOS ୊ܻ оמ
    Gather facts anisible – inventory ֢٘ ੿ࠁ ࣻ૘ 20s X
    Set OS proxy, NSS, hostname . . . ࢸ੿ 10s Image ղ੢ оמ
    450s – (Image ղ੢ + DKOS ୊ܻ) = 150s
    ୨ ࢸ஖ दр: 450s
    

    View Slide

26. ਬ૑ ࠁࣻ
    300 ~ 400 Ansible TASK
    Kubespray ইए਍੼ - ਬ૑ ࠁࣻ
    

    View Slide

27. Ҋ޹੉ ݆ҳ݅…
    ࢶఖ੄ ӝ۽
    

    View Slide

28. ׮ܲ ో ૒੽ ѐߊ
    Cluster API A - Z
    ࢶఖ੄ ӝ۽
    

    View Slide

29. Cluster API
    “ࢿמ ੉गী ؀ೠ ੄ޙ”
    cluster-API Ѩష
    

    View Slide

30. Cluster API
    “ࢿמ ੉गী ؀ೠ ੄ޙ”
    cluster-API Ѩష
    

    View Slide

31. ೐۽࠺੷׬ ో ઁѢ
    Provisioning tool
    

    View Slide

32. ஠஠য়ীࢲ ௢ߡ֎౭झܳ ࢲ࠺झ ೞח ߑध
    ௢ߡ֎౭झ ೐۽࠺੷׬ ో ࢶఖೞӝ
    ਍৔઺ ו՛ ೐۽࠺੷׬ ో(Kubespray)੄ ইए਍੼
    ೐۽࠺੷׬ ో੄ ب਑ হ੉ ௢ߡ֎౭झ ࢲ࠺झ ೞӝ
    ੿ܻ ߂ ݃ޖܻ
    

    View Slide

33. DKOSীࢲ যڃ ੘সਸ ࣻ೯೧ঠ ೡө?
    1. VM ࢤࢿ (OpenStack ӝ߈੄ IaaS ೐۽ં౟ੋ krane ࢎਊ)
    2. kernel / OS ౵ۄ޷ఠ ࢸ੿
    3. ஶప੉ց ۠ఋ੐ ҳ୷
    4. ௢ߡ֎౭झ ҳ୷
    5. managed pod ߓನ
    

    View Slide

34. DKOSীࢲ যڃ ੘সਸ ࣻ೯೧ঠ ೡө?
    1. VM ࢤࢿ (OpenStack ӝ߈੄ IaaS ೐۽ં౟ੋ krane ࢎਊ)
    2. kernel / OS ౵ۄ޷ఠ ࢸ੿
    3. ஶప੉ց ۠ఋ੐ ҳ୷
    4. ௢ߡ֎౭झ ҳ୷
    5. managed pod ߓನ
    Control
    plane
    1 - 5 ױ҅ ૓೯
    ֢٘ ୶о ੿੸
    Data
    plane
    1 - 4 ױ҅ ૓೯
    ֢٘ ୶о ഝߊ
    ࢸ஖ दрী ޹х
    

    View Slide

35. DKOSীࢲ যڃ ੘সਸ ࣻ೯೧ঠ ೡө?
    1. VM ࢤࢿ (OpenStack ӝ߈੄ IaaS ೐۽ં౟ੋ krane ࢎਊ)
    2. kernel / OS ౵ۄ޷ఠ ࢸ੿
    3. ஶప੉ց ۠ఋ੐ ҳ୷
    4. ௢ߡ֎౭झ ҳ୷
    5. managed pod ߓನ
    Control
    plane
    1 - 5 ױ҅ ૓೯
    ֢٘ ୶о ੿੸
    Data
    plane
    1 - 4 ױ҅ ૓೯
    ֢٘ ୶о ഝߊ
    ࢸ஖ दрী ޹х
    

    View Slide

36. DKOSীࢲ যڃ ੘সਸ ࣻ೯೧ঠ ೡө?
    1. VM ࢤࢿ (OpenStack ӝ߈੄ IaaS ೐۽ં౟ੋ krane ࢎਊ)
    2. kernel / OS ౵ۄ޷ఠ ࢸ੿
    3. ஶప੉ց ۠ఋ੐ ҳ୷
    4. ௢ߡ֎౭झ ҳ୷
    5. managed pod ߓನ
    

    View Slide

37. DKOSীࢲ যڃ ੘সਸ ࣻ೯೧ঠ ೡө?
    1. VM ࢤࢿ (OpenStack ӝ߈੄ IaaS ೐۽ં౟ੋ krane ࢎਊ)
    2. kernel / OS ౵ۄ޷ఠ ࢸ੿
    3. ஶప੉ց ۠ఋ੐ ҳ୷
    4. ௢ߡ֎౭झ ҳ୷
    5. managed pod ߓನ
    ੿੸ ؘ੉ఠ
    +
    ز੸ ؘ੉ఠ
    

    View Slide

38. DKOSীࢲ যڃ ੘সਸ ࣻ೯೧ঠ ೡө?
    1. VM ࢤࢿ (OpenStack ӝ߈੄ IaaS ೐۽ં౟ੋ krane ࢎਊ)
    2. kernel / OS ౵ۄ޷ఠ ࢸ੿
    3. ஶప੉ց ۠ఋ੐ ҳ୷
    4. ௢ߡ֎౭झ ҳ୷
    5. managed pod ߓನ
    ੿੸ ؘ੉ఠ
    (Image)
    +
    ز੸ ؘ੉ఠ
    (Cloud-init)
    

    View Slide

39. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द
    

    View Slide

40. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    Pre-processing
    image
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द
    

    View Slide

41. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    DataPlane
    Create VM
    With script
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द
    

    View Slide

42. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    DataPlane
    Create VM
    With script
    Upload script
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द
    

    View Slide

43. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    DataPlane
    Create VM
    With script Create VM
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द
    Pre-processing
    image
    

    View Slide

44. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    1. Request cloud-init script
    2. Get Cert & Config from script
    DataPlane
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द
    

    View Slide

45. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane Config
    Cert
    1. Request cloud-init script
    2. Get Cert & Config from script
    DataPlane
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द
    

    View Slide

46. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane Config
    Cert ControlPlane
    Join
    DataPlane
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द
    

    View Slide

47. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane Config
    Cert ControlPlane
    Join
    DataPlane
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द
    Done!!
    

    View Slide

48. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    2. Create VM
    With script 3. Create VM
    3. Upload script
    Config
    Cert
    6. Join
    ControlPlane
    4. Request cloud-init script
    5. Get Cert & Config from script
    1. Pre-processing
    image
    DataPlane
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द
    

    View Slide

49. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    2. Create VM
    With script 3. Create VM
    3. Upload script
    Config
    Cert
    6. Join
    ControlPlane
    4. Request cloud-init script
    5. Get Cert & Config from script
    1. Pre-processing
    image
    DataPlane
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द
    

    View Slide

50. যڌѱ?
    “ઑੋ җ੿”
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿
    

    View Slide

51. DataPlane ControlPlane
    kube-
    apiserver
    kube-
    scheduler
    controller-manager
    etcd
    Kubelet
    Kube-Proxy
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿ - ৘द
    

    View Slide

52. DataPlane ControlPlane
    Kubelet
    Kube-Proxy
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿ - ৘द
    kube-
    apiserver
    kube-
    scheduler
    controller-manager
    etcd
    

    View Slide

53. DataPlane ControlPlane
    Kubelet
    Kube-Proxy
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿ - ৘द
    Cert
    kube-
    apiserver
    kube-
    scheduler
    controller-manager
    etcd
    

    View Slide

54. DataPlane ControlPlane
    Kubelet
    Kube-Proxy
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿ - ৘द
    kube-
    apiserver
    kube-
    scheduler
    controller-manager
    etcd
    

    View Slide

55. DataPlane ControlPlane
    Kubelet
    Kube-Proxy
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿ - ৘द
    Token
    1. Create Token
    kube-
    apiserver
    kube-
    scheduler
    controller-manager
    etcd
    

    View Slide

56. DataPlane ControlPlane
    Kubelet
    Kube-Proxy
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿ - ৘द
    Token
    Token
    Config
    1. Create Token
    2. Create config
    kube-
    apiserver
    kube-
    scheduler
    controller-manager
    etcd
    

    View Slide

57. DataPlane ControlPlane
    Kubelet
    Kube-Proxy
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿ - ৘द
    Token
    Token
    Config
    request
    cert kube-
    apiserver
    kube-
    scheduler
    controller-manager
    etcd
    

    View Slide

58. DataPlane ControlPlane
    Kubelet
    Kube-Proxy
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿ - ৘द
    Token
    send
    cert
    Cert
    kube-
    apiserver
    kube-
    scheduler
    controller-manager
    etcd
    

    View Slide

59. DataPlane ControlPlane
    Kubelet
    Kube-Proxy
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿ - ৘द
    Token
    Cert
    join
    kube-
    apiserver
    kube-
    scheduler
    controller-manager
    etcd
    

    View Slide

60. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    2. Create VM
    With script 3. Create VM
    3. Upload script
    Config
    Cert
    6. Join
    ControlPlane
    4. Request cloud-init script
    5. Get Cert & Config from script
    1. Pre-processing
    image
    DataPlane
    ௿ۄ਋٘ ੉׫ী ੹׳غח ౵ੌ
    

    View Slide

61. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    2. Create VM
    With script 3. Create VM
    3. Upload script
    Config
    Cert
    6. Join
    ControlPlane
    4. Request cloud-init script
    5. Get Cert & Config from script
    1. Pre-processing
    image
    Token
    Config
    DataPlane
    ௿ۄ਋٘ ੉׫ী ੹׳غח ౵ੌ
    

    View Slide

62. ஠஠য়ীࢲ ௢ߡ֎౭झܳ ࢲ࠺झ ೞח ߑध
    ௢ߡ֎౭झ ೐۽࠺੷׬ ో ࢶఖೞӝ
    ਍৔઺ ו՛ ೐۽࠺੷׬ ో(Kubespray)੄ ইए਍੼
    ೐۽࠺੷׬ ో੄ ب਑ হ੉ ௢ߡ֎౭झ ࢲ࠺झ ೞӝ
    ੿ܻ ߂ ݃ޖܻ
    

    View Slide

63. ੿ܻ - ೐۽࠺੷׬ ో ࢶఖ ߓ҃
    DKOS
    Datacenter of Kakao Operating System Provisioning tool
    +
    

    View Slide

64. ੿ܻ - Kubespray੄ ইए਍੼
    SSH
    450s 300 task
    dependency
    

    View Slide

65. ੿ܻ - ೐۽࠺੷׬ ో ઁѢ
    Provisioning tool
    

    View Slide

66. ੿ܻ - Kubespray੄ ইए਍੼
    SSH
    450s 300 task
    dependency
    

    View Slide

67. ੿ܻ - ೐۽࠺੷׬ ో হ੉ ௢ߡ֎౭झ ࢲ࠺झೠ Ѿҗ
    SSH
    450s 300 task
    cloud-init
    20s
    ੄ઓࢿ X
    ਬ૑ ࠁࣻ X
    dependency
    

    View Slide

68. नࣘ
    উ੹ ҃۝
    ੿ܻ - ೐۽࠺੷׬ ో হ੉ ௢ߡ֎౭झ ࢲ࠺झೠ Ѿҗ
    

    View Slide

69. NEXT
    ControlPlane provisioning
    + managed pod deploy
    খਵ۽ ೧ঠೡ ੌ
    

    View Slide

70. E.O.D
    

    View Slide