Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes provisioning

kakao
PRO
December 09, 2022

Kubernetes provisioning

#Kubernetes #Provisioning

카카오에서는 쿠버네티스를 서비스하기 위해 DKOS를 사용하고 있습니다.

본 세션에서는 DKOS가 쿠버네티스를 서비스 하기 위해, 고려하였던 프로비저닝 툴들과 선택을 시작으로, 시행착오를 거쳐,
프로비저닝 툴을 걷어내기까지의 여정을 공유 드리고자 합니다.

감사합니다

발표자 : rookie.jeon
카카오에서 클라우드 플랫폼을 개발하고 있는 루키입니다.

kakao
PRO

December 09, 2022
Tweet

More Decks by kakao

Other Decks in Programming

Transcript

  1. ੹޹୍ rookie.jeon
    ஠஠য়
    Copyright 2022. Kakao Corp. All rights reserved. Redistribution or public display is not permitted without written permission from Kakao.
    Kubernetes provisioning
    if(kakao)2022
    ೐۽࠺੷׬ ోҗ੄ ݅թࠗఠ ೻য૗ө૑ . . .

    View Slide

  2. ஠஠য়ীࢲ ௢ߡ֎౭झܳ ࢲ࠺झ ೞח ߑध
    ௢ߡ֎౭झ ೐۽࠺੷׬ ో ࢶఖೞӝ
    ਍৔઺ ו՛ ೐۽࠺੷׬ ో(Kubespray)੄ ইए਍੼
    ೐۽࠺੷׬ ో੄ ب਑ হ੉ ௢ߡ֎౭झ ࢲ࠺झ ೞӝ
    ੿ܻ ߂ ݃ޖܻ

    View Slide

  3. ஠஠য়ীࢲ ௢ߡ֎౭झܳ ࢲ࠺झ ೞח ߑध
    ௢ߡ֎౭झ ೐۽࠺੷׬ ో ࢶఖೞӝ
    ਍৔઺ ו՛ ೐۽࠺੷׬ ో(Kubespray)੄ ইए਍੼
    ೐۽࠺੷׬ ో੄ ب਑ হ੉ ௢ߡ֎౭झ ࢲ࠺झ ೞӝ
    ੿ܻ ߂ ݃ޖܻ

    View Slide

  4. DKOSۆ ޖ঺ੋо?
    DKOS
    Datacenter of Kakao Operating System
    - Multi
    - Managed

    View Slide

  5. DKOSۆ ޖ঺ੋо?
    CNI
    Log
    LB Volume
    DNS Ingress
    Managed pod
    Kubernetes cluster
    User
    DKOS

    View Slide

  6. DKOSۆ ޖ঺ੋо?
    CNI
    Log
    LB Volume
    DNS Ingress
    Managed pod
    Kubernetes cluster
    User
    DKOS

    View Slide

  7. DKOSۆ ޖ঺ੋо?
    CNI
    Log
    LB Volume
    DNS Ingress
    Managed pod
    Kubernetes cluster
    User
    DKOS

    View Slide

  8. DKOSۆ ޖ঺ੋо?
    CNI
    Log
    LB Volume
    DNS Ingress
    Managed pod
    Kubernetes cluster
    User
    DKOS

    View Slide

  9. DKOSীࢲ যڃ ੘সਸ ࣻ೯೧ঠ ೡө?
    1. VM ࢤࢿ (OpenStack ӝ߈੄ ࢎղ IaaS ೐۽ં౟ੋ krane ࢎਊ)
    2. kernel / OS ౵ۄ޷ఠ ࢸ੿
    3. ஶప੉ց ۠ఋ੐ ҳ୷
    4. ௢ߡ֎౭झ ҳ୷
    5. managed pod ߓನ

    View Slide

  10. Provisioning tool
    ೐۽࠺੷׬ ో੄ ೙ਃࢿ

    View Slide

  11. ஠஠য়ীࢲ ௢ߡ֎౭झܳ ࢲ࠺झ ೞח ߑध
    ௢ߡ֎౭झ ೐۽࠺੷׬ ో ࢶఖೞӝ
    ਍৔઺ ו՛ ೐۽࠺੷׬ ో(Kubespray)੄ ইए਍੼
    ೐۽࠺੷׬ ో੄ ب਑ হ੉ ௢ߡ֎౭झ ࢲ࠺झ ೞӝ
    ੿ܻ ߂ ݃ޖܻ

    View Slide

  12. Kubeadm
    “рױ”
    kubeadm੉ۆ ޖ঺ੋо?

    View Slide

  13. kubeadm੉ۆ ޖ঺ੋо?
    DKOS
    Kubeadm
    ੢੼
    উ੿ࢿ
    рױೣ
    ױ੼
    ગ਷ ૑ਗ ߧਤ
    (য়૒ ௢ߡ֎౭झ ࢸ஖݅ ૑ਗ)
    ѐ߹ ੘স ࣻ೯
    Node
    A - Z
    A - Z
    A - Z

    View Slide

  14. Kubespray
    “և਷ ߧਤ੄ ੗زച + ׮নೠ ࢶఖ૑ ઁҕ”
    Kubesprayۆ ޖ঺ੋо?

    View Slide

  15. Kubesprayۆ ޖ঺ੋо?
    Kubespray
    ੢੼
    և਷ ߧਤ੄ ੗زച ૑ਗ
    ׮নೠ ࢶఖ૑ ઁҕ
    ױੌ ࣻ೯ਸ ాೠ ׮ࣻ ֢٘ ઁয оמ
    উ੿ࢿ
    DKOS
    Node
    access info
    + ansible script

    View Slide

  16. Kubespray ଻ఖ!!
    Kubespray ଻ఖ

    View Slide

  17. ݥ୾!!
    “Cluster APIח Ҋ۰ೞ૑ ঋওաਃ?”
    ׮ܲ ో਷ ࠺Ү উೞաਃ?

    View Slide

  18. Kubespray ଻ఖ!!
    Kubespray ଻ఖ

    View Slide

  19. Kubespray ଻ఖ
    Kubespray ଻ఖ

    View Slide

  20. Kubespray ଻ఖ??
    Kubespray ଻ఖ

    View Slide

  21. ஠஠য়ীࢲ ௢ߡ֎౭झܳ ࢲ࠺झ ೞח ߑध
    ௢ߡ֎౭झ ೐۽࠺੷׬ ో ࢶఖೞӝ
    ਍৔઺ ו՛ ೐۽࠺੷׬ ో(Kubespray)੄ ইए਍੼
    ೐۽࠺੷׬ ో੄ ب਑ হ੉ ௢ߡ֎౭झ ࢲ࠺झ ೞӝ
    ੿ܻ ߂ ݃ޖܻ

    View Slide

  22. ࠁউ ੄ઓࢿ
    ࢸ஖ दр ਬ૑ ࠁࣻ
    Kubespray ইए਍੼

    View Slide

  23. ࠁউ
    SSH
    Kubespray ইए਍੼ - ࠁউ

    View Slide

  24. Kubespray ইए਍੼ - ੄ઓࢿ
    20.04 20.04 20.04
    18.04 18.04 18.04
    16.04 16.04 16.04
    [email protected]
    Ansible
    Python
    Pkg

    View Slide

  25. Kubespray ইए਍੼ - ֢٘ ୶о दр
    ੘স ࢸݺ ࣻ೯ दр ࢎ੹ ੘স оמ ৈࠗ
    Download binary ‘container runtime, k8s, CNI . . .’ प೯ ౵ੌ ׮਍ 300s Image ղ੢ оמ
    Set container runtime & k8s container runtime & K8s ࢸ੿ ߂ ҳز 90s ੌࠗ ୊ܻ оמ
    Generate etc cert etcd cert ࢤࢿ 30s DKOS ୊ܻ оמ
    Gather facts anisible – inventory ֢٘ ੿ࠁ ࣻ૘ 20s X
    Set OS proxy, NSS, hostname . . . ࢸ੿ 10s Image ղ੢ оמ
    450s – (Image ղ੢ + DKOS ୊ܻ) = 150s
    ୨ ࢸ஖ दр: 450s

    View Slide

  26. ਬ૑ ࠁࣻ
    300 ~ 400 Ansible TASK
    Kubespray ইए਍੼ - ਬ૑ ࠁࣻ

    View Slide

  27. Ҋ޹੉ ݆ҳ݅…
    ࢶఖ੄ ӝ۽

    View Slide

  28. ׮ܲ ో ૒੽ ѐߊ
    Cluster API A - Z
    ࢶఖ੄ ӝ۽

    View Slide

  29. Cluster API
    “ࢿמ ੉गী ؀ೠ ੄ޙ”
    cluster-API Ѩష

    View Slide

  30. Cluster API
    “ࢿמ ੉गী ؀ೠ ੄ޙ”
    cluster-API Ѩష

    View Slide

  31. ೐۽࠺੷׬ ో ઁѢ
    Provisioning tool

    View Slide

  32. ஠஠য়ীࢲ ௢ߡ֎౭झܳ ࢲ࠺झ ೞח ߑध
    ௢ߡ֎౭झ ೐۽࠺੷׬ ో ࢶఖೞӝ
    ਍৔઺ ו՛ ೐۽࠺੷׬ ో(Kubespray)੄ ইए਍੼
    ೐۽࠺੷׬ ో੄ ب਑ হ੉ ௢ߡ֎౭झ ࢲ࠺झ ೞӝ
    ੿ܻ ߂ ݃ޖܻ

    View Slide

  33. DKOSীࢲ যڃ ੘সਸ ࣻ೯೧ঠ ೡө?
    1. VM ࢤࢿ (OpenStack ӝ߈੄ IaaS ೐۽ં౟ੋ krane ࢎਊ)
    2. kernel / OS ౵ۄ޷ఠ ࢸ੿
    3. ஶప੉ց ۠ఋ੐ ҳ୷
    4. ௢ߡ֎౭झ ҳ୷
    5. managed pod ߓನ

    View Slide

  34. DKOSীࢲ যڃ ੘সਸ ࣻ೯೧ঠ ೡө?
    1. VM ࢤࢿ (OpenStack ӝ߈੄ IaaS ೐۽ં౟ੋ krane ࢎਊ)
    2. kernel / OS ౵ۄ޷ఠ ࢸ੿
    3. ஶప੉ց ۠ఋ੐ ҳ୷
    4. ௢ߡ֎౭झ ҳ୷
    5. managed pod ߓನ
    Control
    plane
    1 - 5 ױ҅ ૓೯
    ֢٘ ୶о ੿੸
    Data
    plane
    1 - 4 ױ҅ ૓೯
    ֢٘ ୶о ഝߊ
    ࢸ஖ दрী ޹х

    View Slide

  35. DKOSীࢲ যڃ ੘সਸ ࣻ೯೧ঠ ೡө?
    1. VM ࢤࢿ (OpenStack ӝ߈੄ IaaS ೐۽ં౟ੋ krane ࢎਊ)
    2. kernel / OS ౵ۄ޷ఠ ࢸ੿
    3. ஶప੉ց ۠ఋ੐ ҳ୷
    4. ௢ߡ֎౭झ ҳ୷
    5. managed pod ߓನ
    Control
    plane
    1 - 5 ױ҅ ૓೯
    ֢٘ ୶о ੿੸
    Data
    plane
    1 - 4 ױ҅ ૓೯
    ֢٘ ୶о ഝߊ
    ࢸ஖ दрী ޹х

    View Slide

  36. DKOSীࢲ যڃ ੘সਸ ࣻ೯೧ঠ ೡө?
    1. VM ࢤࢿ (OpenStack ӝ߈੄ IaaS ೐۽ં౟ੋ krane ࢎਊ)
    2. kernel / OS ౵ۄ޷ఠ ࢸ੿
    3. ஶప੉ց ۠ఋ੐ ҳ୷
    4. ௢ߡ֎౭झ ҳ୷
    5. managed pod ߓನ

    View Slide

  37. DKOSীࢲ যڃ ੘সਸ ࣻ೯೧ঠ ೡө?
    1. VM ࢤࢿ (OpenStack ӝ߈੄ IaaS ೐۽ં౟ੋ krane ࢎਊ)
    2. kernel / OS ౵ۄ޷ఠ ࢸ੿
    3. ஶప੉ց ۠ఋ੐ ҳ୷
    4. ௢ߡ֎౭झ ҳ୷
    5. managed pod ߓನ
    ੿੸ ؘ੉ఠ
    +
    ز੸ ؘ੉ఠ

    View Slide

  38. DKOSীࢲ যڃ ੘সਸ ࣻ೯೧ঠ ೡө?
    1. VM ࢤࢿ (OpenStack ӝ߈੄ IaaS ೐۽ં౟ੋ krane ࢎਊ)
    2. kernel / OS ౵ۄ޷ఠ ࢸ੿
    3. ஶప੉ց ۠ఋ੐ ҳ୷
    4. ௢ߡ֎౭झ ҳ୷
    5. managed pod ߓನ
    ੿੸ ؘ੉ఠ
    (Image)
    +
    ز੸ ؘ੉ఠ
    (Cloud-init)

    View Slide

  39. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द

    View Slide

  40. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    Pre-processing
    image
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द

    View Slide

  41. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    DataPlane
    Create VM
    With script
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द

    View Slide

  42. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    DataPlane
    Create VM
    With script
    Upload script
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द

    View Slide

  43. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    DataPlane
    Create VM
    With script Create VM
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द
    Pre-processing
    image

    View Slide

  44. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    1. Request cloud-init script
    2. Get Cert & Config from script
    DataPlane
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द

    View Slide

  45. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane Config
    Cert
    1. Request cloud-init script
    2. Get Cert & Config from script
    DataPlane
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द

    View Slide

  46. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane Config
    Cert ControlPlane
    Join
    DataPlane
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द

    View Slide

  47. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane Config
    Cert ControlPlane
    Join
    DataPlane
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द
    Done!!

    View Slide

  48. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    2. Create VM
    With script 3. Create VM
    3. Upload script
    Config
    Cert
    6. Join
    ControlPlane
    4. Request cloud-init script
    5. Get Cert & Config from script
    1. Pre-processing
    image
    DataPlane
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द

    View Slide

  49. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    2. Create VM
    With script 3. Create VM
    3. Upload script
    Config
    Cert
    6. Join
    ControlPlane
    4. Request cloud-init script
    5. Get Cert & Config from script
    1. Pre-processing
    image
    DataPlane
    ؘ੉ఠ ೒ۨੋ ೐۽࠺੷׬ - ৘द

    View Slide

  50. যڌѱ?
    “ઑੋ җ੿”
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿

    View Slide

  51. DataPlane ControlPlane
    kube-
    apiserver
    kube-
    scheduler
    controller-manager
    etcd
    Kubelet
    Kube-Proxy
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿ - ৘द

    View Slide

  52. DataPlane ControlPlane
    Kubelet
    Kube-Proxy
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿ - ৘द
    kube-
    apiserver
    kube-
    scheduler
    controller-manager
    etcd

    View Slide

  53. DataPlane ControlPlane
    Kubelet
    Kube-Proxy
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿ - ৘द
    Cert
    kube-
    apiserver
    kube-
    scheduler
    controller-manager
    etcd

    View Slide

  54. DataPlane ControlPlane
    Kubelet
    Kube-Proxy
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿ - ৘द
    kube-
    apiserver
    kube-
    scheduler
    controller-manager
    etcd

    View Slide

  55. DataPlane ControlPlane
    Kubelet
    Kube-Proxy
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿ - ৘द
    Token
    1. Create Token
    kube-
    apiserver
    kube-
    scheduler
    controller-manager
    etcd

    View Slide

  56. DataPlane ControlPlane
    Kubelet
    Kube-Proxy
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿ - ৘द
    Token
    Token
    Config
    1. Create Token
    2. Create config
    kube-
    apiserver
    kube-
    scheduler
    controller-manager
    etcd

    View Slide

  57. DataPlane ControlPlane
    Kubelet
    Kube-Proxy
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿ - ৘द
    Token
    Token
    Config
    request
    cert kube-
    apiserver
    kube-
    scheduler
    controller-manager
    etcd

    View Slide

  58. DataPlane ControlPlane
    Kubelet
    Kube-Proxy
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿ - ৘द
    Token
    send
    cert
    Cert
    kube-
    apiserver
    kube-
    scheduler
    controller-manager
    etcd

    View Slide

  59. DataPlane ControlPlane
    Kubelet
    Kube-Proxy
    ؘ੉ఠ ೒ۨੋ੄ ஶ౟܀ ೒ۨੋ ઑੋ җ੿ - ৘द
    Token
    Cert
    join
    kube-
    apiserver
    kube-
    scheduler
    controller-manager
    etcd

    View Slide

  60. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    2. Create VM
    With script 3. Create VM
    3. Upload script
    Config
    Cert
    6. Join
    ControlPlane
    4. Request cloud-init script
    5. Get Cert & Config from script
    1. Pre-processing
    image
    DataPlane
    ௿ۄ਋٘ ੉׫ী ੹׳غח ౵ੌ

    View Slide

  61. DKOS
    Image Repo
    Cloud-init
    Data center
    Krane
    2. Create VM
    With script 3. Create VM
    3. Upload script
    Config
    Cert
    6. Join
    ControlPlane
    4. Request cloud-init script
    5. Get Cert & Config from script
    1. Pre-processing
    image
    Token
    Config
    DataPlane
    ௿ۄ਋٘ ੉׫ী ੹׳غח ౵ੌ

    View Slide

  62. ஠஠য়ীࢲ ௢ߡ֎౭झܳ ࢲ࠺झ ೞח ߑध
    ௢ߡ֎౭झ ೐۽࠺੷׬ ో ࢶఖೞӝ
    ਍৔઺ ו՛ ೐۽࠺੷׬ ో(Kubespray)੄ ইए਍੼
    ೐۽࠺੷׬ ో੄ ب਑ হ੉ ௢ߡ֎౭झ ࢲ࠺झ ೞӝ
    ੿ܻ ߂ ݃ޖܻ

    View Slide

  63. ੿ܻ - ೐۽࠺੷׬ ో ࢶఖ ߓ҃
    DKOS
    Datacenter of Kakao Operating System Provisioning tool
    +

    View Slide

  64. ੿ܻ - Kubespray੄ ইए਍੼
    SSH
    450s 300 task
    dependency

    View Slide

  65. ੿ܻ - ೐۽࠺੷׬ ో ઁѢ
    Provisioning tool

    View Slide

  66. ੿ܻ - Kubespray੄ ইए਍੼
    SSH
    450s 300 task
    dependency

    View Slide

  67. ੿ܻ - ೐۽࠺੷׬ ో হ੉ ௢ߡ֎౭झ ࢲ࠺झೠ Ѿҗ
    SSH
    450s 300 task
    cloud-init
    20s
    ੄ઓࢿ X
    ਬ૑ ࠁࣻ X
    dependency

    View Slide

  68. नࣘ
    উ੹ ҃۝
    ੿ܻ - ೐۽࠺੷׬ ో হ੉ ௢ߡ֎౭झ ࢲ࠺झೠ Ѿҗ

    View Slide

  69. NEXT
    ControlPlane provisioning
    + managed pod deploy
    খਵ۽ ೧ঠೡ ੌ

    View Slide

  70. E.O.D

    View Slide