쿠버네티스 펀하고 쿨하고 섹시하게 만들기

#Kubernetes #ClusterAPI #Kubernetesasaservice

카카오 엔터프라이즈에서는 cluster api와 다양한 plugin을 이용하여, 개발자들에게 용도에 맞는 k8s를 제공하고 있습니다. 그리고 이 환경을 기반으로 모든 클라우드 배포 환경을 k8s를 이용하여 구현할 수 있었습니다.

카카오 엔터프라이즈에서 사용중인 선언적인 kubernetes as a service 개발 방법 공유!, 이것이 선언적인 방법이니까! 음!

발표자 : gonzales.son
카카오엔터프라이즈에서 퍼블릭 클라우드 서비스를 개발하고 있는 벅과 곤잘레스 입니다. 카카오아이클라우드 최고에요~!

buck.hur
카카오엔터프라이즈에서 퍼블릭 클라우드 서비스를 개발하고 있는 벅과 곤잘레스 입니다. 카카오아이클라우드 최고에요~!

Transcript

1. Copyright 2022. Kakao Corp. All rights reserved. Redistribution or public display is not permitted without written permission from Kakao.
   쿠버네티스 펀하고, 쿨하고, 섹시하게
   손준호 gonzales.son, 허진 buck.hur
   
   
   카카오엔터프라이즈, 멀티버스팀, 코어파트
   if(kakao)2022
   

   View Slide

2. Fun
   
   
   1. 인지
   
   
   2. 선택
   
   
   Cool
   
   
   3. 서비스
   
   
   Sexy
   
   
   4. 경험
   
   
   5. 계획
   2
   

   View Slide

3. 펀하게!
   
   
   인지
   

   View Slide

4. Kubernetes 얼마나 필요한가요?
   a lot!
   

   View Slide

5. 왜 이런 구조로 구성되었나?
   Master plan
   
   
   K8S
   
   
   프로비저닝 서비스
   KIC
   
   
   관리형 k8s 서비스
   KIC
   
   
   리전 프로비저닝/
   
   
   인스톨 패키징
   KIC
   
   
   리전 인스톨러
   

   View Slide

6. Kubernetes everywhere
   왜 이런 구조로 구성되었나?
   

   View Slide

7. 어떤 서비스 인가?
   kubernetes as a service
   
   
   Underground
   
   
   IKE
   

   View Slide

8. 설명하기에 앞서 - Term
   Term Description
   KIC kakao i cloud의 약자
   KIC Console kakao i cloud 리소스 생성 및 확인이 가능한 console dashboard
   Overcloud
   KIC Ҋёীѱ ௿ۄ਋٘ ܻࣗझܳ ઁҕೞӝ ਤೠ H/W, S/W੄ ૘೤. IaaS ࢲ࠺झܳ ਤೠ ױة ֎౟ਕ௼
   Fabric ҳઑܳ ыח׮.
   
   ௿ۄ਋٘ী ੄೧ ٜ݅য૓ ܻࣗझо ઓ੤ೞח ৔৉. IKE ࢲ࠺झ۽ vpc ӝ߈੄ k8s ௿۞झఠ ೐۽࠺੷׬
   Undercloud
   Overcloud ҙܻܳ ਤೠ Managementী ೙ਃೠ H/W, S/W੄ ૘೤. Overcloud৬ ҳ࠙غח ױة ֎౟ਕ௼
   Fabric ҳઑܳ ыח׮.
   
   ௿ۄ਋٘ܳ ٜ݅ӝ ਤೠ ܻࣗझо ઓ੤ೞח ৔৉. Underground ࢲ࠺झ۽ k8s ௿۞झఠ ೐۽࠺੷׬
   
   

   View Slide

9. 펀하게!
   
   
   선택
   

   View Slide

10. 어떻게 제공할 것인가?
    Kubernetes Cluster API
    

    View Slide

11. 어떻게 제공할 것인가?
    Reason
    
    
    Kubernetes style api/resource(crd)
    
    
    declarative cluster lifecycle 관리 방법
    
    
    다양한 infrastructure provider 지원
    

    View Slide

12. ୹୊: https://cluster-api.sigs.k8s.io/user/concepts.html
    

    View Slide

13. Cluster API의 이해
    선언적 방식의 cluster lifecycle management
    
    
    Controller patern / Reconcile
    In applications of robotics and automation, a control loop is a non
    -
    terminating loop that
    regulates the state of the system. In Kubernetes, a controller is a control loop that watches the
    shared state of the cluster through the apiserver and makes changes attempting to move the
    current state towards the desired state.
    ୹୊: https://kubernetes.io/docs/concepts/architecture/controller/
    

    View Slide

14. Cluster API의 이해
    선언적 방식의 cluster lifecycle management
    
    
    Custom Resource De
    fi
    nition
    The CustomResourceDe
    fi
    nition API resource allows you to de
    fi
    ne custom resources. De
    fi
    ning a
    CRD object creates a new custom resource with a name and schema that you specify. The
    Kubernetes API serves and handles the storage of your custom resource. The name of a CRD
    object must be a valid DNS subdomain name.
    ୹୊: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
    

    View Slide

15. ୹୊: https://cluster-api.sigs.k8s.io/user/concepts.html
    Cluster API의 이해 - contollers/CRs
    Cluster
    Kubeadm
    
    
    ControlPlane
    InfrastructureCluster
    kubeadm
    
    
    con
    fi
    gtemplate
    Infrastructure
    
    
    machinetemplate
    Machine
    
    
    Deployment
    Infrastructure
    
    
    machinetemplate
    master worker
    

    View Slide

16. ୹୊:https://cluster-api.sigs.k8s.io/developer/architecture/controllers.html
    Cluster API의 이해 - contollers/CRs
    Cluster
    Kubeadm
    
    
    ControlPlane
    InfrastructureCluster
    kubeadm
    
    
    con
    fi
    gtemplate
    Infrastructure
    
    
    machinetemplate
    Machine
    
    
    Deployment
    Infrastructure
    
    
    machinetemplate
    worker
    Machine
    InfrastructureMachine
    machineset
    KubeadmCon
    fi
    g
    Machine
    InfrastructureMachine
    KubeadmCon
    fi
    g
    Reference
    Management
    Core
    
    
    controller
    Controlplane
    
    
    controller
    master
    Bootstrap
    
    
    controller
    infrastructure
    
    
    controller
    Spec.clusterName
    Spec.controlPlaneRef Spec.InfrastructureRef
    providers
    

    View Slide

17. ୹୊:https://cluster-api.sigs.k8s.io/developer/architecture/controllers.html
    Cluster API의 이해 - contollers/CRs
    Cluster
    InfrastructureCluster
    Infrastructure
    
    
    machinetemplate
    Machine
    InfrastructureMachine
    KubeadmCon
    fi
    g
    Reference
    Management
    master
    Spec.clusterName
    Spec.controlPlaneRef Spec.InfrastructureRef
    kubeadm
    
    
    con
    fi
    gtemplate
    Machine
    
    
    Deployment
    Infrastructure
    
    
    machinetemplate
    worker
    machineset
    Machine
    InfrastructureMachine
    KubeadmCon
    fi
    g
    Kubeadm
    
    
    ControlPlane
    Core
    
    
    controller
    Controlplane
    
    
    controller
    Bootstrap
    
    
    controller
    infrastructure
    
    
    controller
    reconcile()
    

    View Slide

18. Kubeadm
    
    
    ControlPlane
    ୹୊:https://cluster-api.sigs.k8s.io/developer/architecture/controllers.html
    Cluster API의 이해 - contollers/CRs
    Cluster
    InfrastructureCluster
    Infrastructure
    
    
    machinetemplate
    Machine
    InfrastructureMachine
    KubeadmCon
    fi
    g
    Reference
    Management
    master
    Spec.clusterName
    Spec.controlPlaneRef Spec.InfrastructureRef
    kubeadm
    
    
    con
    fi
    gtemplate
    Machine
    
    
    Deployment
    Infrastructure
    
    
    machinetemplate
    worker
    machineset
    Machine
    InfrastructureMachine
    KubeadmCon
    fi
    g
    Core
    
    
    controller
    Controlplane
    
    
    controller
    Bootstrap
    
    
    controller
    infrastructure
    
    
    controller
    reconcile() reconcile()
    

    View Slide

19. ୹୊:https://cluster-api.sigs.k8s.io/developer/architecture/controllers.html
    Cluster API의 이해 - contollers/CRs
    worker
    Reference
    Management
    Core
    
    
    controller
    master
    Controlplane
    
    
    controller
    Bootstrap
    
    
    controller
    infrastructure
    
    
    controller
    reconcile() reconcile()
    Cluster
    Kubeadm
    
    
    ControlPlane
    InfrastructureCluster
    kubeadm
    
    
    con
    fi
    gtemplate
    Infrastructure
    
    
    machinetemplate
    Machine
    
    
    Deployment
    Infrastructure
    
    
    machinetemplate
    Machine
    InfrastructureMachine
    machineset
    KubeadmCon
    fi
    g
    Machine
    InfrastructureMachine
    KubeadmCon
    fi
    g
    Spec.clusterName
    Spec.controlPlaneRef Spec.InfrastructureRef
    

    View Slide

20. ୹୊:https://cluster-api.sigs.k8s.io/developer/architecture/controllers.html
    Cluster API의 이해 - contollers/CRs
    worker
    Reference
    Management
    Core
    
    
    controller
    master
    Controlplane
    
    
    controller
    Bootstrap
    
    
    controller
    infrastructure
    
    
    controller
    reconcile() reconcile()
    Cluster
    Kubeadm
    
    
    ControlPlane
    InfrastructureCluster
    kubeadm
    
    
    con
    fi
    gtemplate
    Infrastructure
    
    
    machinetemplate
    Machine
    
    
    Deployment
    Infrastructure
    
    
    machinetemplate
    Machine
    InfrastructureMachine
    machineset
    KubeadmCon
    fi
    g
    Machine
    InfrastructureMachine
    KubeadmCon
    fi
    g
    Spec.clusterName
    Spec.controlPlaneRef Spec.InfrastructureRef
    reconcile() reconcile()
    

    View Slide

21. ୹୊:https://cluster-api.sigs.k8s.io/developer/architecture/controllers.html
    Cluster API의 이해 - contollers/CRs
    worker
    Reference
    Management
    Core
    
    
    controller
    master
    Controlplane
    
    
    controller
    Bootstrap
    
    
    controller
    infrastructure
    
    
    controller
    reconcile()
    Cluster
    Kubeadm
    
    
    ControlPlane
    InfrastructureCluster
    kubeadm
    
    
    con
    fi
    gtemplate
    Infrastructure
    
    
    machinetemplate
    Machine
    
    
    Deployment
    Infrastructure
    
    
    machinetemplate
    Machine
    InfrastructureMachine
    machineset
    KubeadmCon
    fi
    g
    Machine
    InfrastructureMachine
    KubeadmCon
    fi
    g
    Spec.clusterName
    

    View Slide

22. ୹୊:https://cluster-api.sigs.k8s.io/developer/architecture/controllers.html
    Cluster API의 이해 - contollers/CRs
    worker
    Reference
    Management
    Core
    
    
    controller
    master
    Controlplane
    
    
    controller
    Bootstrap
    
    
    controller
    infrastructure
    
    
    controller
    reconcile()
    Cluster
    Kubeadm
    
    
    ControlPlane
    InfrastructureCluster
    kubeadm
    
    
    con
    fi
    gtemplate
    Infrastructure
    
    
    machinetemplate
    Machine
    
    
    Deployment
    Infrastructure
    
    
    machinetemplate
    Machine
    InfrastructureMachine
    machineset
    KubeadmCon
    fi
    g
    Machine
    InfrastructureMachine
    KubeadmCon
    fi
    g
    Spec.clusterName
    reconcile() reconcile()
    

    View Slide

23. ୹୊:https://cluster-api.sigs.k8s.io/developer/crd-relationships.html
    Cluster API의 이해 - contollers/CRs
    master worker
    Reference
    Management
    Infra provision
    bootstrap / init / join
    cluster provision
    Kubeadm
    
    
    ControlPlane
    InfrastructureCluster
    kubeadm
    
    
    con
    fi
    gtemplate
    Infrastructure
    
    
    machinetemplate
    Machine
    
    
    Deployment
    Infrastructure
    
    
    machinetemplate
    Machine
    InfrastructureMachine
    machineset
    KubeadmCon
    fi
    g
    Machine
    InfrastructureMachine
    KubeadmCon
    fi
    g
    Cluster
    

    View Slide

24. provider openstack
    IaaS resource control
    다양한 infrastructure provider 지원
    provider metal3
    IDC resource control
    Cluster API의 이해 - Infrastructure provider
    

    View Slide

25. 쿨하게!
    
    
    서비스
    

    View Slide

26. 어떻게 서비스화 할까?
    Kubernetes Cluster API
    Provider Openstack
    Provider Metal3
    OpenAPI
    Rest API
    
    
    CRUD/Auth
    Cluster provision engine
    
    
    Cluster control
    Infra Provider
    
    
    Infra resource control
    

    View Slide

27. 서비스 Diagram - Underground
    Dashboard
    Ring0
    Cluster API
    Provider Metal3
    Undercloud
    CSP Admin /
    
    
    SVC Developer
    CMDB
    Shared
    Rest API
    Provisioning
    request create CR
    access
    baremetal
    Network metadata
    

    View Slide

28. ୶о ੗ܐ: ഒ੗ࢲ ಌ࠶ܼ ௿ۄ਋٘ ੹୓ܳ ߓನೞח ߑߨ - https://if.kakao.com/session/81
    서비스 Diagram - Underground
    Con
    fi
    dential
    

    View Slide

29. 서비스 Diagram - IKE
    Cluster API
    Provider Openstack
    Undercloud
    Cloud User
    KIC Console
    Shared
    request
    create CR
    access
    IKE Mgmt
    IKE Rest API
    Overcloud
    VPC VPC
    VPC
    Provisioning
    kind: Service
    
    
    apiVersion: v1
    
    
    metadata:
    
    
    ...
    
    
    spec:
    
    
    selector:
    
    
    app: web
    
    
    type: LoadBalancer
    

    View Slide

30. 서비스 Diagram - IKE
    Cloud User Shared
    k8s api access
    IKE Mgmt
    VPC VPC
    VPC
    Provisioning
    Control plan VPC
    service access
    Undercloud
    Overcloud
    

    View Slide

31. 섹시하게!
    
    
    경험
    

    View Slide

32. 개발시 주의해야 할 것들
    

    View Slide

33. 개발시 주의해야 할 것들
    

    View Slide

34. 개발시 주의해야 할 것들
    

    View Slide

35. 개발시 주의해야 할 것들
    

    View Slide

36. 개발시 주의해야 할 것들
    

    View Slide

37. 개발시 주의해야 할 것들
    

    View Slide

38. 개발시 주의해야 할 것들
    

    View Slide

39. 개발시 주의해야 할 것들
    

    View Slide

40. 개발시 주의해야 할 것들
    

    View Slide

41. 개발시 주의해야 할 것들
    

    View Slide

42. 개발시 주의해야 할 것들
    OpenAPI
    

    View Slide

43. 개발시 주의해야 할 것들
    OpenAPI
    

    View Slide

44. 성숙한 k8s 만들기
    Cluster api 성능 향상
    Concurrency
    
    
    Watch
    
    
    ExponentialBackoff
    
    
    Controller per Namespace
    

    View Slide

45. 성숙한 k8s 만들기
    Upgrade
    

    View Slide

46. 성숙한 k8s 만들기
    Upgrade CRD
    Upgrade
    IKE BE Upgrade Diagram
    

    View Slide

47. 성숙한 k8s 만들기
    Upgrade
    Undercloud
    Cloud User Shared IKE Mgmt
    Overcloud
    2. Upgrade MachineDeployments
    Control plan VPC
    upgrade
    request
    1. Upgrade Control Plane
    User VPC
    IKE Rest API IKE Controller CAPI CAPO
    Console
    

    View Slide

48. 성숙한 k8s 만들기
    IKE 사용 데모
    

    View Slide

49. 성숙한 k8s 만들기
    Kubernetes CIS benchmark
    

    View Slide

50. 성숙한 k8s 만들기
    KIC CSAP 인증 획득
    ୹୊ : http://www.itdaily.kr/news/articleView.html?idxno=208856
    

    View Slide

51. 섹시하게!
    
    
    계획
    

    View Slide

52. 앞으로의 계획 - Underground
    Namespace 기반 soft multi
    -
    tanancy 적용
    Namespace User
    Namespace
    Namespace
    X
    

    View Slide

53. 앞으로의 계획 - Underground
    Metadata sync operator 개발
    Ring0
    Cluster API
    Provider Metal3
    CMDB Network metadata
    Sync operator
    

    View Slide

54. 앞으로의 계획 - IKE
    IKE GPU cluster
    IKE Mgmt
    IKE Rest API
    VPC
    Provisioning
    Cluster API
    Provider Openstack
    VPC
    VPC
    GPU VM GPU VM GPU VM
    

    View Slide

55. 앞으로의 계획
    trivy
    
    
    kube
    -
    hunter
    
    
    sigstore
    
    
    vaccine
    ୹୊ : https://blog.sigstore.dev/kubernetes-signals-massive-adoption-of-sigstore-for-protecting-open-source-ecosystem-73a6757da73
    Supplychain security강화
    

    View Slide

56. 앞으로의 계획 - IKE
    terraform IKE plugin
    
    
    Client 다양성 확보
    

    View Slide

57. Fun, Cool, Sexy
    “이것이 KIC의 방식이니까 음!”
    

    View Slide

58. Q&A
    

    View Slide