Upgrade to Pro — share decks privately, control downloads, hide ads and more …

안정성과 유연성을 겸비한 카카오뱅크의 On-premise Kubernetes 구축 여정기

kakao
November 01, 2024

안정성과 유연성을 겸비한 카카오뱅크의 On-premise Kubernetes 구축 여정기

#인프라 #리소스최적화 #온프렘 #쿠버네티스 #아키텍처 #망분리

은행의 세밀한 내부 통제 환경에서 인프라 리소스를 최적화하고, 효율적으로 관리하기 위해 On-premise Kubernetes 아키텍처를 구축했습니다. 클러스터 프로비저닝 과정부터 망 분리 환경에서의 네트워크 통제 및 운영 정책 관리 방법을 소개합니다.

발표자 : robin.hood
카카오뱅크에서 On-Premise Kubernetes 엔지니어링을 담당하고 있는 Robin 입니다.

kakao

November 01, 2024
Tweet

More Decks by kakao

Other Decks in Programming

Transcript

  1. $1 ࢸ੿౵ੌ ష௾ 8PSLFS/PEF 7. 7. 7. ֢٘ࢸ੿౵ੌ ֢٘ࢸ੿౵ੌ ֢٘ࢸ੿౵ੌ

    $1/PEF 7. 7. ֢٘ࢸ੿౵ੌ ֢٘ࢸ੿౵ੌ ֢٘ࢸ੿౵ੌ ஠ߛীࢲ0O1SFN,4ܳ݅٘۰ݶ
  2. .VMUJQMF.BDIJOF1SPWJEFS $MVTUFS"1* #SJOH:PVS0XO)PTU #:0) $MPVE4UBDL $PY&EHF %JHJUBM0DFBO &RVJOJY.FUBM GPSNFSMZ1BDLFU )FU[OFS

    *#.$MPVE ,VCF,FZ ,VCF7JSU .""4 .FUBM .JDSPWN /FTUFE 0VUTDBMF WDMVTUFS FUD ׮নೠ1SPWJEFSоҳഅغয੓חੋఠಕ੉झয়೑ࣗझ *OUFSGBDF *NQMFNFOUT ௿۞झఠٜ݅যભ "NB[PO &,4 "NB[PO &$
  3. apiVersion: cluster.x-k8s.io/v1beta1 kind: Machine spec: bootstrap: configRef: apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 kind:

    KubeadmConfig name: ++++++++ namespace: ++++++++ dataSecretName: ++++++++ clusterName: ++++++++ infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: MaasMachine name: ++++++++ namespace: ++++++++ providerID: ++++++++ version: v1.28.2 .VMUJQMF.BDIJOF1SPWJEFS
  4. apiVersion: cluster.x-k8s.io/v1beta1 kind: Machine spec: bootstrap: configRef: apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 kind:

    KubeadmConfig name: ++++++++ namespace: ++++++++ dataSecretName: ++++++++ clusterName: ++++++++ infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: AWSMachine name: ++++++++ namespace: ++++++++ providerID: ++++++++ version: v1.28.2 .VMUJQMF.BDIJOF1SPWJEFS "NB[PO &$
  5. $MVTUFS"1* apiVersion: controlplane.cluster.x-k8s.io/v1beta1 kind: KubeadmConfigTemplate spec: template: spec: kubeadmConfigSpec: preKubeadmCommands:

    - # Network Setting - # HostOS Setting - # etc… postKubeadmCommands: - # Security, Logging ೐۽࠺੷׬઺ী୶о۽೙ਃೠ੘স੿੄
  6. )BSEXBSF*OUFHSBUJPO %/4 (4-# DPOUSPMMFS BQQ FYUFSOBMEOT /FUXPSL 3FRVFTU DMVTUFSNZDMVTUFS OBNFTQBDFT

    EFGBVMU  TFSWJDFNZTFSWJDF JQ HTMC  EPNBJONZ TFSWJDFFYBNQMFJP QPSU NPOJUPS TFSWJDFUZQF)5514 NFUIPE(&5 VSJ SFTQPOT ;POF'PSXBSE HTMCFYBNQMFJP .(.5$MVTUFS ҙܻ௿۞झఠ 8PSL$MVTUFS ࢲ࠺झ௿۞झఠ values.yaml
  7. DMVTUFSNZDMVTUFS OBNFTQBDFT EFGBVMU  TFSWJDFNZTFSWJDF JQ HTMC  EPNBJONZ TFSWJDFLBCBOHJP

    QPSU NPOJUPS TFSWJDFUZQF)5514 NFUIPE(&5 VSJ SFTQPOT )BSEXBSF*OUFHSBUJPO %/4 (4-# DPOUSPMMFS BQQ FYUFSOBMEOT ;POF'PSXBSE HTMCFYBNQMFJP .(.5$MVTUFS ҙܻ௿۞झఠ 8PSL$MVTUFS ࢲ࠺झ௿۞झఠ values.yaml /FUXPSL 3FRVFTU NZTFSWJDFী *1ܳ ࠗৈೞҊ NZTFSWJDFFYBNQMFJP )PTU۽੽ࣘೡࣻ੓ѱٜ݅য઱ࣁਃ
  8. )BSEXBSF*OUFHSBUJPO %/4 (4-# DPOUSPMMFS BQQ FYUFSOBMEOT $/".&3FDPSE $36% 7*1 "3FDPSE

    $36% )FBMUI $IFDL SFDPODJMF SFDPODJMF ;POF'PSXBSE HTMCFYBNQMFJP /FUXPSL 3FRVFTU
  9. ৵੗ਬ܂ѱޅ ॳחѢભ Әਲ਼׏࠺झ஠ೖ ੹੗Әਲ਼Ѣېߨ ੹੗Әਲ਼хةӏ੿߂ ղӏܳݽف૑ః࣊ঠ೤פ׮ § ੿ࠁ୊ܻदझమ੽Ӕী؀ೠ੽Ӕઁযӏ஗ਸࣻ݀ೞҊ਍৔೧ঠೠ׮ § ੿ࠁ୊ܻदझమ੄੽ӔӂೠࠗৈחथੋѾ੤ܳٙೠറೲоػ؀࢚੗ীೠೞৈӂೠࠗৈоغযঠೠ׮

    § ࢎਊ੗੽ӔӂೠथੋѾ੤೐۽ࣁझࣻ݀ § ѐߊ਍৔ߓನ੺ରળࣻ § ਍৔ߓನӂೠాઁ § ਍৔ߓನ੹ஂড੼੼Ѩ૓೯ § ֎౟ਕ௼੽Ӕઁযӏ஗ਸࣻ݀ೞҊ਍৔೧ঠೠ׮ § ֎౟ਕ௼੄੿଼߸҃਷थੋѾ੤ܳٙೠറೲоػ؀࢚੗ীೠೞৈ੿଼੸ਊ੉غযঠೠ׮ § ੗୓ҳ୷ 1SJWBUF ௿ۄ਋٘ࠁউஂড੼ಣо ӝળ Әਲ਼ஹ೒ۄ੉঱झ৬ࠁউ хࢎоف۰਍ભܰ٣
  10. ஶప੉ցীࢲഐझ౟दझమী੽Ӕೡࣻ੓חࠁউਤഈ੉ߊࢤؼࣻ੓ਵ޲۽  QSJWJMFHFE೒ېӒઁѢৈࠗܳ੼Ѩ Әਲ਼ࠁউਗ apiVersion: kyverno.io/v1 kind: ClusterPolicy # ...

    spec: validate: # ... containers: - =(securityContext): =(privileged): "false" =(hostPID): "false" =(hostIPC): "false" =(hostNetwork): "false"
  11. Әਲ਼ஹ೒ۄ੉঱झ৬ࠁউ ӝמ ,VCFSOFUFT/FUXPSL1PMJDZ $JMJVN/FUXPSL1PMJDZ ੋ߄਍٘౟ې೗ઁয 0 0 ইਓ߄਍٘౟ې೗ઁয 0 0

    ۄ߰ӝ߈౟ې೗ઁয 0 0 ֎੐झಕ੉झӝ߈ઁয 0 0 -BZFS੿଼ - ઁয 9 0 $*%3ߧਤ૑੿ ¦ 0 %/4੿଼ 9 0 ੿଼౟ې೗۽Ӧ 9 0 ੿଼хࢎݽ٘ "VEJU.PEF 9 0
  12. 8PSLMPBE %FQMPZ %FWFMPQFS 0QFSBUPS ,ZWFSOP .VUBUF -BCFM kind: Deployment #

    … template: metadata: labels: app: my-app kind: Deployment # … template: metadata: labels: app: my-app example/cluster-name: my-cluster example/namespace: sandbox example/workload-name: nginx example/workload-type: dp
  13. 8PSLMPBE %FQMPZ %FWFMPQFS 0QFSBUPS ,ZWFSOP /FUXPSL 4FDVSJUZ &OHJOFFS .VUBUF -BCFM

    $JMJVN/FUXPSL1PMJDZ "QQMZ kind: Deployment # … template: metadata: labels: app: my-app kind: Deployment # … template: metadata: labels: app: my-app example/cluster-name: my-cluster example/namespace: sandbox example/workload-name: nginx example/workload-type: dp kind: CiliumNetworkPolicy spec: egress: - toCIDRSet: - cidr: 1.2.3.4/32 toPorts: - ports: - port: "80" protocol: TCP endpointSelector: matchLabels: example/cluster-name: my-cluster example/namespace: sandbox example/workload-name: nginx example/workload-type: dp
  14. 8PSLMPBE %FQMPZ %FWFMPQFS 0QFSBUPS ,ZWFSOP /FUXPSL 4FDVSJUZ &OHJOFFS .VUBUF -BCFM

    $JMJVN/FUXPSL1PMJDZ "QQMZ kind: Deployment # … template: metadata: labels: app: my-app kind: Deployment # … template: metadata: labels: app: my-app example/cluster-name: my-cluster example/namespace: sandbox example/workload-name: nginx example/workload-type: dp kind: CiliumNetworkPolicy spec: egress: - toCIDRSet: - cidr: 1.2.3.4/32 toPorts: - ports: - port: "80" protocol: TCP endpointSelector: matchLabels: example/cluster-name: my-cluster example/namespace: sandbox example/workload-name: nginx example/workload-type: dp ୶࢚ചػ,4-BZFSӝ߈੄֎౟ਕ௼੿଼ tNZDMVTUFS௿۞झఠ੄ TBOECPY֎੐झಕ੉झী੓ח OHJOYEFQMPZNFOUী؀೧ࢲu &HSFTT౟ې೗ tu5$1ాनਸೲਊ
  15. Әਲ਼ஹ೒ۄ੉঱झ৬ࠁউ apiVersion: controlplane.cluster.x-k8s.io/v1beta1 kind: KubeadmControlPlane ... spec: kubeadmConfigSpec: ... initConfiguration:

    nodeRegistration: kubeletExtraArgs: read-only-port: "0" ... clusterConfiguration: extraArgs: audit-log-path: /var/log/kubernetes/audit/audit.log audit-policy-file: /etc/kubernetes/audit-policy.yaml audit-log-maxbackup: "10" audit-log-maxsize: "100" audit-log-maxage: "30" ,4ࠁউಁ஖੘স XJUI$"1* ª/PEF3FDSFBUF
  16. Әਲ਼ஹ೒ۄ੉঱झ৬ࠁউ )JLVCF 404न୒ $MVTUFSCJ[UFTU /BNFTQBDF$PDP 6TFSDIVOTJL 3PMFSFBEPOMZ )JLVCF 404न୒ $MVTUFSCJ[UFTU

    /BNFTQBDF$PDP 6TFS+PSEZ 3PMFTSFVTFSSPMF SFBEPOMZ TSF SFBEPOMZ TSFVTFSSPMF IJLVCF दझమࠁউ౱
  17. Әਲ਼ஹ೒ۄ੉঱झ৬ࠁউ )JLVCF 404न୒ $MVTUFSCJ[UFTU /BNFTQBDF$PDP 6TFSDIVOTJL 3PMFSFBEPOMZ )JLVCF 404न୒ $MVTUFSCJ[UFTU

    /BNFTQBDF$PDP 6TFS+PSEZ 3PMFTSFVTFSSPMF SFBEPOMZ TSFVTFSSPMF SFBEPOMZ TSF ӂೠহ਺ ࢤࢿपಁ IJLVCF दझమࠁউ౱
  18. Әਲ਼ஹ೒ۄ੉঱झ৬ࠁউ )JLVCF 404न୒ $MVTUFSCJ[UFTU /BNFTQBDF$PDP 6TFSDIVOTJL 3PMFSFBEPOMZ )JLVCF 404न୒ $MVTUFSCJ[UFTU

    /BNFTQBDF$PDP 6TFS+PSEZ 3PMFTSFVTFSSPMF SFBEPOMZ TSFVTFSSPMF SFBEPOMZ TSF ࢤࢿೲਊ IJLVCF दझమࠁউ౱
  19. 2"

  20. Q Q Q Q Q Q Q Q ,VCFSOFUFT-PHPIUUQTHJUIVCDPNLVCFSOFUFTLVCFSOFUFTCMPCNBTUFS-PHP-PHP@XJUI@CPSEFSQOH Q

    Q $MVTUFS"1* -PHPIUUQTHJUIVCDPNLVCFSOFUFTTJHTDMVTUFSBQJUSFFNBJO-PHPT Q 0QFOTUBDL -PHPIUUQTPQFOJOGSBEFWCSBOE-PHPT Q Q $BOPOJDBMNBBT-PHPIUUQTEFTJHOVCVOUVDPNSFTPVSDFT Q "[VSF-PHPIUUQTFONXJLJQFEJBPSHXJLJ'JMF.JDSPTPGU@"[VSF@-PHPTWH Q (PPHMFDMPVE-PHPIUUQTFONXJLJQFEJBPSHXJLJ'JMF(PPHMF@$MPVE@-PHPTWH Q Q Q "84-PHPTIUUQTBXTBNB[PODPNLPBSDIJUFDUVSFJDPOT ଵҊޙ೴
  21. Q Q Q (JUMBC-PHPIUUQTBCPVUHJUMBCDPNQSFTTQSFTTLJU Q Q Q "SHP-PHPIUUQTHJUIVCDPNBSHPQSPKBSHPVJCMPCNBTUFSTSDBTTFUTJNBHFT-PHPQOH Q 1BDLFS-PHPIUUQTCSBOEIBTIJDPSQDPNQSPEVDU@-PHPT

    Q "OTJCMF-PHPIUUQTHJUIVCDPNBOTJCMF-PHPTUSFFNBJO Q Q Q ,VCFSOFUFTPCKFDU-PHPTIUUQTHJUIVCDPNLVCFSOFUFTDPNNVOJUZUSFFNBTUFSJDPOTQOHSFTPVSDFT ଵҊޙ೴