Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Yahoo! JAPAN の マネージド Kubernetes サービスを支える技術 / The Technology Behind Managed Kubernetes Service of Yahoo! JAPAN

Yahoo! JAPAN の マネージド Kubernetes サービスを支える技術 / The Technology Behind Managed Kubernetes Service of Yahoo! JAPAN

ヤフー名古屋 Tech Meetup #2 - https://yahoo-nagoya.connpass.com/event/129398/

大規模環境でのコンテナの利用において、Kubernetes がデファクトスタンダードになりました。 Yahoo! JAPAN は、2017年10月頃から一部のサービスが本番環境で Kubernetes の利用を開始し、現在では多くのサービスが利用しています。 その結果、Yahoo! JAPAN では数百の Kubernetes クラスタが運用されています。 本セッションでは、Kubernetes の概要から、数百の Kubernetes クラスタの運用を可能にする Yahoo! JAPAN のプライベートマネージド Kubernetes サービスの紹介とそれを支える技術についてお話します。

Kazuki Suda

May 28, 2019
Tweet

More Decks by Kazuki Suda

Other Decks in Technology

Transcript

  1. ،آؑٝت  ,VCFSOFUFTהכזחַ  َؿ٦ה,VCFSOFUFT  ז׈,VCFSOFUFTזךַ  ,VCFSOFUFTך،٦ؗذؙثٍהؔـآؙؑز 

    :BIPP+"1"/ךوط٦آس,VCFSOFUFT؟٦ؽأ׾佄ִ׷䪮遭  ״׶鑫׃ֻ濼׷חכ  תה׭
  2. acL 3 3A g A3 2 M dT ,VCFSOFUFT Ӝ

    䎃剢ח剑ⴱך؝ىحزָ֮׶ծ䎃剢ח׾ٔٔ٦أ 䎃剢儗挿ך剑倜غ٦آّٝכ Ӝ ⴱ劍כ(PPHMFךاؿزؐؑ،׌׏׋ָծ׉ך䖓
 $MPVE/BUJWF$PNQVUJOH'PVOEBUJPO$/$'ח隁床ׁ׸ծ
 植㖈כ$/$'ָمأزׅ׷ؔ٦فٝا٦أفٗآؙؑز Ӝ ؔٝفٖכ׮׍׹׿ծ⚺銲זؙٓؐسفٗغ؎تדوط٦آس؟٦ؽأָ䲿⣘ׁ׸גְ׷ (PPHMF,VCFSOFUFT&OHJOF "NB[PO&,4 "[VSF,VCFSOFUFT4FSJWJDFT Ӝ չ䎃⟃♳ח床׶؝ٝذش׾劤殢橆㞮ד麊欽׃גֹ׋(PPHMFך穗꿀ה
 㢳ֻך⟰噟ח״׷؝ىُصذ؍ך⮚׸׋،؎ر،ה䩛岀ָ穈׫鴥ת׸גְ׷պ
  3. ,VCFSOFUFTכؔ٦فٝד֮׷ Ӝ ؔ٦فٝا٦أاؿزؐؑ، HJUIVCDPNLVCFSOFUFTLVCFSOFUFT "QBDIF-JDFOTF  Ӝ ؔ٦فٝرؠ؎ٝ HJUIVCDPNLVCFSOFUFTDPNNVOJUZ Ӝ

    ؔ٦فٝ؝ىُصذ؍ $MPVE/BUJWF$PNQVUJOH'PVOEBUJPO 4QFDJBM*OUFSFTU(SPVQT 4*(T  Ӝ 4MBDL 4UBDL0WFSPX Ⰻ⚅歲ד⟃♳ךى٦ز،حف ,VCFSOFUFT4MBDLٙ٦ؙأل٦أTMBDLLTJPKQVTFST bF A03 3 3 )33 A A A03 m S dT
  4. ,VCFSOFUFTכ䧭ꞿ׃גְ׷ ,VCFSOFUFT %PDLFS4XBSN "QBDIF.FTPT Ⰻ窟鎘 $POUSJCVUPST    

    $PNNJUT       麓⿠٠剢 $POUSJCVUPST    $PNNJUT      https://www.openhub.net/p/_compare?project_0=Kubernetes&project_1=docker+swarm&project_2=Apache+Mesos 䎃剢傈儗挿
  5. apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: replicas: 3

    selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.13.3 ports: - containerPort: 80 ,VCFSOFUFTחֶֽ׷㹑鎉涸鏣㹀,VCFSOFUFTوصؿؑأز ٖفٔؕ侧 㹋遤ׁ׸גְ׷ץֹ؝ٝذش侧 㹋遤ׁ׸׷؝ٝذش؎ً٦آ OHJOYךغ٦آّٝ M l k M l dT
  6. ➭ךفٓحزؿؓ٦يהך嫰鯰 Ӝ *OGSBTUSVDUVSFBTB4FSWJDF Ӝ 7. ر؍أؙ طحزٙ٦ؙ ,VCFSOFUFT Ӝ $POUBJOFSBTB4FSWJDF

    Ӝ ؝ٝذشך盖椚ծ㹋遤 Ӝ ֮׵ײ׷،فٔ؛٦ءّٝ $MPVE'PVOESZ Ӝ 1MBUGPSNBTB4FSWJDF Ӝ ؝٦سַ׵رفٗ؎ Ӝ )551ծ"1*ծ8FC 0QFO4UBDL
  7. Ӝ ؝ٝذشד荈歋䏝ה䬄韋⻉׾غٓ ٝأ״ֻ㹋植ׅ׷ Ӝ 䬄韋䏝ָ넝ֻծ欰欵䚍ָ넝ְ Ӝ ♧倯ד荈歋䏝ָ⡚ְ 暟椚؎ٝؿٓ 04 ؟٦غ⟎䟝⻉

    ىسٕؐؑ، ٓٝة؎ي ،فٔ؛٦ءّٝ 暟椚؎ٝؿٓ 04 ؟٦غ⟎䟝⻉ ىسٕؐؑ، ٓٝة؎ي ،فٔ؛٦ءّٝ 暟椚؎ٝؿٓ 04 ؟٦غ⟎䟝⻉ ىسٕؐؑ، ٓٝة؎ي ،فٔ؛٦ءّٝ ؝ٝذش⻉ *BB4 ,VCFSOFUFT 1BB4 Ӝ 荈歋䏝כ넝ְָ盖椚ׅ׷ ׮ךָ㢳ְ 荈歋䏝 欰欵䚍 h
  8. 1PEهحس Ӝ 醱侧ך؝ٝذشה
 醱侧ךنُٔ٦ي Ӝ رفٗ؎ך剑㼭⽃⡘ Ӝ *1QFS1PE apiVersion: v1

    kind: Pod metadata: name: nginx spec: containers: - name: nginx image: nginx:1.15.8 ports: - containerPort: 80
  9. 1PEهحس Ӝ 醱侧ך؝ٝذشה
 醱侧ךنُٔ٦ي Ӝ رفٗ؎ך剑㼭⽃⡘ Ӝ *1QFS1PE 1PE" نُٔ٦ي

    ؝ٝذش ؝ٝذش ظ٦س ظ٦س *1 1PE# *1 1PE$ *1 1PEחろת׸׷؝ٝذشכ
 䗳׆ずׄظ٦س♳ד㹋遤ׁ׸׷ ぐ1PEכؿٓحزזطحزٙ٦ؙך*1،سٖأ׾䭯׍ծظ٦س׾ת׋ְד鸐⥋דֹ׷
  10. 3FQMJDB4FUٖفٔؕإحز Ӝ /⦐ך1PEָ㹋遤ׁ׸גְ׷朐䡾׾⥂א apiVersion: apps/v1 kind: ReplicaSet metadata: name: nginx

    spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.15.8 ports: - containerPort: 80 1PEذٝفٖ٦ز ⡲䧭ׁ׸׷1PEך׮ההז׷鏣㹀 ٖفٔؕ侧
  11. 1PE"" XFCW %FQMPZNFOUرفٗ؎ًٝز Ӝ ٗ٦ؚٔٝ،حفر٦ز Ӝ ٗ٦ٕغحؙ %FQMPZNFOU" DPOUBJOFST JNBHFXFCW

    ٖفٔؕ侧 3FQMJDB4FU" DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE"# XFCW
  12. 1PE"" XFCW %FQMPZNFOUرفٗ؎ًٝز Ӝ ٗ٦ؚٔٝ،حفر٦ز Ӝ ٗ٦ٕغحؙ %FQMPZNFOU" DPOUBJOFST JNBHFXFCW

    ٖفٔؕ侧 3FQMJDB4FU" DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE"# XFCW
  13. 1PE"" XFCW %FQMPZNFOUرفٗ؎ًٝز Ӝ ٗ٦ؚٔٝ،حفر٦ز Ӝ ٗ٦ٕغحؙ %FQMPZNFOU" DPOUBJOFST JNBHFXFCW

    ٖفٔؕ侧 3FQMJDB4FU" DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE"# XFCW 3FQMJDB4FU# DPOUBJOFST JNBHFXFCW ٖفٔؕ侧
  14. 1PE"" XFCW %FQMPZNFOUرفٗ؎ًٝز Ӝ ٗ٦ؚٔٝ،حفر٦ز Ӝ ٗ٦ٕغحؙ %FQMPZNFOU" DPOUBJOFST JNBHFXFCW

    ٖفٔؕ侧 3FQMJDB4FU" DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE"# XFCW 3FQMJDB4FU# DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE#" XFCW
  15. 1PE"" XFCW %FQMPZNFOUرفٗ؎ًٝز Ӝ ٗ٦ؚٔٝ،حفر٦ز Ӝ ٗ٦ٕغحؙ %FQMPZNFOU" DPOUBJOFST JNBHFXFCW

    ٖفٔؕ侧 3FQMJDB4FU" DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE"# XFCW 3FQMJDB4FU# DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE#" XFCW
  16. %FQMPZNFOUرفٗ؎ًٝز Ӝ ٗ٦ؚٔٝ،حفر٦ز Ӝ ٗ٦ٕغحؙ %FQMPZNFOU" DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 3FQMJDB4FU"

    DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE"# XFCW 3FQMJDB4FU# DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE#" XFCW
  17. %FQMPZNFOUرفٗ؎ًٝز Ӝ ٗ٦ؚٔٝ،حفر٦ز Ӝ ٗ٦ٕغحؙ %FQMPZNFOU" DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 3FQMJDB4FU"

    DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE"# XFCW 3FQMJDB4FU# DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE#" XFCW
  18. %FQMPZNFOUرفٗ؎ًٝز Ӝ ٗ٦ؚٔٝ،حفر٦ز Ӝ ٗ٦ٕغحؙ %FQMPZNFOU" DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 3FQMJDB4FU"

    DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE"# XFCW 3FQMJDB4FU# DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE#" XFCW 1PE## XFCW
  19. %FQMPZNFOUرفٗ؎ًٝز Ӝ ٗ٦ؚٔٝ،حفر٦ز Ӝ ٗ٦ٕغحؙ %FQMPZNFOU" DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 3FQMJDB4FU"

    DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 3FQMJDB4FU# DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE#" XFCW 1PE## XFCW
  20. %FQMPZNFOU Ӝ ٗ٦ؚٔٝ،حفر٦ز Ӝ ٗ٦ٕغحؙ apiVersion: apps/v1 kind: Deployment metadata:

    name: nginx spec: replicas: 2 selector: matchLabels: app: nginx strategy: type: RollingUpdate rollingUpdate: maxSurge: 25% maxUnavailable: 25% template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.15.8 ports: - containerPort: 80 رفٗ؎䨌殛 ٖفٔؕ侧 1PEذٝفٖ٦ز
  21. 4FSWJDF؟٦ؽأ Ӝ ⟎䟝*1הه٦ز Ӝ ٓكٕإؙٖةח״׷1PEךؚٕ٦ؾؚٝ Ӝ ؟٦ؽأة؎ف $MVTUFS*1 *OUFSOBM 

    /PEF1PSU -PBE#BMBODFS &YUFSOBM   4FSWJDFNZTWD ة؎ف$MVTUFS*1
 إؙٖةBQQOHJOY ه٦زUDQ̔UDQ DMVTUFS*1 1PE" 1PE# 3FQMJDB4FU BQQOHJOY BQQOHJOY
  22. 4FSWJDF؟٦ؽأ Ӝ ⟎䟝*1הه٦ز Ӝ ٓكٕإؙٖةח״׷1PEךؚٕ٦ؾؚٝ Ӝ ؟٦ؽأة؎ف $MVTUFS*1 *OUFSOBM 

    /PEF1PSU -PBE#BMBODFS &YUFSOBM   apiVersion: v1 kind: Service metadata: name: nginx spec: type: ClusterIP selector: app: nginx ports: - protocol: TCP port: 8080 targetPort: 80
  23. ׉ך➭ך⚺זؔـآؙؑز Ӝ /BNFTQBDFؙٓأة׾锷椚涸חⴓⶴׅ׷ Ӝ $POH.BQ،فٔ؛٦ءّٝה鏣㹀ךⴓꨄ Ӝ 4FDSFU،فٔ؛٦ءّٝהء٦ؙٖحزךⴓꨄ Ӝ 1FSTJTUFOU7PMVNF 1FSTJTUFOU7PMVNF$MBJN

    4UPSBHF$MBTT宕竲نُٔ٦ي Ӝ 4UBUFGVM4FUأذ٦زؿٕ،فٔ؛٦ءّٝ Ӝ +PCٙٝءّحزآّـ Ӝ $SPO+PCآّـך㹀劍㹋遤 Ӝ %BFNPO4FUⰋגךظ٦سד1PE׾㹋遤 Ӝ *OHSFTT)551頾蚚ⴓ侔ծغ٦ثٍٕمأزծ5-4穄畭 Ӝ )PMJ[POUBM1PE"VUP4DBMFS )1" ؔ٦زأ؛٦ٕ
  24. ז׈,VCFSOFUFTBTB4FSWJDFָ䗳銲זךַ 撕꧟ז,VCFSOFUFTךؔلٖ٦ءַّٝ׵麊欽罏׾鍑佝ׅ׷ Ӝ ؙٓأةך⡲䧭ծ⵴ꤐծ鏣㹀㢌刿 Ӝ ظ٦س 7. ך鷄⸇٥⵴ꤐ Ӝ ؙٓأةךئٗتؐٝة؎ي،حفؚٖ٦س

    Ӝ ꥺ㹱װ㉏겗ך֮׷ظ٦سך⥜䗁 Ӝ ر٦ةأز، FUDE ךغحؙ،حفծٔأز، ,VCFSOFUFTכ؝ٝذش⻉ׁ׸׋،فٔ؛٦ءّٝך麊欽ח㣐ֹזًٔحزָ֮׷♧倯דؙٓأةך 麊欽חכ㢳ֻךؔلٖ٦ءָّٝ䗳銲חז׷կ au a b M n L i b t T
  25. $VTUPN3FTPVSDF%FOJUJPOT $3% "1* $POUSPMMFS *OTUBODF 3FQMJDB4FUT %FQMPZNFOUT  3FQMJDB4FU$POUSPMMFS $VTUPN$POUSPMMFS

     %FQMPZNFOU$POUSPMMFS 3FQMJDB4FU %FQMPZNFOU $VTUPN3FTPVSDF $3% 
  26. ,VCFSOFUFT$MVTUFS0QFSBUPSך$3%ה؝ٝزٗ٦ٓ "1* $POUSPMMFS *OTUBODF ,VCFSOFUFT$MVTUFST ,VCFSOFUFTFT ,VCFSOFUFT$MVTUFS
 $POUSPMMFS ,VCFSOFUFT
 $POUSPMMFS

    ,VCFSOFUFT$MVTUFS ,VCFSOFUFT .BDIJOF4FUT .BDIJOFT .BDIJOF4FU
 $POUSPMMFS .BDIJOF
 $POUSPMMFS .BDIJOF4FUT .BDIJOF    cL f O na M dT
  27. apiVersion: apiextensions/k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: kubernetesclusters.kubernetes.zlab.co.jp spec: scope: Namespaced

    group: kubernetes.zlab.co.jp version: v1alpha1 names: plural: kubernetesClusters singular: kubernetesCluster kind: KubernetesCluster shortNames: - kc additionalPrinterColumns: - name: ready type: boolean description: ”Phase of KubernetesCluster” JSONPath: .status.ready ŋŋŋ
  28. $ cat cluster01.yaml apiVersion: kubernetes.zlab.co.jp/v1alpha1 kind: KubernetesCluster metadata: name: cluster01

    spec: version: ”1.14.3” masterReplicas: 3 masterFlavor: large workerReplicas: 5 workerFlavor: large ŋŋŋ $ kubectl apply -f cluster.yaml Kubernetescluster “cluster01” created
  29. ,VCFSOFUFT0QFSBUPSTJOUIF8JME Ӝ DPSFPTFUDEPQFSBUPS $SFBUFDPOHVSFNBOBHFFUDEDMVTUFSTBUPQ,VCFSOFUFT Ӝ DPSFPTQSPNFUIFVTPQFSBUPS $SFBUFDPOHVSFNBOBHF1SPNFUIFVTDMVTUFSTBUPQ,VCFSOFUFT Ӝ SPPLSPPL 4UPSBHF0SDIFTUSBUJPOGPS,VCFSOFUFT$FQI$PDLSPBDI%#

    Ӝ PSBDMFNZTRMPQFSBUPS $SFBUF PQFSBUFBOETDBMFTFMGIFBMJOH.Z42-DMVTUFSTJO,VCFSOFUFT Ӝ LVCFPXUGPQFSBUPS 5PPMTGPS.-5FOTPSPXPO,VCFSOFUFT https://github.com/operator-framework/awesome-operators
  30. ,VCFSOFUFT׾״׶鑫׃ֻ濼׷חכ Ӝ Ⱅ䒭سًُؗٝزLVCFSOFUFTJP Ӝ ּׅ㨣׭׷חכ ٗ٦ٕؕחꟚ涪欽ؙٓأة׾圓眠.JOJLVCF %PDLFS%FTLUPQGPS.BDBOE8JOEPXT NJDSPLT ػـٔحؙؙٓؐس⥋걾ה㹋籐ך(PPHMF,VCFSOFUFT&OHJOF Ӝ

    傈劤؝ىُصذ؍,VCFSOFUFT4MBDLثٍٝطٕTMBDLLTJPKQVTFST KQFWFOUT Ӝ ى٦ز،حف ,VCFSOFUFT.FFUVQ5PLZPIUUQTLTKQDPOOQBTTDPN $MPVE/BUJWF.FFUVQ5PLZPIUUQTDMPVEOBUJWFDPOOQBTTDPN Ӝ ⹛歗 ,VCFSOFUFT.FFUVQ5PLZP:PV5VCFثٍٝطٕ $/$':PV5VCFثٍٝطٕ
  31. acL 3 3A g A3 2 M dT ,VCFSOFUFT Ӝ

    䎃剢ח剑ⴱך؝ىحزָ֮׶ծ䎃剢ח׾ٔٔ٦أ 䎃剢儗挿ך剑倜غ٦آّٝכ Ӝ ⴱ劍כ(PPHMFךاؿزؐؑ،׌׏׋ָծ׉ך䖓
 $MPVE/BUJWF$PNQVUJOH'PVOEBUJPO$/$'ח隁床ׁ׸ծ
 植㖈כ$/$'ָمأزׅ׷ؔ٦فٝا٦أفٗآؙؑز Ӝ ؔٝفٖכ׮׍׹׿ծ⚺銲זؙٓؐسفٗغ؎تדوط٦آس؟٦ؽأָ䲿⣘ׁ׸גְ׷ (PPHMF,VCFSOFUFT&OHJOF "NB[PO&,4 "[VSF,VCFSOFUFT4FSJWJDFT Ӝ չ䎃⟃♳ח床׶؝ٝذش׾劤殢橆㞮ד麊欽׃גֹ׋(PPHMFך穗꿀ה
 㢳ֻך⟰噟ח״׷؝ىُصذ؍ך⮚׸׋،؎ر،ה䩛岀ָ穈׫鴥ת׸גְ׷պ