Yahoo! JAPAN のマネージド Kubernetes サービスを支える技術 / The Technology Behind Managed Kubernetes Service of Yahoo! JAPAN

َؿ٦せ〢㾊5FDI.FFUVQ ,B[VLJ4VEBLTVEB![MBCDPKQ !TVQFSCSPUIFST :BIPP+"1"/ך وط٦آس,VCFSOFUFT؟٦ؽأ׾佄ִ׷䪮遭

,B[VLJ4VEB!TVQFSCSPUIFST Ӝ ئحزٓن吳䒭⠓爡اؿزؐؑ،ؒٝآص، Ӝ 䎃חَؿ٦吳䒭⠓爡ח倜⼼Ⰵ爡ծ痥➿랲䌒։,VCFSOFUFT։ Ӝ 䎃ئحزٓن吳䒭⠓爡ח⳿ぢ Ӝ ,VCFSOFUFT.FFUVQ5PLZP $MPVE/BUJWF%FFQ%JWFⰟず⚺⪵罏
Ӝ $/$'"NCBTTBEPS Ӝ չ,VCFSOFUFT㹋騧ⰅꟌպծչ׫׿זך%PDLFS,VCFSOFUFTպⰟ衼剅 Ӝ չⰅꟌ1SPNFUIFVTպ湊鏬剅

ئحزٓن吳䒭⠓爡;-BC$PSQPSBUJPO Ӝ 䎃ח鏣甧ׁ׸׋َؿ٦吳䒭⠓爡ך㶨⠓爡 Ӝ ؎ٝؿٓ㛇湍䪮遭ך锃叨٥灇瑔Ꟛ涪 Ӝ َؿ٦吳䒭⠓爡ぢֽךوط٦آس,VCFSOFUFT؟٦ؽأךꟚ涪 Ӝ IUUQT[MBCDPKQ

،آؑٝت ,VCFSOFUFTהכזחַ َؿ٦ה,VCFSOFUFT ז׈,VCFSOFUFTזךַ ,VCFSOFUFTך،٦ؗذؙثٍהؔـآؙؑز
:BIPP+"1"/ךوط٦آس,VCFSOFUFT؟٦ؽأ׾佄ִ׷䪮遭 ״׶鑫׃ֻ濼׷חכ תה׭

,VCFSOFUFTהכזחַ

,VCFSOFUFT Ӝ ؝ٝذشؔ٦؛أزٖ٦ءّٝخ٦ٕ 醱侧ךوءٝظ٦سד圓䧭ׁ׸׷ؙٓأةח㼎׃ג  ؝ٝذش،فٔ؛٦ءّٝךꂁ⪒ծ鏣㹀ծ盖椚׾遤ֲ Ӝ κυβερνήτηςؘٔءٍ铂ד乼菭㡦 Ӝ (PPHMFך爡ⰻءأذيַ׵؎ٝأػ؎،ׁ׸׋ b
a xm M kLf “Kubernetes is open source—a contrast to Borg and Omega,  which were developed as purely Google-internal systems. “ Borg, Omega, and Kubernetes

acL 3 3A g A3 2 M dT ,VCFSOFUFT Ӝ
䎃剢ח剑ⴱך؝ىحزָ֮׶ծ䎃剢ח׾ٔٔ٦أ 䎃剢儗挿ך剑倜غ٦آّٝכ Ӝ ⴱ劍כ(PPHMFךاؿزؐؑ،׌׏׋ָծ׉ך䖓  $MPVE/BUJWF$PNQVUJOH'PVOEBUJPO$/$'ח隁床ׁ׸ծ  植㖈כ$/$'ָمأزׅ׷ؔ٦فٝا٦أفٗآؙؑز Ӝ ؔٝفٖכ׮׍׹׿ծ⚺銲זؙٓؐسفٗغ؎تדوط٦آس؟٦ؽأָ䲿⣘ׁ׸גְ׷ (PPHMF,VCFSOFUFT&OHJOF "NB[PO&,4 "[VSF,VCFSOFUFT4FSJWJDFT Ӝ չ䎃⟃♳ח床׶؝ٝذش׾劤殢橆㞮ד麊欽׃גֹ׋(PPHMFך穗꿀ה  㢳ֻך⟰噟ח״׷؝ىُصذ؍ך⮚׸׋،؎ر،ה䩛岀ָ穈׫鴥ת׸גְ׷պ

https://twitter.com/brendandburns/status/585479466648018944 psp FP b KT u o Fu

َؿ٦ה,VCFSOFUFT

َؿ٦ה,VCFSOFUFT 䎃剢갦ַ׵♧鿇؟٦ؽأך劤殢橆㞮ה׃ג,VCFSOFUFTךⵃ欽׾Ꟛ㨣կ植㖈דכծ㢳ֻך ؟٦ؽأדⵃ欽׃גְ׷կ Ӝ ر٦ةإٝة Ӝ ؟٦غ Ӝ
0QFO4UBDLؙٓأة Ӝ ,VCFSOFUFTؙٓأة ˟䎃剢植㖈 n

َؿ٦ך,VCFSOFUFTBTB4FSWJDF َؿ٦吳䒭⠓爡ぢֽחئحزٓنָꟚ涪ׅ׷وط٦آس,VCFSOFUFT؟٦ؽأկ  ؔٝفٖىأך0QFO4UBDL橆㞮ח,VCFSOFUFTؙٓأة׾⡲䧭ծ盖椚ׅ׷կ䎃剢갦ַ׵Ꟛ涪 ׾㨣׭ծَؿ٦ך,VCFSOFUFTךⵃ欽׾佄ִ׷կ Ӝ إٕؿ؟٦ؽأꟚ涪罏ְָאד׮㥨ֹזהֹח Ӝ وط٦آس孡ָ⸬ֹתׅ Ӝ أ؛٦ٓـٕ㼭ְׁ׮ךַ׵㣐ֹז׮ךתד
Ӝ ءؚٕٝذشٝز杝׶⽑׭ m SdT

ז׈,VCFSOFUFTזךַ cL f ka

,VCFSOFUFTכؔ٦فٝד֮׷ Ӝ ؔ٦فٝا٦أاؿزؐؑ، HJUIVCDPNLVCFSOFUFTLVCFSOFUFT "QBDIF-JDFOTF Ӝ ؔ٦فٝرؠ؎ٝ HJUIVCDPNLVCFSOFUFTDPNNVOJUZ Ӝ
ؔ٦فٝ؝ىُصذ؍ $MPVE/BUJWF$PNQVUJOH'PVOEBUJPO 4QFDJBM*OUFSFTU(SPVQT 4*(T Ӝ 4MBDL 4UBDL0WFSPX Ⰻ⚅歲ד⟃♳ךى٦ز،حف ,VCFSOFUFT4MBDLٙ٦ؙأل٦أTMBDLLTJPKQVTFST bF A03 3 3 )33 A A A03 m S dT

,VCFSOFUFTכه٦ةـٕד֮׷ Ӝ ٗ٦سغٓٝ؟ծأزٖ٦آזוך*5؎ٝؿٓך乼⡲׾䬄韋⻉ DMPVEDPOUSPMMFSNBOBHFSؙٓؐسך乼⡲ ($1ծ"84ծ"[VSFծ0QFO4UBDLזו Ӝ ؝ٝذشطحزٙ٦ؙך垥彊⻉ $/*$POUBJOFS/FUXPSL*OUFSGBDF $BMJDPծ'MBOOFMծ8FBWF/FUזו Ӝ
؝ٝذش乼⡲ך垥彊⻉ $3*$POUBJOFS3VOUJNF*OUFSGBDF %PDLFSDPOUBJOFSEծDSJPծSLU t f uu f Kk

,VCFSOFUFTכ䧭ꞿ׃גְ׷ ,VCFSOFUFT %PDLFS4XBSN "QBDIF.FTPT Ⰻ窟鎘 $POUSJCVUPST
$PNNJUT 麓⿠٠剢 $POUSJCVUPST $PNNJUT https://www.openhub.net/p/_compare?project_0=Kubernetes&project_1=docker+swarm&project_2=Apache+Mesos 䎃剢傈儗挿

b a Y dT https://www.ﬂickr.com/photos/143247548@N03/46985430585/in/album-72157707188120301/ KubeCon + CloudNativeCon Europe 2019
Barcelona, Spain (May 20 - 23, 2019)

ז׈,VCFSOFUFTזךַ Ӝ ,VCFSOFUFTכؔ٦فٝד֮׷ Ӝ ,VCFSOFUFTכه٦ةـٕד֮׷ Ӝ ,VCFSOFUFTכ䧭ꞿ׃גְ׷ t F uu
F t f K F oM k

ז׈,VCFSOFUFTזךַ

ז׈,VCFSOFUFTזךַ מ׿ם׿זرفٗ؎ח罣ִ׵׸׷ꬊ䌢ח⮚׸׋堣腉׾䭯א Ӝ ֿ׸תדךꟚ涪כծ،فٔ؛٦ءّٝךرفٗ؎ח侧傈ծ侧鹈꟦ַַ׷ֿה׮ Ӝ 儗꟦ךַַ׷رفٗ؎כծ㉏겗ָ֮׏׋הֹך⥜姻׮鹼ֻז׷ Ӝ 鷞חְֲהמ׿ם׿זرفٗ؎כ㉏겗׾ּׅח⥜姻דֹծٔأؙ׾《׸׷ אמהאךرفٗ؎ָ鯪ֻז׶نُٔ٦يծٔأؙ ꟦麩ְַ׵鴼鸞ח䗁傊דֹ׷״ֲחז׶
״׶㢳ֻךٔأؙ׾ה׷ֿהָדֹ׷ S m L bgO PilkLM ea Y k

מ׿ם׿זرفٗ؎ח罣ִ׵׸׷ꬊ䌢ח⮚׸׋堣腉 Ӝ מ׿ם׿זرفٗ؎׾〳腉חׅ׷չ㹑鎉涸鏣㹀պ Ӝ 䓼⸂ז荈䊹㔐䗁堣腉չإٕؿؼ٦ؚٔٝպ Ӝ 7.⟎䟝وءٝ⚥䗰דכזֻչ؝ٝذش⚥䗰պך؎ٝؿٓ

מ׿ם׿זرفٗ؎׾〳腉חׅ׷չ㹑鎉涸鏣㹀պ ֿ׸תדך"׾㹋遤׃גַ׵#׾㹋遤ׅ׷״ֲז♧鸬ך䩛竲ֹח״׷رفٗ؎כծぐ䩛竲ָֹ⟃⵸ ך朐䡾ח⣛㶷ׅ׷կ Ӝ ♧鿇ךمأزָ✲⵸ח㢌刿ׁ׸גְג׮رفٗ؎ח䧭⸆ׅ׷ Ӝ ،فٔ؛٦ءّٝח㉏겗ָ֮׏׋הֹח然㹋חٗ٦ٕغحؙדֹ׷

♧鸬ך䩛竲ֹח״׷رفٗ؎ B A C ⋆ ♥ 乼⡲ך갫䎷ָ㢌׻׏ג׮䗳׆$ך朐䡾תד׋ו׶滠ֽ׷ "תד姻然חٗ٦ٕغحؙׅ׷חכ "ַ׵#ך㢌刿 #ַ׵$ך㢌刿

㹑鎉涸鏣㹀ח״׷رفٗ؎ رفٗ؎׃׋穠卓ך،فٔ؛٦ءّٝךչ劄ת׃ְ朐䡾պ׾鎸鶢׃׋鏣㹀ؿ؋؎ٕ׾⿾僥ׅ׷կ Ӝ ✲⵸חו׿ז㢌刿ָ⸇ִ׵׸גְג׮ծئַٗ׵ך圓眠ד֮׏ג׮䗳׆鏣㹀ؿ؋؎ٕח鎸鶢ׁ׸ ׋չ劄ת׃ְ朐䡾պחז׷ Ӝ ٗ٦ٕغحؙכ⟃⵸ך䨱׃׋ְ儗挿דך鏣㹀ؿ؋؎ٕ׾⿾僥ׅ׷׌ֽחז׷ 3 f T

㹑鎉涸鏣㹀ח״׷رفٗ؎ A C C 劄ת׃ְ朐䡾ָ鎸鶢ׁ׸׋鏣㹀ؿ؋؎ٕ "ַ׵$חז׷״ֲח荈䖒涸ח⹛⡲ׅ׷

㹑鎉涸鏣㹀ח״׷ٗ٦ٕغحؙ C A A ٗ٦ٕغحؙכ⟃⵸ך朐䡾ך鏣㹀ؿ؋؎ٕ׾⿾僥ׅ׷׌ֽ $ַ׵"חז׷״ֲח荈䖒涸ח⹛⡲ׅ׷

apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: replicas: 3
selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.13.3 ports: - containerPort: 80 ,VCFSOFUFTחֶֽ׷㹑鎉涸鏣㹀,VCFSOFUFTوصؿؑأز ٖفٔؕ侧 㹋遤ׁ׸גְ׷ץֹ؝ٝذش侧 㹋遤ׁ׸׷؝ٝذش؎ً٦آ OHJOYךغ٦آّٝ M l k M l dT

䓼⸂ז荈䊹㔐䗁堣腉չإٕؿؼ٦ؚٔٝպ ꥺ㹱ח״׶7.װوءٝכ㠨׸׷կ㠨׸׋7.♳ד؟٦ؽأך竰竲ח䗳銲ז،فٔ؛٦ءָّٝ  㹋遤ׁ׸גְ׋׵וֲׅ׷ַ 劄ת׃ְ朐䡾植㖈ך朐䡾 !

䓼⸂ז荈䊹㔐䗁堣腉չإٕؿؼ٦ؚٔٝպ ,VCFSOFUFTכؙٓأةך朐䡾׾竰竲涸ח劄ת׃ְ朐䡾ד֮׶竲ֽ״ֲה⹛⡲ׅ׷կ ظ٦سوءٝָ侧〴تؐٝ׃ג׮،فٔ؛٦ءّٝך㹋遤ח䗳銲זٔا٦أָ婍׏גְ׷  ꣲ׶ծ绒㌀噟傈ח䗁傊ׅ׸לְְկ 劄ת׃ְ朐䡾植㖈ך朐䡾
!

7.⚥䗰דכזֻչ؝ٝذش⚥䗰պך؎ٝؿٓ ,VCFSOFUFTכծ醱侧ך7. ظ٦س ׾אך㣐ֹזٔا٦أف٦ٕ ؙٓأة ה׃ג䪔ֲֿהד  7.׾䬄韋⻉ׅ׷կ Ӝ ؙٓأةח㼎׃ג؝ٝذش׾رفٗ؎ׅ׷הծ׉ך؝ٝذشך㹋遤ח䗳銲זٔا٦أָ瑞ְגְ ׷ظ٦سחأ؛آُ٦ׁٕ׸׷
Ӝ Ꟛ涪罏כծ暴㹀ך؝ٝذشָؙٓأةⰻךוךظ٦سד㹋遤ׁ׸גְ׷ַ׾孡חז׷䗳銲ָזְ

ؙٓأةכظ٦سך꧊さ⡤ $16 3".(# ظ٦س $16 3".(# (16 $16 3".(# (16
k p a f dT

7.ך䬄韋⻉ח⸇ִגծ*5؎ٝؿٓ׮䬄韋⻉ 7.ך䬄韋⻉ח⸇ִגծ؟٦ؽأך圓䧭׾չ،فٔ؛٦ءّٝ䭷ぢպח䬄韋⻉ׁ׸׋  ,VCFSOFUFTؔـآؙؑزד䪔ֲկ Ӝ ٗ٦سغٓٝ؟װ宕竲أزٖ٦آזוך*5؎ٝؿٓ׾䬄韋⻉ Ӝ أذ٦زٖأַ׵أذ٦زؿٕתדׁתׂתז珏겲ך،فٔ؛٦ءّٝח㼎䘔ׅ׷  㢳ֻךؔـآؙؑزָ欽䠐ׁ׸גְ׷ a b
3 Lf y mdkY A03 3 3 r ) ! k

מ׿ם׿זرفٗ؎ח罣ִ׵׸׷ꬊ䌢ח⮚׸׋堣腉 Ӝ מ׿ם׿זرفٗ؎׾〳腉חׅ׷չ㹑鎉涸鏣㹀պ رفٗ؎׃׋穠卓׾չ㹑鎉涸חպ鎸鶢׃׋鏣㹀ؿ؋؎ٕח״׷رفٗ؎ 然㹋זٗ٦ٕغحؙ׮㺁僒ח㹋植 Ӝ 䓼⸂ז荈䊹㔐䗁堣腉չإٕؿؼ٦ؚٔٝպ 䌢ח⨳Ⰻז侧ך،فٔ؛٦ءָّٝ㹋遤ׁ׸גְ׷朐䡾׾⥂הֲהׅ׷ Ӝ 7.⟎䟝وءٝ⚥䗰דכזֻչ؝ٝذش⚥䗰ך؎ٝؿٓպ
⦐ղךوءٝך盖椚כ遤׻ז׻׆ծؙٓأة׾㣐ֹזوءٝה׃ג䪔ֲ ٗ٦سغٓٝ؟זוך*5؎ٝؿٓ׾䬄韋⻉׃ծ؟٦ؽأ圓䧭ך盖椚׾荈⹛⻉

➭ךفٓحزؿؓ٦يהך嫰鯰 Ӝ *OGSBTUSVDUVSFBTB4FSWJDF Ӝ 7. ر؍أؙ طحزٙ٦ؙ ,VCFSOFUFT Ӝ $POUBJOFSBTB4FSWJDF
Ӝ ؝ٝذشך盖椚ծ㹋遤 Ӝ ֮׵ײ׷،فٔ؛٦ءّٝ $MPVE'PVOESZ Ӝ 1MBUGPSNBTB4FSWJDF Ӝ ؝٦سַ׵رفٗ؎ Ӝ )551ծ"1*ծ8FC 0QFO4UBDL

Ӝ ؝ٝذشד荈歋䏝ה䬄韋⻉׾غٓ ٝأ״ֻ㹋植ׅ׷ Ӝ 䬄韋䏝ָ넝ֻծ欰欵䚍ָ넝ְ Ӝ ♧倯ד荈歋䏝ָ⡚ְ 暟椚؎ٝؿٓ 04 ؟٦غ⟎䟝⻉
ىسٕؐؑ، ٓٝة؎ي ،فٔ؛٦ءّٝ 暟椚؎ٝؿٓ 04 ؟٦غ⟎䟝⻉ ىسٕؐؑ، ٓٝة؎ي ،فٔ؛٦ءّٝ 暟椚؎ٝؿٓ 04 ؟٦غ⟎䟝⻉ ىسٕؐؑ، ٓٝة؎ي ،فٔ؛٦ءّٝ ؝ٝذش⻉ *BB4 ,VCFSOFUFT 1BB4 Ӝ 荈歋䏝כ넝ְָ盖椚ׅ׷ ׮ךָ㢳ְ 荈歋䏝欰欵䚍 h

,VCFSOFUFTך،٦ؗذؙثٍהؔـآؙؑز

,VCFSOFUFTך،٦ؗذؙثٍ Node Networking kubelet Container Runtime OS Master component etcd
apiserver controller-manager scheduler kubectl REST

,VCFSOFUFTؔـآؙؑز ٙ٦ؙٗ٦سװ*5؎ٝؿٓ׾չ،فٔ؛٦ءّٝ䭷ぢպח䬄韋⻉׃׋׮ךדծꟚ涪罏כؔـآؙؑ ز׾وصؿؑأزהㄎל׸׷:".-ת׋כ+40/䕎䒭ךؿ؋؎ٕד鎸鶢ׅ׷կ Ӝ 1PE醱侧ך؝ٝذشה醱侧ךنُٔ٦ي Ӝ -BCFM㈓♧ךؚٕ٦ؾؚٝ堣腉 Ӝ 3FQMJDB4FU/⦐ך1PEָ㹋遤ׁ׸גְ׷朐䡾׾⥂א Ӝ
%FQMPZNFOUٗ٦ؚٔٝ،حفر٦زծٗ٦ٕغحؙ Ӝ 4FSWJDF⟎䟝*1הه٦زծ؟٦ؽأךؒٝزٔه؎ٝز

kubectl apply -f manifest.yaml

1PEهحس Ӝ 醱侧ך؝ٝذشה  醱侧ךنُٔ٦ي Ӝ رفٗ؎ך剑㼭⽃⡘ Ӝ *1QFS1PE 1PE" نُٔ٦ي
؝ٝذش ؝ٝذش *1

1PEهحس Ӝ 醱侧ך؝ٝذشה  醱侧ךنُٔ٦ي Ӝ رفٗ؎ך剑㼭⽃⡘ Ӝ *1QFS1PE apiVersion: v1
kind: Pod metadata: name: nginx spec: containers: - name: nginx image: nginx:1.15.8 ports: - containerPort: 80

1PEهحس Ӝ 醱侧ך؝ٝذشה  醱侧ךنُٔ٦ي Ӝ رفٗ؎ך剑㼭⽃⡘ Ӝ *1QFS1PE 1PE" نُٔ٦ي
؝ٝذش ؝ٝذش ظ٦س ظ٦س *1 1PE# *1 1PE$ *1 1PEחろת׸׷؝ٝذشכ  䗳׆ずׄظ٦س♳ד㹋遤ׁ׸׷ ぐ1PEכؿٓحزזطحزٙ٦ؙך*1،سٖأ׾䭯׍ծظ٦س׾ת׋ְד鸐⥋דֹ׷

3FQMJDB4FUٖفٔؕإحز Ӝ /⦐ך1PEָ㹋遤ׁ׸גְ׷朐䡾׾⥂א 3FQMJDB4FU ٖفٔؕ侧  إؙٖةBQQOHJOY ظ٦س" 1PEذٝفٖ٦ز ظ٦س# 1PE$
BQQOHJOY BQQOHJOY 1PE# BQQOHJOY 1PE" BQQOHJOY 劄ת׃ְ侧植㖈ך侧

3FQMJDB4FUٖفٔؕإحز Ӝ /⦐ך1PEָ㹋遤ׁ׸גְ׷朐䡾׾⥂א apiVersion: apps/v1 kind: ReplicaSet metadata: name: nginx
spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.15.8 ports: - containerPort: 80 1PEذٝفٖ٦ز ⡲䧭ׁ׸׷1PEך׮ההז׷鏣㹀 ٖفٔؕ侧

1PE"" XFCW %FQMPZNFOUرفٗ؎ًٝز Ӝ ٗ٦ؚٔٝ،حفر٦ز Ӝ ٗ٦ٕغحؙ %FQMPZNFOU" DPOUBJOFST JNBHFXFCW
ٖفٔؕ侧 3FQMJDB4FU" DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE"# XFCW

ٖفٔؕ侧 3FQMJDB4FU" DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE"# XFCW 3FQMJDB4FU# DPOUBJOFST JNBHFXFCW ٖفٔؕ侧

ٖفٔؕ侧 3FQMJDB4FU" DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE"# XFCW 3FQMJDB4FU# DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE#" XFCW

%FQMPZNFOUرفٗ؎ًٝز Ӝ ٗ٦ؚٔٝ،حفر٦ز Ӝ ٗ٦ٕغحؙ %FQMPZNFOU" DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 3FQMJDB4FU"
DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE"# XFCW 3FQMJDB4FU# DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE#" XFCW

DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE"# XFCW 3FQMJDB4FU# DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE#" XFCW 1PE## XFCW

DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 3FQMJDB4FU# DPOUBJOFST JNBHFXFCW ٖفٔؕ侧 1PE#" XFCW 1PE## XFCW

%FQMPZNFOU Ӝ ٗ٦ؚٔٝ،حفر٦ز Ӝ ٗ٦ٕغحؙ apiVersion: apps/v1 kind: Deployment metadata:
name: nginx spec: replicas: 2 selector: matchLabels: app: nginx strategy: type: RollingUpdate rollingUpdate: maxSurge: 25% maxUnavailable: 25% template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.15.8 ports: - containerPort: 80 رفٗ؎䨌殛 ٖفٔؕ侧 1PEذٝفٖ٦ز

4FSWJDF؟٦ؽأ Ӝ ⟎䟝*1הه٦ز Ӝ ٓكٕإؙٖةח״׷1PEךؚٕ٦ؾؚٝ Ӝ ؟٦ؽأة؎ف $MVTUFS*1 *OUFSOBM
/PEF1PSU -PBE#BMBODFS &YUFSOBM  4FSWJDFNZTWD ة؎ف$MVTUFS*1  إؙٖةBQQOHJOY ه٦زUDQ̔UDQ DMVTUFS*1 1PE" 1PE# 3FQMJDB4FU BQQOHJOY BQQOHJOY

NZTWD 1PEē 4FSWJDFせח״׷؟٦ؽأر؍أؕغٔ 4FSWJDFNZTWD ة؎ف$MVTUFS*1  إؙٖةBQQOHJOY ه٦زUDQ̔UDQ DMVTUFS*1 1PE" 1PE#
3FQMJDB4FU BQQOHJOY BQQOHJOY

4FSWJDF؟٦ؽأ Ӝ ⟎䟝*1הه٦ز Ӝ ٓكٕإؙٖةח״׷1PEךؚٕ٦ؾؚٝ Ӝ ؟٦ؽأة؎ف $MVTUFS*1 *OUFSOBM
/PEF1PSU -PBE#BMBODFS &YUFSOBM  apiVersion: v1 kind: Service metadata: name: nginx spec: type: ClusterIP selector: app: nginx ports: - protocol: TCP port: 8080 targetPort: 80

׉ך➭ך⚺זؔـآؙؑز Ӝ /BNFTQBDFؙٓأة׾锷椚涸חⴓⶴׅ׷ Ӝ $POH.BQ،فٔ؛٦ءّٝה鏣㹀ךⴓꨄ Ӝ 4FDSFU،فٔ؛٦ءّٝהء٦ؙٖحزךⴓꨄ Ӝ 1FSTJTUFOU7PMVNF 1FSTJTUFOU7PMVNF$MBJN
4UPSBHF$MBTT宕竲نُٔ٦ي Ӝ 4UBUFGVM4FUأذ٦زؿٕ،فٔ؛٦ءّٝ Ӝ +PCٙٝءّحزآّـ Ӝ $SPO+PCآّـך㹀劍㹋遤 Ӝ %BFNPO4FUⰋגךظ٦سד1PE׾㹋遤 Ӝ *OHSFTT)551頾蚚ⴓ侔ծغ٦ثٍٕمأزծ5-4穄畭 Ӝ )PMJ[POUBM1PE"VUP4DBMFS )1" ؔ٦زأ؛٦ٕ

:BIPP+"1"/ך وط٦آس,VCFSOFUFT؟٦ؽأ׾佄ִ׷䪮遭

Ӝ ءؚٕٝذشٝز杝׶⽑׭

ז׈,VCFSOFUFTBTB4FSWJDFָ䗳銲זךַ 撕꧟ז,VCFSOFUFTךؔلٖ٦ءַّٝ׵麊欽罏׾鍑佝ׅ׷ Ӝ ؙٓأةך⡲䧭ծ⵴ꤐծ鏣㹀㢌刿 Ӝ ظ٦س 7. ך鷄⸇٥⵴ꤐ Ӝ ؙٓأةךئٗتؐٝة؎ي،حفؚٖ٦س
Ӝ ꥺ㹱װ㉏겗ך֮׷ظ٦سך⥜䗁 Ӝ ر٦ةأز، FUDE ךغحؙ،حفծٔأز، ,VCFSOFUFTכ؝ٝذش⻉ׁ׸׋،فٔ؛٦ءّٝך麊欽ח㣐ֹזًٔحزָ֮׷♧倯דؙٓأةך 麊欽חכ㢳ֻךؔلٖ٦ءָّٝ䗳銲חז׷կ au a b M n L i b t T

撕꧟ז,VCFSOFUFTؙٓأةךؔلٖ٦ءّٝ Ӝ קה׿וכ寸ת׶ֹ׏׋⡲噟ך粸׶鵤׃ Ӝ :BIPP+"1"/כ㕂ⰻ㢩ח醱侧ךر٦ةإٝة׾䭯׍ծ侧♰〴鋉垷ך؟٦غך  ؔلٖ٦ءָّٝ䗳銲חז׷ ➂꟦ךװ׷➬✲ׄׯזְկاؿزؐؑ،חװ׵ׇ׷կ AUTOMATE ALL THE
THINGS

,VCFSOFUFTBTB4FSWJDFך銲⟝ Ӝ أ؛٦ٓـٕ 盖椚㼎韋ָ侧♰〴ד׮㉏겗זֻ⹛⡲ׅ׷ Ӝ ꬊず劍ٌرٕ وءٝ 7. װٗ٦سغٓٝ؟ך彊⪒זוⳢ椚ָ㸣✪ׅ׷תד儗꟦ַַָ׷ Ӝ
㛙暕䚍 Ⳣ椚ָ㣟侁׃ג׮ⱄꟚדֹ׷ ءأذيך♧鿇ָ㠨׸ג׮ծⰋ⡤ָꥺ㹱׾饯ֿֿׅהכ鏩ׁ׸זְ

,VCFSOFUFT׾ⴓ侔ءأذيךؿٖ٦يٙ٦ؙה׃ג⢪ֲ ,VCFSOFUFTך㣐ֹז暴䗙ךאָ䭁䓸䚍 &YUFOTJCJMJUZ Ӝ $VTUPN3FTPVSDF%FOJUJPOT $3% ,VCFSOFUFTח⟣䠐ךٔا٦أة؎ف׾鷄⸇ׅ׷ Ӝ
$VTUPN$POUSPMMFS ,VCFSOFUFTח⟣䠐ךؽآطأٗآحؙ׾鷄⸇ׅ׷

$VTUPN3FTPVSDF%FOJUJPOT $3% "1* $POUSPMMFS *OTUBODF 3FQMJDB4FUT %FQMPZNFOUT  3FQMJDB4FU$POUSPMMFS $VTUPN$POUSPMMFS
 %FQMPZNFOU$POUSPMMFS 3FQMJDB4FU %FQMPZNFOU $VTUPN3FTPVSDF $3% 

,VCFSOFUFT$MVTUFS0QFSBUPSך$3%ה؝ٝزٗ٦ٓ "1* $POUSPMMFS *OTUBODF ,VCFSOFUFT$MVTUFST ,VCFSOFUFTFT ,VCFSOFUFT$MVTUFS  $POUSPMMFS ,VCFSOFUFT  $POUSPMMFS
,VCFSOFUFT$MVTUFS ,VCFSOFUFT .BDIJOF4FUT .BDIJOFT .BDIJOF4FU  $POUSPMMFS .BDIJOF  $POUSPMMFS .BDIJOF4FUT .BDIJOF    cL f O na M dT

apiVersion: apiextensions/k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: kubernetesclusters.kubernetes.zlab.co.jp spec: scope: Namespaced
group: kubernetes.zlab.co.jp version: v1alpha1 names: plural: kubernetesClusters singular: kubernetesCluster kind: KubernetesCluster shortNames: - kc additionalPrinterColumns: - name: ready type: boolean description: ”Phase of KubernetesCluster” JSONPath: .status.ready ŋŋŋ

$ cat cluster01.yaml apiVersion: kubernetes.zlab.co.jp/v1alpha1 kind: KubernetesCluster metadata: name: cluster01
spec: version: ”1.14.3” masterReplicas: 3 masterFlavor: large workerReplicas: 5 workerFlavor: large ŋŋŋ $ kubectl apply -f cluster.yaml Kubernetescluster “cluster01” created

,VCFSOFUFT$POUSPMMFS Ӝ ,VCFSOFUFTؔـآؙؑزך鷄⸇⵴ꤐ㢌刿׾湊鋔׃גծؔـآؙؑز׾乼⡲ׅ׷  ؽآطأٗآحؙ׾㹋遤ׅ׷ 0CTFSWF "OBMZ[F "DU 6TFSBDUJPOT  /FXPCKFDU SFDPOHVSF
,VCFSOFUFTFWFOUT  $VSSFOUTUBUFPGDMVTUFS

,VCFSOFUFT0QFSBUPS Ӝ ,VCFSOFUFTךِ٦ؠח㢌׻׏ג醱꧟זأذ٦زؿٕ،فٔ؛٦ءّٝך؎ٝأةٝأ׾⡲䧭ծ أ؛٦ٕծ،حفؚٖ٦سծ鏣㹀㢌刿זו׾遤ֲ؝ٝزٗ٦ٓ Ӝ 㛇劤涸ז,VCFSOFUFTךٔا٦أה؝ٝزٗ٦ٓך嚊䙀׾׮הח׃גծ,VCFSOFUFTך䭁䓸堣腉׾ 崞欽׃،فٔ؛٦ءّٝ㔿剣ך麊欽شٖحآ׾اؿزؐؑ،ה׃גفؚٗٓيׅ׷ An Operator represents
human operational knowledge in software to reliably manage an application. https://coreos.com/operators/

,VCFSOFUFT0QFSBUPSTJOUIF8JME Ӝ DPSFPTFUDEPQFSBUPS $SFBUFDPOHVSFNBOBHFFUDEDMVTUFSTBUPQ,VCFSOFUFT Ӝ DPSFPTQSPNFUIFVTPQFSBUPS $SFBUFDPOHVSFNBOBHF1SPNFUIFVTDMVTUFSTBUPQ,VCFSOFUFT Ӝ SPPLSPPL 4UPSBHF0SDIFTUSBUJPOGPS,VCFSOFUFT$FQI$PDLSPBDI%#
Ӝ PSBDMFNZTRMPQFSBUPS $SFBUF PQFSBUFBOETDBMFTFMGIFBMJOH.Z42-DMVTUFSTJO,VCFSOFUFT Ӝ LVCFPXUGPQFSBUPS 5PPMTGPS.-5FOTPSPXPO,VCFSOFUFT https://github.com/operator-framework/awesome-operators

4JUF3FMJBCJMJUZ&OHJOFFSJOH  اؿزؐؑ،׾剅ֻֿהד،فٔ؛٦ءّٝ׾麊欽ח荈⹛⻉׾׮׋׵ׅٗ٦ٕ ˑ׃׋ָ׏ג43&כծֿ׸תד麊欽ث٦يָ遤׏גֹ׋ֿה׾ اؿزؐؑ،ך㼔Ꟍ䚍׾䭯אؒٝآص،ָ遤ְծؒٝآص، ָ➂䩛ח״׷盖椚׾荈⹛⻉ׅ׷اؿزؐؑ،׾鏣鎘׃㹋鄲ׅ ׷腉⸂׾䭯׍ծ׉׸׾ְה׻זְהְֲֿהַ׵䧭׶甧׏ג ְתׅկ˒ https://www.oreilly.co.jp/books/9784873117911/

,VCFSOFUFT$MVTUFS0QFSBUPS 0CTFSWF "OBMZ[F "DU 6TFSBDUJPOT  /FXDMVTUFS SFDPOHVSF ,VCFSOFUFTFWFOUT  $VSSFOUTUBUFPGDMVTUFS 3

clusterctl Kubectl 銮%$ 匌%$ 6TFS,VCFSOFUFT$MVTUFS ,VCFSOFUFTBTB4FSWJDF$MVTUFS

,VCFSOFUFTBTB4FSWJDFך⚺ז堣腉 Ӝ ؙٓأةךإٕؿؼ٦ؚٔٝ ꥺ㹱ך涪欰׃׋ظ٦س׾荈⹛涸חⴖ׶ꨄ׃ծ倜׃ְظ٦س׾鷄⸇ׅ׷ Ӝ ؙٓأةךأ؛٦ٕ،ؐز٥أ؛٦ٕ؎ٝ ظ٦س侧ך؝وٝسדך㢌刿 Ӝ ؙٓأةך荈⹛،حفؚٖ٦س ظ٦تؐٝة؎يծٗ٦ؚٔٝ،حفر٦ز
Ӝ FUDEؙٓأةך荈⹛غحؙ،حف٥ٔأز، ر؍ؠأةٔؕغٔ

ؙٓأةךإٕؿؼ٦ؚٔٝ Ӝ ꥺ㹱ך涪欰׃׋ظ٦س׾荈⹛涸ח⵴ꤐ׃ծ  倜׋ח䗳銲זظ٦س׾ؙٓأةח鷄⸇ׅ׷կ ,VCFSOFUFT$MVTUFS0QFSBUPS ظ٦س$ ظ٦س# ظ٦س" ⵴ꤐ 鷄⸇
!

״׶鑫׃ֻ濼׷חכ

,VCFSOFUFT׾״׶鑫׃ֻ濼׷חכ Ӝ Ⱅ䒭سًُؗٝزLVCFSOFUFTJP Ӝ ּׅ㨣׭׷חכ ٗ٦ٕؕחꟚ涪欽ؙٓأة׾圓眠.JOJLVCF %PDLFS%FTLUPQGPS.BDBOE8JOEPXT NJDSPLT ػـٔحؙؙٓؐس⥋걾ה㹋籐ך(PPHMF,VCFSOFUFT&OHJOF Ӝ
傈劤؝ىُصذ؍,VCFSOFUFT4MBDLثٍٝطٕTMBDLLTJPKQVTFST KQFWFOUT Ӝ ى٦ز،حف ,VCFSOFUFT.FFUVQ5PLZPIUUQTLTKQDPOOQBTTDPN $MPVE/BUJWF.FFUVQ5PLZPIUUQTDMPVEOBUJWFDPOOQBTTDPN Ӝ ⹛歗 ,VCFSOFUFT.FFUVQ5PLZP:PV5VCFثٍٝطٕ $/$':PV5VCFثٍٝطٕ

َؿ٦ך,VCFSOFUFTBTB4FSWJDF׾״׶鑫׃ֻ濼׷חכ +BQBO$POUBJOFS%BZTW 0QFO4UBDL%BZT5PLZP $MPVE/BUJWF%BZT5PLZP

תה׭

,VCFSOFUFT Ӝ ؝ٝذشؔ٦؛أزٖ٦ءّٝخ٦ٕ 醱侧ךوءٝظ٦سד圓䧭ׁ׸׷ؙٓأةח㼎׃ג  ؝ٝذش،فٔ؛٦ءّٝךꂁ⪒ծ鏣㹀ծ盖椚׾遤ֲ Ӝ κυβερνήτηςؘٔءٍ铂ד乼菭㡦 Ӝ (PPHMFך爡ⰻءأذيַ׵؎ٝأػ؎،ׁ׸׋ “Kubernetes
is open source—a contrast to Borg and Omega,  which were developed as purely Google-internal systems. “ Borg, Omega, and Kubernetes

acL 3 3A g A3 2 M dT ,VCFSOFUFT Ӝ
䎃剢ח剑ⴱך؝ىحزָ֮׶ծ䎃剢ח׾ٔٔ٦أ 䎃剢儗挿ך剑倜غ٦آّٝכ Ӝ ⴱ劍כ(PPHMFךاؿزؐؑ،׌׏׋ָծ׉ך䖓  $MPVE/BUJWF$PNQVUJOH'PVOEBUJPO$/$'ח隁床ׁ׸ծ  植㖈כ$/$'ָمأزׅ׷ؔ٦فٝا٦أفٗآؙؑز Ӝ ؔٝفٖכ׮׍׹׿ծ⚺銲זؙٓؐسفٗغ؎تדوط٦آس؟٦ؽأָ䲿⣘ׁ׸גְ׷ (PPHMF,VCFSOFUFT&OHJOF "NB[PO&,4 "[VSF,VCFSOFUFT4FSJWJDFT Ӝ չ䎃⟃♳ח床׶؝ٝذش׾劤殢橆㞮ד麊欽׃גֹ׋(PPHMFך穗꿀ה  㢳ֻך⟰噟ח״׷؝ىُصذ؍ך⮚׸׋،؎ر،ה䩛岀ָ穈׫鴥ת׸גְ׷պ

Ӝ ءؚٕٝذشٝز杝׶⽑׭

2VFTUJPO Ӝ Kazuki Suda <LTVEB![MBCDPKQ !TVQFSCSPUIFST Ӝ 4MJEFIUUQTTQFBLFSEFDLDPNTVQFSCSPUIFST

Yahoo! JAPAN の マネージド Kubernetes サービスを支える技術 / Th...

Yahoo! JAPAN の マネージド Kubernetes サービスを支える技術 / The Technology Behind Managed Kubernetes Service of Yahoo! JAPAN

More Decks by Kazuki Suda

Other Decks in Technology

Featured

Transcript

Yahoo! JAPAN のマネージド Kubernetes サービスを支える技術 / Th...

Yahoo! JAPAN のマネージド Kubernetes サービスを支える技術 / The Technology Behind Managed Kubernetes Service of Yahoo! JAPAN