Upgrade to Pro — share decks privately, control downloads, hide ads and more …

やるぜ!Kubernetesハンズオン!

 やるぜ!Kubernetesハンズオン!

2019/05/09 Fusic社内Kubernetesハンズオン 資料

A9f8382f6cdd535370c723eba233a8f7?s=128

Keita Mohri

May 09, 2019
Tweet

More Decks by Keita Mohri

Other Decks in Technology

Transcript

  1. ΍Δͥʂ,VCFSOFUFTϋϯζΦϯʂ  ໟརܒଠ

  2. ຊ೔ͷ໨ඪ ‣ ,VCFSOFUFTͱ͓༑ୡʹͳΔ  ΠϝʔδΛ௫Ή  ৮ͬͯΈΔ ϩʔΧϧɾ&,4ɾͪΐͬͱ(,&  

    ߏஙɾӡ༻͕Ͱ͖ΔΑ͏ͳؾ෼ʹͳΔ ‣ ޽ʹ͋;ΕΔ,VCFSOFUFTͷهࣄɾεϥΠυ͕
 ཧղͰ͖ΔΑ͏ʹͳΔ 2
  3. 3 લఏ஌ࣝɾ४උ

  4. %PDLFS%PDLFS$PNQPTF ‣ ͜͜Ͱ͸લఏ஌ࣝͱͯ͠ѻ͏ͷͰઆ໌͠·ͤΜ  IUUQTHJUIVCDPNOPIFEPDLFSMFTTPOT  IUUQTXXXTMJEFTIBSFOFU[FNCVUTVEPDLFSDPNQPTF HVJEFCPPL 4

  5. %PDLFSGPS%FTLUPQ ‣ .BD8JOEPXT༻ͷ%PDLFS؀ڥ  IUUQTXXXEPDLFSDPNQSPEVDUTEPDLFSEFTLUPQ 5

  6. LVCFDUM ‣ ,VCFSOFUFTΫϥελΛૢ࡞͢Δ$-*  IUUQTLVCFSOFUFTJPEPDTUBTLTUPPMTJOTUBMMLVCFDUM ‣ .BDͷ৔߹  CSFXJOTUBMMLVCFSOFUFTDMJ 6

  7. "84ΞΧ΢ϯτ*".Ϣʔβʔ"84$-* ‣ ݸਓͷ"84ΞΧ΢ϯτ ‣ *".Ϣʔβʔ  "ENJOJTUSBUPS"DDFTTͷϙϦγʔΛΞλον w ݖݶߜΔͷ͕໘౗ͩͬͨ 

     ΞΫηεΩʔΛ࡞੒ ‣ "84$-*  IUUQTHJUIVCDPNBXTBXTDMJ  ࡞੒ͨ͠*".ϢʔβʔΞΫηεΩʔΛ࢖ͬͯQSPpMFΛઃఆ͓ͯ͘͠  QSPpMFΛ؀ڥม਺"84@130'*-&ʹઃఆ 7
  8. BXTJBNBVUIFOUJDBUPS ‣ *".ͷೝূ৘ใΛ࢖༻ͯ͠,VCFSOFUFTΫϥελʔ΁ͷ
 ೝূΛߦ͏πʔϧ  IUUQTEPDTBXTBNB[PODPNKB@KQFLTMBUFTU VTFSHVJEFJOTUBMMBXTJBNBVUIFOUJDBUPSIUNM ‣ .BDͷ৔߹ 

    CSFXJOTUBMMBXTJBNBVUIFOUJDBUPS 8
  9. FLTDUM ‣ &,4ͰΫϥελΛ࡞੒͢ΔͨΊͷ$-*  IUUQTFLTDUMJP ‣ .BDͷ৔߹  CSFXUBQXFBWFXPSLTUBQ 

    CSFXJOTUBMMXFBWFXPSLTUBQFLTDUM 9
  10. ؀ڥ֬ೝ 10 $ docker -v Docker version 18.09.2, build 6247962

    $ kubectl version --short --client Client Version: v1.13.4 $ aws --version aws-cli/1.16.125 Python/2.7.14 Darwin/18.2.0 botocore/1.12.115 $ aws-iam-authenticator help (লུ) $ eksctl version [ℹ] version.Info{BuiltAt:"", GitCommit:"", GitTag:"0.1.31"}
  11. 11 ,VCFSOFUFTͱ͸

  12. ,VCFSOFUFTͱ͸ ‣ ίϯςφΦʔέετϨʔγϣϯγεςϜ  ͨ͘͞Μͷαʔόʔʹ  ͨ͘͞ΜͷίϯςφΛஔ͍ͯ  ࿈ܞͤ͞ΔΑ͏ͳΞϓϦέʔγϣϯΛ 

    σϓϩΠɾ؅ཧɾεέʔϧͱ͔ͤ͞Δ΍ʔͭ ‣ ͷɺσϑΝΫτελϯμʔυ 12
  13. ,VCFSOFUFTͱ͸ ‣ ೥ʹ(PPHMF͕044ͱͯ͠ެ։  (PPHMFͷ௕͖ʹΘͨΔίϯςφӡ༻ͷ஌ݟ͕٧·͍ͬͯΔ ‣ ݱࡏ͸$/$'(Cloud Native Computing Foundation)͕؅ཧ

     Ϋϥ΢υωΠςΟϒͳ044ٕज़ͷਪਐΛߦ͏ஂମ ‣ %PDLFS΋ެࣜʹαϙʔτ  %PDLFS4XBSN͋Δͷʹʂ 13
  14. ,VCFSOFUFTͱ͸ ‣ ($1͚ͩͰͳ͘ΦϯϓϨ΍ଞͷΫϥ΢υͰ΋ಈ͘ ‣ ֤Ϋϥ΢υͰϚωʔδυαʔϏε͕ग़͍ͯΔ  ($1(,&  "84&,4 &$4͋Δͷʹʂ

      "[VSF",4  *#.$MPVE چ#MVFNJY *,4  "MJCBCB$POUBJOFS4FSWJDFGPS,VCFSOFUFT ‣ ਺೥ܦͯ͹౰ͨΓલͷٕज़ʹͳ͍ͬͯΔ͔΋ 14
  15. 15 ,VCFSOFUFTͬͯ
 Կ͕͏Ε͍͠ͷʁ

  16. ίϯςφͰΞϓϦέʔγϣϯͷӡ༻ ‣ ͨ͘͞Μͷαʔόʔʹͨ͘͞ΜͷίϯςφΛஔ͍ͯ࿈ܞ  %PDLFS$PNQPTFͰͲ͏ʹ͔഑ஔͯ͠Έͨͱ͜Ζ 16 MySQL Nginx App Redis

    SERVER Nginx App SERVER
  17. ίϯςφͰΞϓϦέʔγϣϯͷӡ༻ 17 docker-compose01.yaml docker-compose02.yaml MySQL Nginx App Redis SERVER Nginx

    App SERVER Ͳͷαʔόʔʹ
 Ͳͷίϯςφ͕͋Δ͔
 ؅ཧ͠ͳͪ͘Ό ‣ ͨ͘͞Μͷαʔόʔʹͨ͘͞ΜͷίϯςφΛஔ͍ͯ࿈ܞ  %PDLFS$PNQPTFͰͲ͏ʹ͔഑ஔͯ͠Έͨͱ͜Ζ
  18. App ίϯςφͰΞϓϦέʔγϣϯͷӡ༻ 18 docker-compose01.yaml docker-compose02.yaml MySQL Nginx App Redis SERVER

    Nginx SERVER ‣ ͨ͘͞Μͷαʔόʔʹͨ͘͞ΜͷίϯςφΛஔ͍ͯ࿈ܞ  %PDLFS$PNQPTFͰͲ͏ʹ͔഑ஔͯ͠Έͨͱ͜Ζ ίϯςφ͕ࢮΜͩΒ
 Ͳ͏΍ͬͯؾͮ͘ʁ Ͳ͏΍ͬͯճ෮͢Δʁ Ͳͷαʔόʔʹ
 Ͳͷίϯςφ͕͋Δ͔
 ؅ཧ͠ͳͪ͘Ό
  19. App ίϯςφͰΞϓϦέʔγϣϯͷӡ༻ 19 docker-compose01.yaml docker-compose02.yaml MySQL Nginx App Redis SERVER

    Nginx SERVER ‣ ͨ͘͞Μͷαʔόʔʹͨ͘͞ΜͷίϯςφΛஔ͍ͯ࿈ܞ  %PDLFS$PNQPTFͰͲ͏ʹ͔഑ஔͯ͠Έͨͱ͜Ζ αʔόʔΛލ͍ͩ
 ίϯςφؒͷ௨৴͸
 Ͳ͏͠Α͏ʁ ίϯςφ͕ࢮΜͩΒ
 Ͳ͏΍ͬͯؾͮ͘ʁ Ͳ͏΍ͬͯճ෮͢Δʁ Ͳͷαʔόʔʹ
 Ͳͷίϯςφ͕͋Δ͔
 ؅ཧ͠ͳͪ͘Ό
  20. App ίϯςφͰΞϓϦέʔγϣϯͷӡ༻ 20 docker-compose01.yaml docker-compose02.yaml MySQL Nginx App Redis SERVER

    Nginx SERVER ‣ ͨ͘͞Μͷαʔόʔʹͨ͘͞ΜͷίϯςφΛஔ͍ͯ࿈ܞ  %PDLFS$PNQPTFͰͲ͏ʹ͔഑ஔͯ͠Έͨͱ͜Ζ αʔόʔΛލ͍ͩ
 ίϯςφؒͷ௨৴͸
 Ͳ͏͠Α͏ʁ v1.1.0 ίϯςφͷߋ৽ Ͳ͏͠Α͏ʁ
 ॱংɺґଘؔ܎etc… ίϯςφ͕ࢮΜͩΒ
 Ͳ͏΍ͬͯؾͮ͘ʁ Ͳ͏΍ͬͯճ෮͢Δʁ Ͳͷαʔόʔʹ
 Ͳͷίϯςφ͕͋Δ͔
 ؅ཧ͠ͳͪ͘Ό
  21. App ίϯςφͰΞϓϦέʔγϣϯͷӡ༻ 21 docker-compose01.yaml docker-compose02.yaml MySQL Nginx App Redis SERVER

    Nginx SERVER ‣ ͨ͘͞Μͷαʔόʔʹͨ͘͞ΜͷίϯςφΛஔ͍ͯ࿈ܞ  %PDLFS$PNQPTFͰͲ͏ʹ͔഑ஔͯ͠Έͨͱ͜Ζ αʔόʔΛލ͍ͩ
 ίϯςφؒͷ௨৴͸
 Ͳ͏͠Α͏ʁ Load Balancer ϩʔυόϥϯγϯά
 ͠ͳ͍ͱͶ ίϯςφ͕ࢮΜͩΒ
 Ͳ͏΍ͬͯؾͮ͘ʁ Ͳ͏΍ͬͯճ෮͢Δʁ v1.1.0 ίϯςφͷߋ৽ Ͳ͏͠Α͏ʁ
 ॱংɺґଘؔ܎etc… Ͳͷαʔόʔʹ
 Ͳͷίϯςφ͕͋Δ͔
 ؅ཧ͠ͳͪ͘Ό
  22. App SERVER App ίϯςφͰΞϓϦέʔγϣϯͷӡ༻ 22 docker-compose01.yaml docker-compose02.yaml MySQL Nginx App

    Redis SERVER Nginx SERVER ‣ ͨ͘͞Μͷαʔόʔʹͨ͘͞ΜͷίϯςφΛஔ͍ͯ࿈ܞ  %PDLFS$PNQPTFͰͲ͏ʹ͔഑ஔͯ͠Έͨͱ͜Ζ αʔόʔΛލ͍ͩ
 ίϯςφؒͷ௨৴͸
 Ͳ͏͠Α͏ʁ Load Balancer ϩʔυόϥϯγϯά
 ͠ͳ͍ͱͶ ίϯςφ͕ࢮΜͩΒ
 Ͳ͏΍ͬͯؾͮ͘ʁ Ͳ͏΍ͬͯճ෮͢Δʁ v1.1.0 ίϯςφͷߋ৽ Ͳ͏͠Α͏ʁ
 ॱংɺґଘؔ܎etc… Nginx εέʔϦϯά… Ͳͷαʔόʔʹ
 Ͳͷίϯςφ͕͋Δ͔
 ؅ཧ͠ͳͪ͘Ό
  23. App SERVER App ίϯςφͰΞϓϦέʔγϣϯͷӡ༻ 23 docker-compose01.yaml docker-compose02.yaml MySQL Nginx App

    Redis SERVER Nginx SERVER ‣ ͨ͘͞Μͷαʔόʔʹͨ͘͞ΜͷίϯςφΛஔ͍ͯ࿈ܞ  %PDLFS$PNQPTFͰͲ͏ʹ͔഑ஔͯ͠Έͨͱ͜Ζ αʔόʔΛލ͍ͩ
 ίϯςφؒͷ௨৴͸
 Ͳ͏͠Α͏ʁ Load Balancer ϩʔυόϥϯγϯά
 ͠ͳ͍ͱͶ ίϯςφ͕ࢮΜͩΒ
 Ͳ͏΍ͬͯؾͮ͘ʁ Ͳ͏΍ͬͯճ෮͢Δʁ v1.1.0 ίϯςφͷߋ৽ Ͳ͏͠Α͏ʁ
 ॱংɺґଘؔ܎etc… Nginx εέʔϦϯά… Ͳͷαʔόʔʹ
 Ͳͷίϯςφ͕͋Δ͔
 ؅ཧ͠ͳͪ͘Ό ϩάͷ؅ཧ…
  24. ਓྨʹ͸ߴ౓͗͢Δ 24

  25. Ͱ͖ΔΜͰ͢ ͦ͏ɺ,VCFSOFUFTͳΒͶ 25

  26. 26 ,VCFSOFUFTͷ
 ΠϝʔδΛ௫Ή

  27. ϕʔεͱͳΔΞΠσΞ ‣ ίϯςφͷ؅ཧΛ͍͍ײ͡ʹ΍ͬͯ͘ΕΔγεςϜ͕͋Ε͹͍͍͡ΌΜ 27 SERVER SERVER ͍͍ײ͡ͷ γεςϜ ஫จॻ

  28. ϕʔεͱͳΔΞΠσΞ ‣ ίϯςφͷ؅ཧΛ͍͍ײ͡ʹ΍ͬͯ͘ΕΔγεςϜ͕͋Ε͹͍͍͡ΌΜ 28 SERVER SERVER ͍͍ײ͡ͷ γεςϜ ஫จॻ Nginx

    App Redis MySQL Nginx App ͍͍ײ͡ʹ৔ॴΛ൑அͯ͠ ίϯςφΛ഑ஔ
  29. ϕʔεͱͳΔΞΠσΞ ‣ ίϯςφͷ؅ཧΛ͍͍ײ͡ʹ΍ͬͯ͘ΕΔγεςϜ͕͋Ε͹͍͍͡ΌΜ 29 SERVER SERVER ͍͍ײ͡ͷ γεςϜ ஫จॻ Nginx

    App Redis MySQL Nginx ͍͍ײ͡ʹ৔ॴΛ൑அͯ͠ ίϯςφΛ഑ஔ App App ίϯςφ͕ࢮΜͩΒ ࣗಈճ෮
  30. ϕʔεͱͳΔΞΠσΞ ‣ ίϯςφͷ؅ཧΛ͍͍ײ͡ʹ΍ͬͯ͘ΕΔγεςϜ͕͋Ε͹͍͍͡ΌΜ 30 SERVER SERVER ͍͍ײ͡ͷ γεςϜ ஫จॻ Nginx

    App Redis MySQL Nginx ͍͍ײ͡ʹ৔ॴΛ൑அͯ͠ ίϯςφΛ഑ஔ ίϯςφ͕ࢮΜͩΒ ࣗಈճ෮ αʔόʔؒͷ ωοτϫʔΫ΋ ͍͍ײ͡ʹ App App
  31. ϕʔεͱͳΔΞΠσΞ ‣ ίϯςφͷ؅ཧΛ͍͍ײ͡ʹ΍ͬͯ͘ΕΔγεςϜ͕͋Ε͹͍͍͡ΌΜ 31 SERVER SERVER ͍͍ײ͡ͷ γεςϜ ஫จॻv1.1.0 Nginx

    App Redis MySQL Nginx ͍͍ײ͡ʹ৔ॴΛ൑அͯ͠ ίϯςφΛ഑ஔ ίϯςφ͕ࢮΜͩΒ ࣗಈճ෮ αʔόʔؒͷ ωοτϫʔΫ΋ ͍͍ײ͡ʹ App App App v1.1.0 App v1.1.0 ϩʔϦϯάΞοϓσʔτ Blue/GreenσϓϩΠϝϯτ
  32. ϕʔεͱͳΔΞΠσΞ ‣ ίϯςφͷ؅ཧΛ͍͍ײ͡ʹ΍ͬͯ͘ΕΔγεςϜ͕͋Ε͹͍͍͡ΌΜ 32 SERVER SERVER ͍͍ײ͡ͷ γεςϜ ஫จॻv1.1.0 Nginx

    App Redis MySQL Nginx ͍͍ײ͡ʹ৔ॴΛ൑அͯ͠ ίϯςφΛ഑ஔ ίϯςφ͕ࢮΜͩΒ ࣗಈճ෮ αʔόʔؒͷ ωοτϫʔΫ΋ ͍͍ײ͡ʹ App App App v1.1.0 App v1.1.0 ϩʔϦϯάΞοϓσʔτ Blue/GreenσϓϩΠϝϯτ Load Balancer ϩʔυόϥϯγϯά΋ Ͱ͖Δ
  33. App SERVER Nginx ϕʔεͱͳΔΞΠσΞ ‣ ίϯςφͷ؅ཧΛ͍͍ײ͡ʹ΍ͬͯ͘ΕΔγεςϜ͕͋Ε͹͍͍͡ΌΜ 33 SERVER SERVER ͍͍ײ͡ͷ

    γεςϜ ஫จॻv1.1.0 Nginx App Redis MySQL Nginx ͍͍ײ͡ʹ৔ॴΛ൑அͯ͠ ίϯςφΛ഑ஔ ίϯςφ͕ࢮΜͩΒ ࣗಈճ෮ αʔόʔؒͷ ωοτϫʔΫ΋ ͍͍ײ͡ʹ App App App v1.1.0 App v1.1.0 ϩʔϦϯάΞοϓσʔτ Blue/GreenσϓϩΠϝϯτ Load Balancer ϩʔυόϥϯγϯά΋ Ͱ͖Δ αʔόʔͷ ΦʔτεέʔϦϯάʂ
  34. App SERVER Nginx ϕʔεͱͳΔΞΠσΞ ‣ ίϯςφͷ؅ཧΛ͍͍ײ͡ʹ΍ͬͯ͘ΕΔγεςϜ͕͋Ε͹͍͍͡ΌΜ 34 SERVER SERVER ͍͍ײ͡ͷ

    γεςϜ ஫จॻv1.1.0 Nginx App Redis MySQL Nginx ͍͍ײ͡ʹ৔ॴΛ൑அͯ͠ ίϯςφΛ഑ஔ ίϯςφ͕ࢮΜͩΒ ࣗಈճ෮ αʔόʔؒͷ ωοτϫʔΫ΋ ͍͍ײ͡ʹ App App App v1.1.0 App v1.1.0 ϩʔϦϯάΞοϓσʔτ Blue/GreenσϓϩΠϝϯτ Load Balancer ϩʔυόϥϯγϯά΋ Ͱ͖Δ αʔόʔͷ ΦʔτεέʔϦϯάʂ ϩάऩू΋Χϯλϯ
  35. Ұ୴͜͜·ͰΛ,VCFSOFUFT༻ޠʹม׵ 35 Kubernetes Cluster ϚχϑΣετ kubectl (CLI for k8s) =

    Control Plane Data Plane
  36. ϚχϑΣετ ‣ γεςϜͷ͋Δ΂͖࢟Λॻ͍͍ͯΔZBNMϑΝΠϧ  ʮએݴతઃఆʯͱݺ͹ΕΔ ʮ໋ྩతઃఆʯ   એݴͨ͠ঢ়ଶΛҡ࣋͠Α͏ͱͯ͘͠ΕΔ ‣

    ໋ྩతઃఆී௨ͷͦ͹԰͞Μ  ͟Δͦ͹ͭ஫จ͢Δͦ͹Λग़ͨ͋͠ͱ͸஌Βͳ͍  ͓͔ΘΓ͢Δͱ͖͸ผ్஫จ͕ඞཁ ‣ એݴతઃఆΘΜͦ͜͹  ͓࿶ʹͦ͹͕͋Δঢ়ଶ͕͋Δ΂͖࢟ͦ͹Λৗʹ؂ࢹ͍ͯ͠Δ  ͦ͹Λ৯΂ͨΒࣗಈͰ௥Ճͯ͘͠ΕΔ 36
  37. ,VCFSOFUFTͷΠϝʔδ ‣ એݴతʹॻ͍ͨϚχϑΣετΛ ‣ LVCFDUMΛ࢖ͬͯNBTUFSʹ౉͢ͱ ‣ ֤OPEFʹίϯςφΛσϓϩΠͨ͠Γͯ͘͠Εͯ ‣ ͦͷޙ͸͍͍ײ͡ʹ؂ࢹɾҡ࣋Λͯ͘͠ΕΔ΍ʔͭ 37

    Kubernetes Cluster ϚχϑΣετ kubectl (CLI for k8s) = Control Plane Data Plane
  38. ͱΓ͋͑ͣ΍ͬͯΈΑ͏ 38

  39. 39 ,VCFSOFUFTΛ
 ϩʔΧϧͰࢼͯ͠ΈΑ͏

  40. ,VCFSOFUFT༗ޮԽ 40 1SFGFSFODFTը໘ ,VCFSOFUFTλϒΛબ୒ νΣοΫʂ "QQMZ ॳճ͸༗ޮʹͳΔ·Ͱ
 ෼΄Ͳ͔͔Γ·͢

  41. ,VCFSOFUFT༗ޮԽ 41 Ҏલ࢖ͬͨ͜ͱ͋Δਓ͸Ұ୴Ϧηοτ͓ͯ͘͠ͱ͍͍͔΋ 3FTFUλϒΛબ୒ Ϧηοτ

  42. LVCFDUMͷDPOpHઃఆ 42 # kubectlͰૢ࡞͢ΔKubernetesΫϥελʔΛdocker-for-desktopʹ $ kubectl config user-context docker-for-desktop #

    ͪΌΜͱઃఆ͞Ε͍ͯΔ͔֬ೝ $ kubectl config current-context docker-for-desktop
  43. ಈ࡞֬ೝͱԼ४උ 43 # ಈ࡞֬ೝɻͳΜ͔͍Ζ͍Ζಈ͍ͯΔ(આ໌͸͋ͱͰ) $ kubectl get pods --namespace=kube-system NAMESPACE

    NAME READY STATUS RESTARTS AGE kube-system etcd-docker-for-desktop 1/1 Running 0 1m kube-system kube-apiserver-docker-for-desktop 1/1 Running 0 1m kube-system kube-controller-manager-docker-for-desktop 1/1 Running 0 1m kube-system kube-dns-86f4d74b45-xb4qh 3/3 Running 0 2m kube-system kube-proxy-8r45p 1/1 Running 0 2m kube-system kube-scheduler-docker-for-desktop 1/1 Running 0 1m # docker-for-desktopͰingress͕࢖͑ΔΑ͏Լ४උɻ͋·Γؾʹ͠ͳͯ͘OK $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/ mandatory.yaml $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/ provider/cloud-generic.yaml
  44. σϓϩΠ 44 # αϯϓϧΞϓϦέʔγϣϯͷϚχϑΣετϑΝΠϧΛऔͬͯ͘Δ $ git clone git@github.com:kubernetes/examples.git # PCͷෛ୲ΛԼ͛ΔͨΊͪΐͬͱௐ੔

    $ vi examples/guestbook/frontend-deployment.yaml 10ߦ໨ replicas: 3 <- ͜ΕΛ1ʹมߋ $ vi examples/guestbook/redis-slave-deployment.yaml 11ߦ໨ replicas: 2 <- ͜ΕΛ1ʹมߋ # σϓϩΠ $ kubectl apply -f examples/guestbook/
  45. ϒϥ΢β͔ΒΞΫηεͰ͖ΔΑ͏ʹ 45 $ cat << 'EOT' >./guestbook-ingress.yaml apiVersion: extensions/v1beta1 kind:

    Ingress metadata: name: guestbook-ingress spec: rules: - http: paths: - path: / backend: serviceName: frontend servicePort: 80 EOT # σϓϩΠ $ kubectl apply -f guestbook-ingress.yaml # ADDRESS͕localhostʹͳΔ·Ͱ଴ػ $ kubectl get ingress NAME HOSTS ADDRESS PORTS AGE guestbook-ingress * localhost 80 24m
  46. Ͱ͖ͨʂ 46 IUUQMPDBMIPTU 1$ͷঢ়گʹΑͬͯ͸͏·͘ΞΫηεͰ͖ͳ͔ͬͨΓ͢Δ͔΋ IUUQͩͱ͏·͍͘͘৔߹΋͋Δͱ͔

  47. Կ͕ىͬͨ͜ 47

  48. ͜Μͳͷ͕Ͱ͖ͯ·͢ 48 Kubernetes Cluster Redis-Master Guestbook-App gcr.io/google-samples/gb-frontend:v4 k8s.gcr.io/redis:e2e Redis-Slave gcr.io/google_samples/gb-redisslave:v1

    localhost
  49. ୯७ͳΞϓϦͰ͢Ͷ 49 Kubernetes Cluster Redis-Master Guestbook-App gcr.io/google-samples/gb-frontend:v4 k8s.gcr.io/redis:e2e Redis-Slave gcr.io/google_samples/gb-redisslave:v1

    localhost
  50. 1PE ‣ ,VCFSOFUFTͷ࠷খσϓϩΠ୯Ґ ‣ ͭҎ্ͷίϯςφͱετϨʔδϘϦϡʔϜͷू·Γ ‣ ಉҰ1PE಺ͷίϯςφ͸ಉҰ/PEFʹ഑ஔ͞ΕΔ  ʮಉҰ/PEFͰಈ࡞͢Δඞཁ͕͋Δ͔ʁʯ͕1PEߏ੒ͷҰͭͷج४ ‣

    ͭͷ1PE಺ͷίϯςφ͸ಉ͡*1ΞυϨεͱϙʔτΛ࢖༻͢Δ  1PE಺ͷίϯςφؒͷ௨৴͸ϓϩηεؒ௨৴ͱͯ͠ߦ͏ 50
  51. 3FQMJDB4FU ‣ ಉ͡࢓༷ͷ1PE͕ࢦఆͨ͠਺͚ͩଘࡏ͢ΔΑ͏ੜ੒ɾ؅ཧ͢Δ  1PE͕ࢮΜͩͱ͖΋ࢦఆͨ͠਺ʹͳΔΑ͏ࣗಈճ෮ͯ͘͠ΕΔ ‣ 1PEͱ3FQMJDB4FU͸ૄ݁߹  -BCFMͱ͍͏ϝλσʔλΛ࢖ͬͯ౎౓ݕࡧ͍ͯ͠Δ 

    खಈͰ1PEͷ-BCFMΛॻ͖׵͑Ε͹ɺ
 3FQMJDB4FU͔Β੾Γ཭ͯ͠σόοά͢Δͱ͍ͬͨ͜ͱ΋Մೳ 51
  52. %FQMPZNFOU ‣ ৽͍͠όʔδϣϯͷϦϦʔεΛ؅ཧ͢ΔͨΊͷ࢓૊Έ  3FQMJDB4FUͷมߋΛ҆શʹ൓өͤ͞Δੈ୅؅ཧ͢Δ  1PEͷεέʔϧɺίϯςφͷߋ৽ɺϩʔϧόοΫFUD ‣ ͭͷ%FQMPZNFOUઓུ 

    3FDSFBUF  3PMMJOH6QEBUF ‣ 3FQMJDB4FUͱ%FQMPZNFOU΋ૄ݁߹ 52
  53. GSPOUFOEEFQMPZNFOUZBNM 53 apiVersion: apps/v1 # apply࣌ʹ࢖༻͢ΔAPIͷछผɻϦιʔε(kind)ʹΑܾͬͯ·Δ kind: Deployment # DeploymentͷϚχϑΣετ

    metadata: name: frontend # DeploymentϦιʔεͷ໊લɻʮmetadata.name + ϥϯμϜจࣈྻʯͷ໊લͰReplicaSet͕ੜ੒͞ΕΔ spec: selector: matchLabels: # ReplicaSet͕PodΛݕࡧ͢Δͱ͖ͷLabel app: guestbook tier: frontend replicas: 1 # ReplicaSet͕ੜ੒ɾ؅ཧ͢ΔPodͷ਺ template: # ---͔͜͜ΒPodͷఆٛ-------------------------------------------- metadata: labels: # PodͷLabelɻReplicaSet͕؅ཧԼͷPodΛݕࡧ͢Δͱ͖ʹ࢖͏ app: guestbook tier: frontend spec: containers: - name: php-redis # ίϯςφ໊ image: gcr.io/google-samples/gb-frontend:v4 # ίϯςφΠϝʔδ resources: # ࢖༻͢ΔCPU, Memoryͷࢦఆ requests: cpu: 100m memory: 100Mi env: # ؀ڥม਺ - name: GET_HOSTS_FROM value: dns ports: # EXPOSE͢Δϙʔτͷࢦఆ - containerPort: 80
  54. SFEJTNBTUFSEFQMPZNFOUZBNM 54 apiVersion: apps/v1 kind: Deployment metadata: name: redis-master spec:

    selector: matchLabels: app: redis role: master tier: backend replicas: 1 template: metadata: labels: app: redis role: master tier: backend spec: containers: - name: master image: k8s.gcr.io/redis:e2e resources: requests: cpu: 100m memory: 100Mi ports: - containerPort: 6379
  55. SFEJTTMBWFEFQMPZNFOUZBNM 55 apiVersion: apps/v1 kind: Deployment metadata: name: redis-slave spec:

    selector: matchLabels: app: redis role: slave tier: backend replicas: 1 template: metadata: labels: app: redis role: slave tier: backend spec: containers: - name: slave image: gcr.io/google_samples/gb-redisslave:v1 resources: requests: cpu: 100m memory: 100Mi env: - name: GET_HOSTS_FROM value: dns ports: - containerPort: 6379
  56. 4FSWJDF ‣ 1PEͷू߹ ओʹ3FQMJDB4FU ʹର͢Δܦ࿏΍
 αʔϏεσΟεΧόϦΛఏڙ  Ϋϥελ಺%/4Ͱɺ4FSWJDF໊/BNFTQBDF໊Ͱ໊લղܾՄೳʹ  ಉ͡/BNFTQBDF಺ͳΒ4FSWJDF໊͚ͩͰ0,

    ‣ ͜͜Ͱ΋-BCFMʹΑͬͯର৅ͷ1PE͕ݕࡧ͞ΕΔ  ର৅ͷ1PE͕ಈతʹೖΕସΘͬͨΓͯ͠΋ɺ
 -BCFM͍͍͑ͭͯ͞Ε͹Ұ؏໊ͨ͠લͰΞΫηεͰ͖Δ 56
  57. GSPOUFOETFSWJDFZBNM 57 apiVersion: v1 kind: Service # ServiceͷϚχϑΣετ metadata: name:

    frontend # ServiceϦιʔεͷ໊લ labels: # Serviceʹ͚ͭΔLabel app: guestbook tier: frontend spec: type: NodePort # ServiceͷछผɻNodePort͸Ϋϥελ֎͔ΒΞΫηεͰ͖Δ΍ͭ ports: - port: 80 # ΞΫηεΛड͚෇͚Δϙʔτ selector: # ର৅ͷPodΛݕࡧ͢Δͱ͖ͷLabel app: guestbook tier: frontend
  58. SFEJTNBTUFSTFSWJDFZBNM 58 apiVersion: v1 kind: Service metadata: name: redis-master labels:

    app: redis role: master tier: backend spec: # লུ͞Ε͍ͯΔ͚Ͳtype͸σϑΥϧτͷ"ClusterIP"ɻΫϥελ্ͷ಺෦IPΞυϨεʹServiceΛެ։ ports: - port: 6379 targetPort: 6379 selector: app: redis role: master tier: backend
  59. SFEJTTMBWFTFSWJDFZBNM 59 apiVersion: v1 kind: Service metadata: name: redis-slave labels:

    app: redis role: slave tier: backend spec: ports: - port: 6379 selector: app: redis role: slave tier: backend
  60. *OHSFTT ‣ 4FSWJDFΛΫϥελ֎ʹެ։ ‣ /PEF1PSUλΠϓͷ4FSWJDFͱҧ͍ɺ
 ύεϕʔεͰసૹઌͷ4FSWJDFΛ੾Γସ͑Δͱ͍ͬͨ͜ͱ΋Մೳ  4FSWJDF /PEF1PSU -ϨϕϧͰͷ੍ޚ

     *OHSFTT-ϨϕϧͰͷ੍ޚ 60
  61. HVFTUCPPLJOHSFTTZBNM 61 apiVersion: extensions/v1beta1 kind: Ingress # IngressͷϚχϑΣετ metadata: name:

    guestbook-ingress # IngressϦιʔεͷ໊લ spec: rules: # ϧʔςΟϯάͷϧʔϧͷ഑ྻ - http: paths: - path: / backend: # "frontend"Serviceͷ80൪ϙʔτʹΞΫηε serviceName: frontend servicePort: 80
  62. ͜Θ͘ͳʔ͍ 62 Kubernetes Cluster Redis-Master Guestbook-App gcr.io/google-samples/gb-frontend:v4 k8s.gcr.io/redis:e2e Redis-Slave gcr.io/google_samples/gb-redisslave:v1

    localhost
  63. தΛ೷͍ͯΈΑ͏ 63

  64. %FQMPZNFOUΛݟͯΈΔ 64 # Deploymentৄࡉ (ը໘ʹೖΒͳ͍ͷͰखݩͰݟͯʂ) $ kubectl describe deploy frontend

    3FQMJDB4FUΛݕࡧ͢ΔͨΊͷ-BCFMηϨΫλ kubectl get rs -l app=guestbook,tier=frontend ͱ͍ͬͨײ͡Ͱର৅ͷ3FQMJDB4FUΛݕࡧͰ͖Δ # DeoloymentҰཡ ("-o wide"͸ৄࡉΛݟΔͨΊͷΦϓγϣϯ) $ kubectl get deploy -o wide NAME (ུ) SELECTOR frontend ... app=guestbook,tier=frontend redis-master ... app=redis,role=master,tier=backend redis-slave ... app=redis,role=slave,tier=backend
  65. 3FQMJDB4FUΛݟͯΈΔ 65 # ReplicaSetҰཡ $ kubectl get rs -o wide

    NAME (ུ) SELECTOR frontend-5c548f4769 ... app=guestbook,pod-template-hash=1710490325,tier=frontend redis-master-55db5f7567 ... app=redis,pod-template-hash=1186193123,role=master,tier=backend redis-slave-584c66c5b5 ... app=redis,pod-template-hash=1407227161,role=slave,tier=backend # ReplicaSetৄࡉ (ը໘ʹೖΒͳ͍ͷͰखݩͰݟͯʂ) $ kubectl describe rs frontend-5c548f4769 %FQMPZNFOU໊ ϥϯμϜจࣈྻ 3PMMJOH6QEBUFͳͲͰಉ͡UFNQMBUFͷ1PEΛ
 ؅ཧ͢Δ3FQMJDB4FU͕ෳ਺ࠞࡏͯ͠΋େৎ෉ͳΑ͏ʹ
 ݻ༗஋ͷ-BCFMΛࣗಈͰೖΕ͍ͯΔ
  66. 1PEΛݟͯΈΔ 66 # PodҰཡ $ kubectl get pod -o wide

    NAME READY STATUS RESTARTS AGE IP NODE frontend-5c548f4769-xhpxz 1/1 Running 0 1d 10.1.1.65 docker-for-desktop redis-master-55db5f7567-2n4qp 1/1 Running 0 1d 10.1.1.67 docker-for-desktop redis-slave-584c66c5b5-z2fvj 1/1 Running 0 1d 10.1.1.66 docker-for-desktop # Podৄࡉ (ը໘ʹೖΒͳ͍ͷͰखݩͰݟͯʂ) # ઃఆ΍ىಈ೔࣌ɺঢ়ଶɺΠϕϯτ౳֬ೝͰ͖Δ $ kubectl describe pod frontend-5c548f4769-xhpxz # Podͷϩά (ը໘ʹೖΒͳ͍ͷͰखݩͰݟͯʂ) $ kubectl logs frontend-5c548f4769-xhpxz 3FQMJDB4FU໊ ϥϯμϜจࣈྻ
  67. 4FSWJDFΛݟͯΈΔ 67 # ServiceҰཡ $ kubectl get svc -o wide

    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR frontend NodePort 10.102.204.76 <none> 80:30590/TCP 1d app=guestbook,tier=frontend kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1d <none> redis-master ClusterIP 10.98.133.213 <none> 6379/TCP 1d app=redis,role=master,tier=backend redis-slave ClusterIP 10.107.141.173 <none> 6379/TCP 1d app=redis,role=slave,tier=backend # Serviceৄࡉ (ը໘ʹೖΒͳ͍ͷͰखݩͰݟͯʂ) $ kubectl describe svc frontend GSPOUFOE͕/PEF1PSUʹͳ͍ͬͯΔͷͰɺ IUUQMPDBMIPTUͰ΋ΞΫηεͰ͖Δʢͨͩ͠-੍ޚʣ
  68. *OHSFTTΛݟͯΈΔ 68 # IngressҰཡ $ kubectl get ing NAME HOSTS

    ADDRESS PORTS AGE guestbook-ingress * localhost 80 1d # Ingressৄࡉ (ը໘ʹೖΒͳ͍ͷͰखݩͰݟͯʂ) $ kubectl describe ing guestbook-ingress
  69. εέʔϧͤͯ͞ΈΑ͏ 69

  70. GSPOUFOEͷ1PEΛͭʹ૿΍͢ 70 $ vi examples/guestbook/frontend-deployment.yaml 10ߦ໨ replicas: 1 <- ͜ΕΛ2ʹมߋ

    # σϓϩΠ $ kubectl apply -f examples/guestbook/frontend-deployment.yaml # ૿͑ͯΔʂ $ kubectl get pod NAME READY STATUS RESTARTS AGE frontend-5c548f4769-vltkv 1/1 Running 0 15s frontend-5c548f4769-xhpxz 1/1 Running 0 1d redis-master-55db5f7567-2n4qp 1/1 Running 0 1d redis-slave-584c66c5b5-z2fvj 1/1 Running 0 1d
  71. ࣗಈճ෮ͤͯ͞ΈΑ͏ 71

  72. ҙਤతʹ1PEΛ࡟আͯ͠ΈΔ 72 # ઌ΄Ͳ૿͑ͨ2ͭΊͷPodΛ࡟আͯ͠ΈΔ $ kubectl delete pod frontend-5c548f4769-vltkv #

    ৽͍͠pod͕Ͱ͖ͯΔʂ $ kubectl get pod NAME READY STATUS RESTARTS AGE frontend-5c548f4769-ns5q2 1/1 Running 0 8s frontend-5c548f4769-xhpxz 1/1 Running 0 1d redis-master-55db5f7567-2n4qp 1/1 Running 0 1d redis-slave-584c66c5b5-z2fvj 1/1 Running 0 1d
  73. %FQMPZNFOUͷ σϓϩΠ؅ཧͬ΀ΓΛݟͯΈΑ͏ 73

  74. 1PEͷઃఆΛม͑ͯΈΔ 74 # Podͷมߋ͕ى͜Βͳ͍ͱཤྺ͕ه࿥͞Εͳ͍(εέʔϧ͡Όμϝ)ͷͰɺࢼ͠ʹ࢖༻ϝϞϦΛม͑ͯΈΔ $ examples/guestbook/frontend-deployment.yaml 23ߦ໨ memory: 100Mi <-

    ͜ΕΛ120Miʹมߋ $ kubectl apply -f examples/guestbook/frontend-deployment.yaml # ঃʑʹ੾ΓସΘ͍ͬͯΔʂ $ kubectl get pod NAME READY STATUS RESTARTS AGE frontend-5c548f4769-ns5q2 1/1 Running 0 11m frontend-5c548f4769-xhpxz 1/1 Running 0 1d frontend-68dd74b969-ztcdw 0/1 ContainerCreating 0 5s redis-master-55db5f7567-2n4qp 1/1 Running 0 1d redis-slave-584c66c5b5-z2fvj 1/1 Running 0 1d $ kubectl get pod NAME READY STATUS RESTARTS AGE frontend-5c548f4769-xhpxz 1/1 Running 0 1d frontend-68dd74b969-6shhj 0/1 ContainerCreating 0 6s frontend-68dd74b969-ztcdw 1/1 Running 0 21s redis-master-55db5f7567-2n4qp 1/1 Running 0 1d redis-slave-584c66c5b5-z2fvj 1/1 Running 0 1d $ kubectl get pod NAME READY STATUS RESTARTS AGE frontend-68dd74b969-6shhj 1/1 Running 0 26s frontend-68dd74b969-ztcdw 1/1 Running 0 41s redis-master-55db5f7567-2n4qp 1/1 Running 0 1d redis-slave-584c66c5b5-z2fvj 1/1 Running 0 1d
  75. ཤྺΛݟͯΈΔ 75 # REVISIONͷ਺஋͕େ͖͍΄͏͕৽͍͠ # CHANGE-CAUSE͸ϚχϑΣετϑΝΠϧʹ"Annotation"ͱݺ͹ΕΔ৘ใΛ෇͚Ճ͑Δͱग़ͯ͘Δ(ࠓճ͸ؾʹ͠ͳ͍) $ kubectl rollout history

    deployments frontend deployment.extensions/frontend REVISION CHANGE-CAUSE 1 <none> 2 <none> # REVISION=2ͷৄࡉΛݟͯΈΔ(ը໘ʹೖΒͳ͍ͷͰखݩͰݟͯʂ) $ kubectl rollout history deployments frontend --revision=2
  76. ϩʔϧόοΫͯ͠ΈΔ 76 # 1ͭલʹ໭Δͱ͖͸"--to-revision"͸লུͰ͖Δ $ kubectl rollout undo deployments frontend

    --to-revision=1 # ΋͏Ұ౓ཤྺΛݟΔͱɺREVISION=1͕ফ͍͑ͯΔ # ϩʔϧόοΫͰ΋ϦϏδϣϯ͸ੵ·ΕΔ # ಉ಺༰ͷϦϏδϣϯ͸ཤྺ͔Βফ͑Δ $ kubectl rollout history deployments frontend deployment.extensions/frontend REVISION CHANGE-CAUSE 2 <none> 3 <none>
  77. ͱ͜ΖͰ.BTUFSͬͯ Կ΍ͬͯΔͷʁ 77

  78. ࠷ॳʹݟͨ͜ͷதʹ͋Δ 78 # ಈ࡞֬ೝɻͳΜ͔͍Ζ͍Ζಈ͍ͯΔ(આ໌͸͋ͱͰ) $ kubectl get pods --namespace=kube-system NAMESPACE

    NAME READY STATUS RESTARTS AGE kube-system etcd-docker-for-desktop 1/1 Running 0 1m kube-system kube-apiserver-docker-for-desktop 1/1 Running 0 1m kube-system kube-controller-manager-docker-for-desktop 1/1 Running 0 1m kube-system kube-dns-86f4d74b45-xb4qh 3/3 Running 0 2m kube-system kube-proxy-8r45p 1/1 Running 0 2m kube-system kube-scheduler-docker-for-desktop 1/1 Running 0 1m
  79. .BTUFS΋1PEͷू·Γ ‣ FUDE  Ϋϥελ಺ͷ͞·͟·ͳσʔλΛอଘ͍ͯ͠ΔҰ؏ੑͷ͋ΔߴՄ༻ੑͷ,74 ‣ LVCFBQJTFSWFS  Ϋϥελʹର͢Δશͯͷૢ࡞Λ࢘Δ"1*αʔόʔ 

    ೝূ΍ೝՄͷॲཧͳͲ΋ߦ͏ ‣ LVCFTDIFEVMFS  1PEͷ/PEF΁ͷׂΓ౰ͯΛߦ͏εέδϡʔϥʔ  1PEΛ഑ஔ͢Δ/PEFͷબ୒΋ߦ͏ ‣ LVCFDPOUSPMMFSNBOBHFS  ֤छ,VCFSOFUFTΦϒδΣΫτͷίϯτϩʔϥʔΛىಈ͠؅ཧ͢ΔϚωʔδϟʔ 79
  80. /PEFʹ΋.BTUFSͱ࿈ܞ͢Δ΋ͷ͕͍Δ ‣ LVCFMFU  /PEFͷϝΠϯॲཧͰ͋Δ1PEͷىಈɾ؅ཧΛߦ͏ΤʔδΣϯτ ‣ LVCFQSPYZ  4FSWJDF͕࣋ͭԾ૝తͳ*1ΞυϨε $MVTUFS*1

    ΁ͷΞΫηεΛ
 ϧʔςΟϯά͢Δ 80
  81. ͭ·Γ͸͜Μͳײ͡ 81 Kubernetes Cluster kubectl (CLI for k8s) ,74 "1*αʔόʔ

    1PEΛ ϊʔυ΁ׂ౰ͯ ֤छ
 ίϯτϩʔϥʔ ϧʔςΟϯά 1PE؅ཧ
 ΤʔδΣϯτ ͢΂ͯͷૢ࡞͸
 APIαʔόʔܦ༝
  82. ϩʔΧϧͰ༡Ϳͷ͸͜͜·Ͱ✋ 82

  83. HVFTUCPPLΞϓϦέʔγϣϯΛ࡟আ 83 # guestbookΞϓϦέʔγϣϯΛ࡟আ $ kubectl delete -f examples/guestbook/ #

    ͠͹Β͘͢Δͱ࡟আ͞ΕΔ $ kubectl get pod No resources found.
  84. ,VCFSOFUFTແޮԽ 84 1SFGFSFODFTը໘ ,VCFSOFUFTλϒΛબ୒ νΣοΫΛ֎͢ "QQMZ 3FTFUλϒΛબ୒ ҰԠϦηοτ

  85. 85 &,4ͰΫϥελΛ࡞ͬͯΈΑ͏

  86. &,4ͬͯͲΜͳαʔϏεʁ 86 Kubernetes Cluster ϚχϑΣετ kubectl (CLI for k8s) =

    Control Plane Data Plane ͜ΕͷϚωʔδυαʔϏε
  87. &,4ͬͯͲΜͳαʔϏεʁ ‣ ෳ਺";ͰNBTUFSΛ৑௕ߏ੒࣮ͯ͠ߦ ‣ NBTUFSͷ؂ࢹɾࣗಈճ෮ ‣ ࣗಈΞοϓάϨʔυɾύονద༻ ‣ ଞͷ"84αʔϏεͱͷ౷߹ ‣

    %BUB1MBOFʢ&$ʣ͸ࣗલͰ༻ҙ͢Δඞཁ͕͋Δ ‣ ͬ͘͟Γඅ༻ײ݄ ݱࡏɾ౦ژϦʔδϣϯ&$අ༻͸ผ్ 87
  88. &,4αʔϏεϩʔϧΛ࡞੒ 88 $ cat << 'EOT' >./eks_iam_role-trust-policy.json { "Version": "2012-10-17",

    "Statement": [ { "Effect": "Allow", "Principal": { "Service": "eks.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } EOT $ aws iam create-role --role-name eksServiceRole --assume-role-policy-document file:// eks_iam_role-trust-policy.json $ aws iam attach-role-policy --role-name eksServiceRole --policy-arn "arn:aws:iam::aws:policy/ AmazonEKSClusterPolicy" $ aws iam attach-role-policy --role-name eksServiceRole --policy-arn "arn:aws:iam::aws:policy/ AmazonEKSServicePolicy" $ export EKS_SERVICE_ROLE=`aws iam get-role --role-name eksServiceRole2 --query "Role.Arn" -- output text`
  89. υΩϡϝϯτʹԊͬͯߏஙͯ͠ΈΑ͏ ‣ https://docs.aws.amazon.com/ja_jp/eks/latest/userguide/getting- started.html ‣ ʜͱ͸ݴ͍ͭͭࠓճ͸खॱΛ֬ೝ͢Δ͚ͩ  ຊདྷͲΜͳखॱ͕ඞཁͳͷ͔͸஌͓ͬͯ͘ͱ͍͍  ΋ͬͱ͍͍΍Γํ͕͋ΔͷͰ࣮ࡍʹ΍Δͷ͸͕࣌ؒ΋͍ͬͨͳ͍

    ‣ ڵຯ͕͋Ε͹΍ͬͯΈͯ 89
  90. $GOͰ71$·ΘΓΛ࡞੒ 90 $ aws cloudformation create-stack --stack-name eks-vpc --template-url https://amazon-eks.s3-us-

    west-2.amazonaws.com/cloudformation/2018-11-07/amazon-eks-vpc-sample.yaml $ export SECURITY_GROUPS=`aws cloudformation describe-stacks --stack-name eks-vpc --query "Stacks[0].Outputs[?OutputKey=='SecurityGroups'].OutputValue | [0]" --output text` $ export VPC_ID=`aws cloudformation describe-stacks --stack-name eks-vpc --query "Stacks[0].Outputs[?OutputKey=='VpcId'].OutputValue | [0]" --output text` $ export SUBNET_IDS=`aws cloudformation describe-stacks --stack-name eks-vpc --query "Stacks[0].Outputs[?OutputKey=='SubnetIds'].OutputValue | [0]" --output text`
  91. $GOͰ71$·ΘΓΛ࡞੒ 91 AWS Cloud VPC 192.168.0.0/16 ap-northeast-1a ap-northeast-1c ap-northeast-1d Internet

    Gateway Subnet01 Subnet02 Subnet03 192.168.64.0/18 192.168.128.0/18 192.168.192.0/18 ap-northeast-1
  92. &,4$MVTUFSΛ࡞੒ 92 $ aws eks create-cluster --name eks-handson --role-arn $EKS_SERVICE_ROLE

    --resources-vpc-config subnetIds=$SUBNET_IDS,securityGroupIds=$SECURITY_GROUPS # CREATING -> ACTIVEʹͳͬͨΒ׬ྃ(8෼͙Β͍) $ aws eks describe-cluster --name eks-handson --query cluster.status
  93. &,4$MVTUFSΛ࡞੒ 93 AWS Cloud VPC 192.168.0.0/16 ap-northeast-1a ap-northeast-1c ap-northeast-1d Internet

    Gateway Subnet01 Subnet02 Subnet03 192.168.64.0/18 192.168.128.0/18 192.168.192.0/18 ap-northeast-1 EKS
  94. LVCFDUMͷ&,4༻DPOpHϑΝΠϧΛ࡞੒ 94 $ aws eks update-kubeconfig --name eks-handson # ಈ࡞֬ೝ

    $ kubectl get all NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 1h
  95. $GOͰ/PEFΛ࡞੒ 95 $ aws cloudformation create-stack --stack-name eks-handson-workernodes --template-url https://

    amazon-eks.s3-us-west-2.amazonaws.com/cloudformation/2018-11-07/amazon-eks-nodegroup.yaml -- parameters ParameterKey=KeyName,ParameterValue=k_mohri ParameterKey=NodeImageId,ParameterValue=ami-07fdc9272ce5b0ce5 ParameterKey=NodeInstanceType,ParameterValue=t2.small ParameterKey=NodeAutoScalingGroupMinSize,ParameterValue=2 ParameterKey=NodeAutoScalingGroupMaxSize,ParameterValue=3 ParameterKey=NodeVolumeSize,ParameterValue=10 ParameterKey=ClusterName,ParameterValue=eks- handson ParameterKey=NodeGroupName,ParameterValue=eks-handson-node-group ParameterKey=ClusterControlPlaneSecurityGroup,ParameterValue=$SECURITY_GROUPS ParameterKey=VpcId,ParameterValue=$VPC_ID ParameterKey=Subnets,ParameterValue='$SUBNET_IDS' -- capabilities CAPABILITY_IAM /PEF*NBHF*E͸͜͜Λࢀর IUUQTEPDTBXTBNB[PODPNFLTMBUFTUVTFSHVJEFFLTPQUJNJ[FEBNJIUNM ೔ຊޠ൛͸ߋ৽͞Ε͍ͯͳ͍ͷͰ஫ҙ
  96. $GOͰ/PEFΛ࡞੒ 96 AWS Cloud VPC 192.168.0.0/16 ap-northeast-1a ap-northeast-1c ap-northeast-1d Internet

    Gateway Subnet01 Subnet02 Subnet03 192.168.64.0/18 192.168.128.0/18 192.168.192.0/18 ap-northeast-1 EKS Auto Scaling Group
  97. /PEFͱ&,4Λ݁߹͢Δ 97 # ΠϯελϯεϩʔϧΛऔಘ $ aws cloudformation describe-stacks --stack-name eks-handson-workernodes

    --query "Stacks[0].Outputs[?OutputKey=='NodeInstanceRole'].OutputValue | [0]" --output text # EKSͱNodeΛ݁߹͢ΔͨΊͷConfigMapΛ࡞੒ $ curl -O https://amazon-eks.s3-us-west-2.amazonaws.com/cloudformation/2018-11-07/aws-auth- cm.yaml $ vi aws-auth-cm.yaml apiVersion: v1 kind: ConfigMap metadata: name: aws-auth namespace: kube-system data: mapRoles: | - rolearn: <͜͜ʹΠϯελϯεϩʔϧΛೖΕΔ> username: system:node:{{EC2PrivateDNSName}} groups: - system:bootstrappers - system:nodes
  98. /PEFͱ&,4Λ݁߹͢Δ 98 # ConfigMapΛద༻ $ kubectl apply -f aws-auth-cm.yaml #

    ͠͹Β͘͢ΔͱNode͕ೝࣝ͞ΕΔ $ kubectl get nodes NAME STATUS ROLES AGE VERSION ip-192-168-103-86.ap-northeast-1.compute.internal Ready <none> 24s v1.11.5 ip-192-168-183-21.ap-northeast-1.compute.internal Ready <none> 27s v1.11.5 ip-192-168-247-164.ap-northeast-1.compute.internal Ready <none> 22s v1.11.5
  99. /PEFͱ&,4Λ݁߹͢Δ 99 AWS Cloud VPC 192.168.0.0/16 ap-northeast-1a ap-northeast-1c ap-northeast-1d Internet

    Gateway Subnet01 Subnet02 Subnet03 192.168.64.0/18 192.168.128.0/18 192.168.192.0/18 ap-northeast-1 EKS Auto Scaling Group
  100. $MVTUFS͕Ͱ͖ͨʂ 100 Kubernetes Cluster Data Plane

  101. αϯϓϧΞϓϦΛσϓϩΠ͢Δ 101 # Ұ౓มߋΛ΋ͱʹ໭͢ $ cd example $ cd git

    reset --hard $ cd ../ # frontend-serviceͷλΠϓΛLoadBalancerʹมߋ # Ϋϥ΢υͷϩʔυόϥϯαʔ(͜͜Ͱ͸ELB)ͱ࿈ܞ͢ΔλΠϓ $ vi examples/guestbook/frontend-service.yaml 9-13ߦ໨ # comment or delete the following line if you want to use a LoadBalancer type: NodePort <- ͜͜ΛίϝϯτΞ΢τ # if your cluster supports it, uncomment the following to automatically create # an external load-balanced IP for the frontend service. # type: LoadBalancer <- ͜͜ΛΞϯίϝϯτ # apply! $ kubectl apply -f examples/guestbook/
  102. αϯϓϧΞϓϦΛσϓϩΠ͢Δ 102 # ͠͹Β͘͢Δͱߏங׬ྃ $ kubectl get all NAME READY

    STATUS RESTARTS AGE pod/frontend-56f7975f44-2vtbr 1/1 Running 0 8s pod/frontend-56f7975f44-j25zn 1/1 Running 0 8s pod/frontend-56f7975f44-mss7q 1/1 Running 0 8s pod/redis-master-6b464554c8-wrjrp 1/1 Running 0 8s pod/redis-slave-b58dc4644-ft2fd 1/1 Running 0 7s pod/redis-slave-b58dc4644-p59fk 1/1 Running 0 7s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/frontend LoadBalancer 10.100.61.11 xxxxxx.ap-northeast-1.elb.amazonaws.com 80:31673/TCP 8s service/kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 23m service/redis-master ClusterIP 10.100.137.217 <none> 6379/TCP 7s service/redis-slave ClusterIP 10.100.217.57 <none> 6379/TCP 7s NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deployment.apps/frontend 3 3 3 3 8s deployment.apps/redis-master 1 1 1 1 8s deployment.apps/redis-slave 2 2 2 2 7s NAME DESIRED CURRENT READY AGE replicaset.apps/frontend-56f7975f44 3 3 3 8s replicaset.apps/redis-master-6b464554c8 1 1 1 8s replicaset.apps/redis-slave-b58dc4644 2 2 2 7s http://xxxxxx.ap-northeast-1.elb.amazonaws.comʹΞΫηεʂ
  103. ΋Ζ΋Ζ࡟আ 103 # Cluster্ʹߏஙͨ͠΋Ζ΋ΖΛ࡟আ $ kubectl delete -f examples/guestbook/ #

    ͠͹Β͘͢Δͱݩ௨Γ $ kubectl get all NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 1h # stack, EKS ClusterΛ࡟আ $ aws cloudformation delete-stack --stack-name eks-handson-workernodes $ aws eks delete-cluster --name eks-handson $ aws cloudformation delete-stack --stack-name eks-vpc
  104. LVCFDUMͷίϯςΩετͷ੾Γସ͑ɾ࡟আ 104 # ίϯςΩετҰཡ $ kubectl config get-contexts CURRENT NAME

    CLUSTER AUTHINFO NAMESPACE * arn:aws:...:cluster/eks-handson arn:aws:...:cluster/eks-handson arn:aws:...:cluster/eks-handson docker-for-desktop docker-for-desktop-cluster docker-for-desktop # ίϯςΩετΛdocker-for-desktopʹ໭͢ $ kubectl config use-context docker-for-desktop # EKSͷίϯςΩετΛ࡟আ $ kubectl config delete-context arn:aws:...:cluster/eks-handson
  105. ΊΜͲ͗͘͢͞ͳ͍ʁ 105

  106. ΋͏ͪΐͬͱ͏·͍΍Γํ ‣ FLTDUM ඇެࣜɾσϑΝΫτελϯμʔυ   https://eksctl.io/  ίϚϯυҰͭͰΫϥελߏங ‣

    ΫΠοΫελʔτ ެࣜɾ࠷ۙग़ͨ   https://aws.amazon.com/jp/quickstart/architecture/amazon-eks/  $GOΛ࢖ͬͯΫϥελߏங  ϕετϓϥΫςΟεʹै͍ͬͯΔͷͰ
 ݁ߏ߽՚ͳߏ੒ 106
  107. 107 FLTDUMΛ࢖ͬͯΈΑ͏

  108. FLTDUMͱ͸ʁ ‣ ίϚϯυҰͭͰ&,4ͷ$MVTUFS͕Ͱ͖ͪΌ͏πʔϧ ‣ $GOͷςϯϓϨʔτΛࣗಈੜ੒ͯ͠ߏஙͯ͠Δ ‣ OPEFͷΦʔτεέʔϦϯάͳͲɺศརͳػೳ΋ ‣ ͱ͸͍͑·ͩൃల్্ ‣

    ͝ཡͷͱ͓ΓHPͰͰ͖ͯ·͢ 108
  109. ΍ͬͯΈΑ͏ʂ ‣ ͜Ε͚ͩʂ ‣ ͦͷଞΦϓγϣϯ͸ϔϧϓΛࢀর 109 $ eksctl create cluster

    \ --name eksctl-handson \ --region ap-northeast-1 \ --nodes 3 \ --nodes-min 3 \ --nodes-max 3 \ --node-type t2.medium \ --ssh-public-key <ΩʔϖΞ໊> $ eksctl create cluster -h ˞ͨͩ͠ߏங׬ྃ·Ͱ෼͙Β͍͔͔Γ·͢ʜ ˞BQOPSUIFBTUC͕બ୒Ͱ͖Δݹ͍"84ΞΧ΢ϯτ͸ɺ";ࢦఆ΋ඞཁ
  110. ͖ͬ͞ͷͱ΄΅ಉ͡ͷ͕Ͱ͖Δ 110 AWS Cloud VPC ap-northeast-1a ap-northeast-1c ap-northeast-1d Internet Gateway

    PublicSubnet01 PublicSubnet02 PublicSubnet03 ap-northeast-1 EKS Auto Scaling Group NAT Gateway ΦϓγϣϯʹΑͬͯ PrivateSubnetʹNodeΛ ഑ஔՄೳͳͷͰɺ ͦͷͱ͖༻ʁ PrivateSubnet01 PrivateSubnet02 PrivateSubnet03
  111. ͱΓ͋͑ͣσϓϩΠ͠Α͏ 111

  112. αϯϓϧΞϓϦΛσϓϩΠ͢Δ 112 # Ұ౓มߋΛ΋ͱʹ໭͢ $ cd examples $ git reset

    --hard $ cd ../ # frontend-serviceͷλΠϓΛLoadBalancerʹมߋ # Ϋϥ΢υͷϩʔυόϥϯαʔ(͜͜Ͱ͸ELB)ͱ࿈ܞ͢ΔλΠϓ $ vi examples/guestbook/frontend-service.yaml 9-13ߦ໨ # comment or delete the following line if you want to use a LoadBalancer type: NodePort <- ͜͜ΛίϝϯτΞ΢τ # if your cluster supports it, uncomment the following to automatically create # an external load-balanced IP for the frontend service. # type: LoadBalancer <- ͜͜ΛΞϯίϝϯτ # apply! $ kubectl apply -f examples/guestbook/
  113. αϯϓϧΞϓϦΛσϓϩΠ͢Δ 113 # ͠͹Β͘͢Δͱߏங׬ྃ $ kubectl get all NAME READY

    STATUS RESTARTS AGE pod/frontend-56f7975f44-2vtbr 1/1 Running 0 8s pod/frontend-56f7975f44-j25zn 1/1 Running 0 8s pod/frontend-56f7975f44-mss7q 1/1 Running 0 8s pod/redis-master-6b464554c8-wrjrp 1/1 Running 0 8s pod/redis-slave-b58dc4644-ft2fd 1/1 Running 0 7s pod/redis-slave-b58dc4644-p59fk 1/1 Running 0 7s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/frontend LoadBalancer 10.100.61.11 xxxxxx.ap-northeast-1.elb.amazonaws.com 80:31673/TCP 8s service/kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 23m service/redis-master ClusterIP 10.100.137.217 <none> 6379/TCP 7s service/redis-slave ClusterIP 10.100.217.57 <none> 6379/TCP 7s NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deployment.apps/frontend 3 3 3 3 8s deployment.apps/redis-master 1 1 1 1 8s deployment.apps/redis-slave 2 2 2 2 7s NAME DESIRED CURRENT READY AGE replicaset.apps/frontend-56f7975f44 3 3 3 8s replicaset.apps/redis-master-6b464554c8 1 1 1 8s replicaset.apps/redis-slave-b58dc4644 2 2 2 7s http://xxxxxx.ap-northeast-1.elb.amazonaws.comʹΞΫηεʂ
  114. μογϡϘʔυΛ Πϯετʔϧͯ͠ΈΑ͏ 114

  115. μογϡϘʔυͷΠϯετʔϧ 115 # μογϡϘʔυ༻ͷPodΛapply $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/ recommended/kubernetes-dashboard.yaml

    $ μογϡϘʔυʹϩάΠϯ͢ΔͨΊͷτʔΫϯΛऔಘ $ aws-iam-authenticator token -i eksctl-handson | jq -r '.status.token' # ϓϩΩγܦ༝ͰμογϡϘʔυʹΞΫηε $ kubectl proxy --port=8000 --address='0.0.0.0' --disable-filter=true http://localhost:8000/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ ͜͜ʹτʔΫϯΛೖྗͯ͠
 αΠϯΠϯ
  116. ͜Μͳײ͡ 116

  117. ϩάΛऩूͯ͠ΈΑ͏ 117

  118. ,VCFSOFUFTͷϩάΛ$MPVE8BUDI-PHT΁ ‣ %PDLFSίϯςφͰ͸ɺඪ४ग़ྗ͕ϩάͱͯ͠ѻΘΕΔ ‣ ϩά͸σϑΥϧτͰKTPOܕࣜͷϑΝΠϧʹͳΔ ‣ ֤/PEFʹqVFOUEͷ1PEΛஔ͖ɺ/PEF಺ͷ1PE͕ग़ྗͨ͠ϩάϑΝΠϧΛऩूͯ͠
 $MPVE8BUDIʹૹΔ 118 CloudWatchLogs

  119. ,VCFSOFUFTͷϩάΛ$MPVE8BUDI-PHT΁ ‣ %PDLFSίϯςφͰ͸ɺඪ४ग़ྗ͕ϩάͱͯ͠ѻΘΕΔ ‣ ϩά͸σϑΥϧτͰKTPOܕࣜͷϑΝΠϧʹͳΔ ‣ ֤/PEFʹqVFOUEͷ1PEΛஔ͖ɺ/PEF಺ͷ1PE͕ग़ྗͨ͠ϩάϑΝΠϧΛऩूͯ͠
 $MPVE8BUDIʹૹΔ 119 CloudWatchLogs

    Ͳ͏΍ͬͯʜʁ
  120. %BFNPO4FU ‣ Ϋϥελ಺ͷ֤/PEFʹରͯ͠ඞͣͭ഑ஔ͞ΕΔ1PEΛ؅ཧ  ֤/PEFͷΤʔδΣϯτͳͲͷ༻్ʹ 120 CloudWatchLogs

  121. *".3PMFͷ͓ੈ࿩ 121 # NodeͷEC2ʹඥ෇͍͍ͯΔIAM Roleͷ໊લΛऔಘ $ INSTANCE_PROFILE_NAME=$(aws iam list-instance-profiles |

    jq -r '.InstanceProfiles[].InstanceProfileName' | grep nodegroup) $ ROLE_NAME=$(aws iam get-instance-profile --instance-profile-name $INSTANCE_PROFILE_NAME | jq -r '.InstanceProfile.Roles[] | .RoleName') # IAM Roleʹϩάऩू༻ͷΠϯϥΠϯϙϦγʔΛ௥Ճ $ cat << "EoF" > ./k8s-logs-policy.json { "Version": "2012-10-17", "Statement": [ { "Action": [ "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": "*", "Effect": "Allow" } ] } EoF $ aws iam put-role-policy --role-name $ROLE_NAME --policy-name Logs-Policy-For-Worker --policy-document file://k8s- logs-policy.json
  122. qVFOUEΛσϓϩΠ 122 # ϚχϑΣετϑΝΠϧΛऔಘ $ wget https://eksworkshop.com/logging/deploy.files/fluentd.yml # Ϋϥελ໊Λมߋ $

    vi fluentd.yml 197ߦ໨ value: us-east-1 <- ap-northeast-1 ʹมߋ 199ߦ໨ value: eksworkshop-eksctl <- eksctl-handson ʹมߋ # σϓϩΠ $ kubectl apply -f fluentd.yml
  123. $MPVE8BUDIͷϩάʹ্͕ͬͯ͘Δ 123 ͋ͱ͸&MBTUJDTFBSDI4FSWJDFʹྲྀ͢ͳΓԿͳΓͱ

  124. qVFOUEZNM 124 (ུ) --- apiVersion: extensions/v1beta1 kind: DaemonSet # DaemonSetͷϚχϑΣετ

    metadata: name: fluentd-cloudwatch namespace: kube-system labels: k8s-app: fluentd-cloudwatch spec: template: # ---͔͜͜ΒPodͷఆٛ-------------------------------------------- metadata: labels: k8s-app: fluentd-cloudwatch spec: serviceAccountName: fluentd terminationGracePeriodSeconds: 30 # Because the image's entrypoint requires to write on /fluentd/etc but we mount configmap there which is read-only, # this initContainers workaround or other is needed. # See https://github.com/fluent/fluentd-kubernetes-daemonset/issues/90 initContainers: - name: copy-fluentd-config image: busybox command: ['sh', '-c', 'cp /config-volume/..data/* /fluentd/etc'] volumeMounts: - name: config-volume mountPath: /config-volume - name: fluentdconf mountPath: /fluentd/etc ʢུʣ
  125. )FMNΛ࢖ͬͯΈΑ͏⛑ 125

  126. )FMN ‣ ,VCFSOFUFT༻ͷύοέʔδ؅ཧπʔϧ  ύοέʔδ͸$IBSUͱݺ͹ΕɺϚχϑΣετϑΝΠϧͷςϯϓϨʔτؚ͕·ΕΔ  5JMMFSͱݺ͹ΕΔαʔόʔΞϓϦέʔγϣϯ ͜Ε΋1PE Λհͯ͠Ϋϥελ಺ʹύοέʔδΛΠϯετʔϧ ‣

    ͪͳΈʹIFMN͸דͰ͸ͳ͘ધͷ଩ɺDIBSU͸ւਤɺUJMMFS͸଩ฑͷҙຯ 126 helm (CLI) Tiller Repository Chart A Chart B Chart C Manifest Files Custom Values
  127. 3PMF#BTFE"DDFTT$POUSPM 3#"$ ‣ ,VCFSOFUFTͷݖݶ੍ޚͷ࢓૊Έ  ,VCFSOFUFTͷϦιʔε΁ͷΞΫηεΛϩʔϧʹΑ੍ͬͯޚ  ϢʔβʔͱϩʔϧΛ#JOEJOHʹΑͬͯඥ෇͚Δ͜ͱʹΑͬͯػೳ͢Δ ‣ Ϣʔβʔछผ

     ೝূϢʔβʔɾάϧʔϓΫϥελ֎͔Β,VCFSOFUFT"1*Λૢ࡞͢ΔͨΊͷϢʔβʔ  4FSWJDF"DDPVOU1PE͕,VCFSOFUFT"1*Λૢ࡞͢ΔͨΊͷϢʔβʔ ‣ ϩʔϧछผ  3PMFࢦఆͷOBNFTQBDF಺ͰͷΈ༗ޮ  $MVTUFS3PMFΫϥελશମͰ༗ޮ ‣ )FMNʹ΋3#"$Λ༗ޮʹͰ͖Δ$IBSU͕
 ଟ͘؅ཧ͞Ε͍ͯΔ 127 ServiceAccount 認証ユーザーグループ 認証ユーザー RoleBinding ClusterRoleBinding Role ClusterRole
  128. )FMNͷΠϯετʔϧ ‣ https://helm.sh/docs/using_helm/#installing-helm ‣ .BDͷ৔߹  CSFXJOTUBMMLVCFSOFUFTIFMN 128

  129. )FMNͷηοτΞοϓ 129 # Tiller༻ͷαʔϏεΞΧ΢ϯτͷϚχϑΣετϑΝΠϧΛ࡞੒ # "cluster-admin"͸σϑΥϧτͰଘࡏ͢ΔClusterRole $ cat <<EoF >

    tiller_rbac.yaml --- apiVersion: v1 kind: ServiceAccount metadata: name: tiller namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: tiller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: tiller namespace: kube-system EoF # Tiller༻ͷαʔϏεΞΧ΢ϯτΛ࡞੒ $ kubectl apply -f tiller_rbac.yaml # TillerͷαʔϏεΞΧ΢ϯτΛࢦఆͯ͠ΫϥελʹHelmΛಋೖ # ͜ΕͰTillerͷPod͕kube-systemωʔϜεϖʔεʹσϓϩΠ͞ΕΔ $ helm init --service-account tiller
  130. )FMNͰ+FOLJOTΛΠϯετʔϧ 130 # CustomValueϑΝΠϧΛ࡞੒ # ύϥϝʔλʔৄࡉ͸ "helm inspect values stable/jenkins"

    Ͱ֬ೝ $ cat <<EoF > jenkins.yaml rbac: create: true master: service_port: 8080 persistence: size: 1Gi EoF # JenkinsΛΠϯετʔϧ $ helm install -f jenkins.yaml --name jenkins stable/jenkins # ͠͹Β͘͢ΔͱσϓϩΠ׬ྃ(2-3෼͙Β͍) $ kubectl get all NAME READY STATUS RESTARTS AGE pod/jenkins-f65b9477-89s69 1/1 Running 0 33m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/jenkins LoadBalancer 10.100.22.208 xxxxxx.ap-northeast-1.elb.amazonaws.com 8081:30196/TCP 33m service/jenkins-agent ClusterIP 10.100.58.26 <none> 50000/TCP 33m service/kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 2h NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deployment.apps/jenkins 1 1 1 1 33m NAME DESIRED CURRENT READY AGE replicaset.apps/jenkins-f65b9477 1 1 1 33m
  131. Πϯετʔϧͨ͠+FOLJOTʹϩάΠϯ 131 # ύεϫʔυऔಘ $ printf $(kubectl get secret --namespace

    default jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo XXXXXXXX # ϩάΠϯURLऔಘ $ export SERVICE_IP=$(kubectl get svc --namespace default jenkins --template "{{ range (index .status.loadBalancer.ingress 0) }} {{ . }}{{ end }}") $ echo http://$SERVICE_IP:8080/login http://xxxxxx.ap-northeast-1.elb.amazonaws.com:8080/login VTFSOBNFBENJO QBTTXPSEίϚϯυͰऔಘͨ͠΍ͭ
  132. +FOLJOTΛΞϯΠϯετʔϧ 132 # JenkinsΛΞϯΠϯετʔϧ # --purge͸Φϓγϣϯɻ෇͚ͳ͍৔߹ɺϦϏδϣϯͷه࿥͕࢒ΓɺϩʔϧόοΫ͕Ͱ͖Δ $ helm delete --purge

    jenkins
  133. ؂ࢹͰ͖ΔΑ͏ʹͯ͠ΈΑ͏2 133

  134. 1SPNFUIFVT(SBGBOB ‣ 1SPNFUIFVT  044ͷϦιʔε؂ࢹπʔϧ  ಋೖ͕Χϯλϯɺ͍͍ײ͡ʹ௨஌͘ΕΔɺߴੑೳͳͲͰਓؾ͕ߴ͍  ͨͩɺσʔλͷՄࢹԽ͕ຊۀͰ͸ͳ͍ͷͰྗෆ଍ ‣

    (SBGBOB  044ͷϩάɾσʔλՄࢹԽπʔϧ  1SPNFUIFVT͕ऩूͨ͠σʔλΛ͔ͬ͜Α͘දࣔͰ͖Δ 134
  135. )FMNͰ1SPNFUIFVTΛΠϯετʔϧ 135 # CustomValueϑΝΠϧΛ࡞੒ # ύϥϝʔλʔৄࡉ͸ "helm inspect values stable/prometheus"

    Ͱ֬ೝ $ cat <<EoF > prometheus.yaml alertmanager: persistentVolume: size: 1Gi storageClass: "gp2" server: persistentVolume: size: 1Gi storageClass: "gp2" retention: "12h" pushgateway: enabled: false EoF # PrometheusΛΠϯετʔϧ $ kubectl create namespace prometheus $ helm install -f prometheus.yaml --name prometheus --namespace prometheus stable/prometheus # σϓϩΠ͕׬ྃͨ͠ΒΞΫηεͯ͠ΈΔʢ্ཱ͕ͪΔ·Ͱ਺෼͔͔Δʣ # URLͷऔಘίϚϯυ͕Α͘มΘΔͷͰɺhelm installͰग़͖ͯͨϩάͷํΛࢀߟʹͨ͠΄͏͕͍͍͔΋ $ export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}") $ kubectl --namespace prometheus port-forward $POD_NAME 9090
  136. 1SPNFUIFVTʹΞΫηε 136 http://localhost:9090/targets

  137. )FMNͰ(SBGBOBΛΠϯετʔϧ 137 # CustomValueϑΝΠϧΛ࡞੒ # ύϥϝʔλʔৄࡉ͸ "helm inspect values stable/grafana"

    Ͱ֬ೝ $ cat <<EoF > grafana.yaml persistence: storageClassName: gp2 adminPassword: password datasources: datasources.yaml: apiVersion: 1 datasources: - name: Prometheus type: prometheus url: "http://prometheus-server.prometheus.svc.cluster.local" access: proxy isDefault: true service: type: LoadBalancer EoF # GrafanaΛΠϯετʔϧ $ kubectl create namespace grafana $ helm install -f grafana.yaml --name grafana --namespace grafana stable/grafana # σϓϩΠ͕׬ྃͨ͠ΒΞΫηεͯ͠ΈΔʢ্ཱ͕ͪΔ·Ͱ਺෼͔͔Δʣ # URLͷऔಘίϚϯυ͕Α͘มΘΔͷͰɺhelm installͰग़͖ͯͨϩάͷํΛࢀߟʹͨ͠΄͏͕͍͍͔΋ $ export ELB=$(kubectl get svc -n grafana grafana -o jsonpath='{.status.loadBalancer.ingress[0].hostname}') $ echo "http://$ELB" http://xxxxxx.ap-northeast-1.elb.amazonaws.com
  138. (SBGBOBʹΞΫηε 138 VTFSOBNFBENJO QBTTXPSEQBTTXPSE HSBGBOBZBNMʹॻ͍ͨ

  139. μογϡϘʔυͷ*%Λೖྗ 139 Λೖྗ ެ։͞Ε͍ͯΔςϯϓϨʔτͷ൪߸ ΫϦοΫ

  140. Φϓγϣϯઃఆ 140 ΫϦοΫ 1SPNFUIFVTΛબ୒

  141. ͦΕΒ͍͠ͷ͕ग़͖ͯͨʂ 141

  142. ͳΒ͜Μͳײ͡ 142

  143. ͓ย෇͚ 143

  144. ͓ย෇͚ 144 # PrometheusͱGrafanaΛ࡟আ # EBSΛ࡞͍ͬͯΔͷͰɺ͍͖ͳΓeksctlͰΫϥελΛ࡟আ͢ΔͱEBS͕࢒ͬͯ͠·͏ $ helm delete --purge

    prometheus $ helm delete --purge grafana # fluentdΛ࡟আ $ kubectl delete -f fluentd.yml # μογϡϘʔυΛ࡟আ $ kubectl delete -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/ kubernetes-dashboard.yaml # guestbookΞϓϦΛ࡟আ $ kubectl delete -f examples/guestbook/ # ϩάऩू༻ʹIAM Roleʹ͚ͬͭͨ͘PolicyΛ֎͢ # $ROLE_NAME͸121ϖʔδͰઃఆ͞Ε͍ͯΔ # ֎͓͔ͯ͠ͳ͍ͱeksctlͰΫϥελΛ࡟আ͢Δͱ͖ʹcloudformationʹౖΒΕΔ $ aws iam delete-role-policy --role-name $ROLE_NAME --policy-name Logs-Policy-For-Worker # eksctlͰΫϥελΛ࡟আ # ్த·Ͱ͔͠ݟͯ͘Εͳ͍ͷͰɺAWSίϯιʔϧͰຊ౰ʹ࡟আ͞Ε͔ͨ֬ೝͨ͠΄͏͕͍͍ $ eksctl delete cluster --name eksctl-handson
  145. 145 ͓·͚  (,&ͰΫϥελΛ࡞ͬͯΈΑ͏

  146. (,&ͰΫϥελΛ࡞Δ ‣ ΍ͬͺΓຊՈ΋ݟ͓͔ͯͳ͍ͱɺͱ͍͏͜ͱͰͪΐͬͱ͚ͩ ‣ $-*΍ϓϩδΣΫτͷઃఆ͸ऴΘ͍ͬͯΔલఏ 146

  147. (,&ͰΫϥελΛ࡞Δ 147 # ίϚϯυ1ͭͰNode΋ؚΊͯΫϥελΛ࡞ͬͯ͘ΕΔ # 3෼൒͙Β͍ͰͰ͖Δʂʂ $ gcloud container clusters

    create gke-handson --cluster-version=1.12.7-gke.10 --machine-type=n1-standard-1 --num-nodes=3 # kubectlͷconfig΋มΘͬͯΔ
 $ kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE docker-for-desktop docker-for-desktop-cluster docker-for-desktop * xxxxxx_asia-northeast1-a_gke-handson xxxxxx_asia-northeast1-a_gke-handson xxxxxx_asia-northeast1-a_gke-handson # σϑΥϧτͰfluentd΍prometheus͕ೖͬͯΔ!? $ kubectl get pods --namespace=kube-system NAME READY STATUS RESTARTS AGE event-exporter-v0.2.3-f9c896d75-52nkr 2/2 Running 0 2m5s fluentd-gcp-scaler-69d79984cb-zm56b 1/1 Running 0 113s fluentd-gcp-v3.2.0-5mncb 2/2 Running 0 63s fluentd-gcp-v3.2.0-9sdg7 2/2 Running 0 74s fluentd-gcp-v3.2.0-t59sr 2/2 Running 0 54s heapster-v1.6.0-beta.1-6fc8df6cb8-54qrk 3/3 Running 0 85s kube-dns-autoscaler-76fcd5f658-22l8j 1/1 Running 0 104s kube-dns-b46cc9485-5kspm 4/4 Running 0 92s kube-dns-b46cc9485-j8fmn 4/4 Running 0 2m5s kube-proxy-gke-gke-handson-default-pool-d757b1ec-9ld7 1/1 Running 0 108s kube-proxy-gke-gke-handson-default-pool-d757b1ec-lcl8 1/1 Running 0 110s kube-proxy-gke-gke-handson-default-pool-d757b1ec-z2m6 1/1 Running 0 110s l7-default-backend-6f8697844f-s8lgv 1/1 Running 0 2m6s metrics-server-v0.3.1-5b4d6d8d98-tn8cw 2/2 Running 0 87s prometheus-to-sd-4v26j 1/1 Running 0 111s prometheus-to-sd-k4jj7 1/1 Running 0 110s prometheus-to-sd-mhs8h 1/1 Running 0 110s
  148. /PEFͷΫϥελ͕Ͱ͖ͨ 148

  149. ΞϓϦέʔγϣϯΛσϓϩΠ 149 # ΋ͪΖΜࠓ·Ͱ࢖͖ͬͯͨϚχϑΣετϑΝΠϧ͕࢖͑Δ $ kubectl apply -f examples/guestbook/

  150. ͓ย෇͚ 150 # ࡟আ΋3෼൒͙Β͍ͰऴΘΔʂʂ $ gcloud container clusters delete gke-handson

    # kubectlͷconfig΋ফ͑ͯΔ
 $ kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE docker-for-desktop docker-for-desktop-cluster docker-for-desktop # ೦ͷͨΊ࢒͍ͬͯΔϦιʔε͕ແ͍͔ίϯιʔϧͰ֬ೝ͓ͯ͘͠ͱ٢
  151. &,4ͱ(,& ‣ ॊೈੑ  &,4͸ϢʔβʔʹΑΔΧελϚΠζͷ෯͕େ͖͍ w Ͱ΋΍ͬͺΓ໘౗ͳͷͰFLTDUMͱ͔͕஀ੜ͍ͯͨ͠Γ͢Δʜ  (,&͸͍͍ײ͡ʹͯ͘͠ΕΔ ‣

    ্ཱ͕ͪΓͷ଎͞  &,4͸ FLTDUMར༻Ͱ ෼͙Β͍  (,&͸෼൒͙Β͍ ‣ අ༻  &,4͸NBTUFSʹඅ༻͕͔͔Δ ౦ژϦʔδϣϯͰ݄͙Β͍   (,&͸NBTUFSʹඅ༻͕͔͔Βͳ͍ʂ ‣ ͲͬͪΛબ΅͏ʁ  ७ਮʹ,VCFSOFUFTΛ࢖͍͍͚ͨͩͳΒѹ౗తʹ(,&  "84ͷ΋Ζ΋ΖͷαʔϏεͱ߹Θͤͯ࢖͍͍ͨͳΒ&,4 151
  152. 152 ͦͷଞ΋Ζ΋Ζ

  153. ొ৔͠ͳ͔ͬͨϦιʔεͨͪ 153

  154. +PC ‣ ୯ൃͷॲཧΛ؅ཧ  ࢦఆͨ͠਺͚ͩ1PEΛ࡞੒ͯ͠ॲཧΛ࣮ߦ 154 apiVersion: batch/v1 kind: Job

    # JobͷϚχϑΣετ metadata: name: example_job labels: app: example spec: parallelism: 3 # ಉ࣌ʹ࣮ߦ͢ΔPodͷ਺ template: # ---͔͜͜ΒPodͷఆٛ-------------------------------------------- metadata: labels: app: example spec: (ུ)
  155. $SPO+PC ‣ ఆظ࣮ߦ͢ΔॲཧΛ؅ཧ  εέδϡʔϧʹԊͬͯ1PEΛ࡞੒ͯ͠ॲཧΛ࣮ߦ 155 apiVersion: batch/v1beta1 kind: CronJob

    # CronJobͷϚχϑΣετ metadata: name: example_job labels: app: example spec: schedule: "*/1 * * * *" # ىಈεέδϡʔϧΛcronͱಉ͡ܕࣜͰఆٛ jobTemplate:
 spec: template: # ---͔͜͜ΒPodͷఆٛ-------------------------------------------- metadata: labels: app: example spec: (ུ)
  156. $POpH.BQ ‣ ΞϓϦέʔγϣϯͷઃఆ৘ใΛఆٛͯ͠1PEʹఏڙ  ؀ڥม਺ͱͯ͠ఏڙ  7PMVNFͱͯ͠ఏڙ 156

  157. $POpH.BQ 157 apiVersion: v1 kind: ConfigMap metadata: name: cm-example data:

    # key-valueܕࣜͰઃఆ৘ใΛॻ͍͍ͯ͘ EXAMPLE: this_is_example example.txt: | this is example # Podͷఆٛத (ུ) env: # ؀ڥม਺ͱͯ͠ఏڙ - name: EXAMPLE valueFrom: configMapKeyRef: name: cm-example key: EXAMPLE (ུ) containers: # Volumeͱͯ͠ఏڙɻ͜ΕͰ /config/example.txt ͕ѻ͑ΔΑ͏ʹͳΔ - image: alpine (ུ) volumeMounts: ## ίϯςφ಺ͰͷVolumeͷϚ΢ϯτઃఆ - name: cm-volume mountPath: /config volumes: ## Volumeͷఆٛ - name: cm-volume configMap: name: cm-example (ུ)
  158. 4FDSFU ‣ ΞϓϦέʔγϣϯͷػີ৘ใΛఆٛͯ͠1PEʹఏڙ  ؀ڥม਺ͱͯ͠ఏڙ  7PMVNFͱͯ͠ఏڙ ‣ $POpH.BQͱͷҧ͍ 

    จࣈྻΛ#BTFΤϯίʔυͨ͠ঢ়ଶͰѻ͏ όΠφϦσʔλΛѻ͑ΔΑ͏ʹ  w ΋ͪΖΜ҉߸Խ໨త͡Όͳ͍ͷͰɺͦͷ··(JUIVCʹ্͛ͨΓͨ͠Βμϝ  ઃఆʹΑͬͯ FUDE্ʹ҉߸Խͨ͠ঢ়ଶͰอଘ͞ΕΔ  /PEFͰ͸1PEͷUNQGT ཁ͢ΔʹϝϞϦ ্ʹอଘ͞ΕΔ  /PEF͸ׂΓ౰ͯΒΕͨ1PE͕ࢀর͢Δ4FDSFUҎ֎͸ΞΫηεͰ͖ͳ͍  ͍͔ͭ͘5ZQF͕͋Δ w 0QBRVF$POpH.BQͱಉ͡ߏ଄Խ͞Εͯͳ͍,FZ7BMVFܗࣜ w LVCFSOFUFTJPUMT5-4ͷൿີݤͱެ։ݤΛ֨ೲ w LVCFSOFUFTJPTFSWJDFBDDPVOUUPLFO,VCFSOFUFTͷαʔϏεΞΧ΢ϯτͷΫϨσϯγϟϧ 158
  159. 4FDSFU 159 apiVersion: v1 kind: Secret metadata: name: secret-example stringData:

    password: xxxxxxxxxxxx # Base64Τϯίʔυ͞Εͨจࣈྻ credential.txt: | xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx # Podͷఆٛத (ུ) env: # ؀ڥม਺ͱͯ͠ఏڙ - name: PASSWORD valueFrom: secretKeyRef: name: secret-example key: password (ུ) containers: # Volumeͱͯ͠ఏڙɻ͜ΕͰ /secrets/credential.txt ͕ѻ͑ΔΑ͏ʹͳΔ - image: alpine (ུ) volumeMounts: ## ίϯςφ಺ͰͷVolumeͷϚ΢ϯτઃఆ - name: secret-volume mountPath: /secrets volumes: ## Volumeͷఆٛ - name: secret-volume secret: secretName: secret-data (ུ)
  160. ετϨʔδؔ࿈΋Ζ΋Ζ ‣ 1FSTJTUFOU7PMVNF  ετϨʔδͷ࣮ମ  "84ͳΒ/PEFͷ&$͕࣋ͭ&#4 ‣ 1FSTJTUFOU7PMVNF$MBJN 

    ετϨʔδΛ࿦ཧతʹந৅Խͨ͠Ϧιʔε  1FSTJTUFOU7PMVNFʹରͯ͠ඞཁͳ༰ྔΛಈతʹ֬อ ‣ 4UPSBHF$MBTT  1FSTJTUFOU7PMVNF͕֬อ͢ΔετϨʔδͷछྨΛఆٛ  "84ͳΒJP HQTDTU ‣ 4UBUFGVM4FU  ܧଓతʹσʔλΛӬଓԽ͢ΔεςʔτϑϧͳΞϓϦέʔγϣϯͷ؅ཧʹ޲͍ͨϦιʔε  ؅ཧԼͷ1PEʹ͸࿈൪ͷࣝผࢠ͕෇༩͞Εɺ࠶࡞੒͞Εͯ΋ಉࣝ͡ผࢠͰ͋Ε͹ಉ͡ετϨʔδΛࢀর͢Δ ‣ Ͱ΋ɺΫϥ΢υͰ,VCFSOFUFT࢖͏ͳΒجຊతʹϚωʔδυͷ%#αʔϏεΛ࢖͏ 160
  161. ϚχϑΣετϑΝΠϧͷ ϑΥʔϚοτΛௐ΂Δ 161

  162. ϚχϑΣετϑΝΠϧͷϑΥʔϚοτ 162 # "kubectl explain [Ϧιʔε] --recursive" ͰશମͷϑΥʔϚοτΛදࣔ $ kubectl

    explain service --recursive KIND: Service VERSION: v1 DESCRIPTION: Service is a named abstraction of software service (for example, mysql) consisting of local port (for example 3306) that the proxy listens on, and the selector that determines which pods will answer requests sent through the proxy. FIELDS: apiVersion <string> kind <string> metadata <Object> annotations <map[string]string> (ུ) # ֊૚Λࢦఆ͢Δͱͦͷ߲໨ͷઆ໌͕֬ೝͰ͖Δ $ kubectl explain service.spec.type KIND: Service VERSION: v1 FIELD: type <string> DESCRIPTION: type determines how the Service is exposed. Defaults to ClusterIP. Valid options are ExternalName, ClusterIP, NodePort, and LoadBalancer. (ུ)
  163. ӡ༻࣌ͷجຊߏ੒Πϝʔδ 163

  164. ӡ༻࣌ͷجຊߏ੒Πϝʔδ 164 GitHub CircleCI DockerImage Deploy ECR DockerHub CloudWatchLogs Prometheus

    Grafana Log Metrics Kubernetes Cluster CI/CD ؂ࢹ ϩά؅ཧ
  165. 165 ·ͱΊ

  166. ຊ೔ͷ໨ඪ ࠶ܝ ‣ ,VCFSOFUFTͱ͓༑ୡʹͳΔ  ΠϝʔδΛ௫Ή  ৮ͬͯΈΔ ϩʔΧϧɾ&,4ɾͪΐͬͱ(,& 

     ߏஙɾӡ༻͕Ͱ͖ΔΑ͏ͳؾ෼ʹͳΔ ‣ ޽ʹ͋;ΕΔ,VCFSOFUFTͷهࣄɾεϥΠυ͕
 ཧղͰ͖ΔΑ͏ʹͳΔ 166
  167. ୡ੒Ͱ͖ͨͰ͠ΐ͏͔ 167

  168. ͱ͸͍͑ʜ ‣ ,VCFSOFUFT͸େن໛ͰෳࡶͳγεςϜͰਅՁΛൃش ‣ ୯७ͳ΋ͷͳΒ'BSHBUFͱ͔&$4ͱ͔ͷ΄͏͕ɻɻ  ৔߹ʹΑͬͯ͸&$Λ࢖ͬͨϨΨγʔͳߏ੒Ͱ΋ࣄ଍ΓΔ 168

  169. ͱ͸͍͑ʜ ‣ ,VCFSOFUFT͸େن໛ͰෳࡶͳγεςϜͰਅՁΛൃش ‣ ୯७ͳ΋ͷͳΒ'BSHBUFͱ͔&$4ͱ͔ͷ΄͏͕ɻɻ  ৔߹ʹΑͬͯ͸&$Λ࢖ͬͨϨΨγʔͳߏ੒Ͱ΋ࣄ଍ΓΔ ‣ Ͱ΋ɺ਺೥ޙʹ͸౰ͨΓલͷٕज़ʹͳ͍ͬͯΔՄೳੑ͕ߴ͍ 

    ʜͱࢥ͍ͬͯΔ  ΋ͬͱΧϯλϯʹ࢖͑ΔΑ͏ʹͳ͍ͬͯΔ͔΋ʜ  &,4ͷ'BSHBUFରԠ΋͋Δͱ͔ͳ͍ͱ͔ʜ ‣ ݱ࣌఺Ͱ΋ࣗ෼ͷબ୒ࢶ͕૿͑Δͷ͸͍͍͜ͱͩΑͶʂ 169
  170. ·ͱΊ ‣ ͱΓ͋͑ͣ໨ඪ͕ୡ੒Ͱ͖͍ͯͨͳΒ޾͍Ͱ͢ ‣ ,VCFSOFUFTপͷೖΓޱʹཱͬͨ͹͔Γ  ࠓճ঺հ͖͠Εͳ͔ͬͨ༻ޠɺΤίγεςϜͳͲ·ͩ๲େʹʜ ‣ Զͨͪͷઓ͍͸͜Ε͔Βͩʂ 170

  171. 171 ࢀߟࢿྉ

  172. ࢀߟࢿྉ ‣ ,VCFSOFUFTެࣜαΠτ - https://kubernetes.io/ ‣ ,VCFSOFUFTΞΠίϯ - https://github.com/kubernetes/community/tree/master/icons ‣

    %PDLFS,VCFSOFUFT࣮ફίϯςφ։ൃೖ໳ - https://www.amazon.co.jp/dp/4297100339/ref=cm_sw_em_r_mt_dp_U_vNpSCbV87QB5G ‣ ೖ໳,VCFSOFUFT - https://www.amazon.co.jp/dp/4873118409/ref=cm_sw_em_r_mt_dp_U_AMpSCb8MKY1MN ‣ "NB[PO&,4ͷ࢖༻։࢝ - https://docs.aws.amazon.com/ja_jp/eks/latest/userguide/getting-started.html ‣ "NB[PO&,48PSLTIPQ - https://eksworkshop.com/jenkinsworld/ ‣ ,VCFSOFUFTPO"84&,4ϕετϓϥΫςΟε - https://speakerdeck.com/mumoshu/eksbesutopurakuteisu2019-dot-2-number-jawsdays ‣ ,VCFSOFUFT͸ۜͷ஄ؙͰ͸ͳ͍"84ʹ͓͚ΔίϯςφΦʔέετϨʔγϣϯͷݱࡏ஍ͱՄೳੑ - https://logmi.jp/tech/articles/305690 172
  173. ࢀߟࢿྉ ‣ ෼͘Β͍ͰΘ͔Δɺ,VCFSOFUFTͱ&,4ͷԿ͕ศརͳͷ͔ - https://qiita.com/masachaco/items/3e50a1ac65cdd661a734 ‣ ,VCFSOFUFTߏ੒ίϯϙʔωϯτҰཡ - https://qiita.com/tkusumi/items/c2a92cd52bfdb9edd613 ‣

    5PLVT#MPH,VCFSOFUFTಓ৔ - https://cstoku.dev/tags/kubernetes-dojo/ 173