デプロイすると色々なPodが死ぬk8sで何が起きていたのか

4d78b749c315e21a956053dcbf0508a9?s=47 po3rin
July 27, 2020

 デプロイすると色々なPodが死ぬk8sで何が起きていたのか

Taint/Toleration/Node Affinity/Taint based Eviction/Soft Eviction/Hard Eviction/QoS

4d78b749c315e21a956053dcbf0508a9?s=128

po3rin

July 27, 2020
Tweet

Transcript

  1. σϓϩΠ͢Δͱ৭ʑͳ1PE͕ࢮ͵LTͰ Կ͕ى͖͍ͯͨͷ͔ ,VCFSOFUFT/PWJDF5PLZP +VMZ  CZQPSJO 8IBUIBQQFOTXJUILTXIFSFWBSJPVTQPETEJFXIFOEFQMPZFE

  2. pon ( @po3rin )

  3. ɾ&WJDUBUJPOͱ͸ ɾ/PEF"⒏OJUZ ɾ3PMMJOH6QEBUF ɾ4DIFEVMFSͷ͜ͱ গ͠ ֶ΂Δ͜ͱ

  4. Introduction \

  5. ࣾ಺Ͱ&,4΁ͷҠߦϓϩδΣΫτ͕࢝·ͬͨͷͰษڧͷͨΊʹԿ͔Λ LTͰϚωʔδυͰӡ༻ͯ͠Έ͍ͨ✊ ʂʂ

  6. &,4Ͱӡ༻ͯ͠Δϒϩά✌ QPSJODPN

  7. ϒϩάͷΞʔΩςΫνϟ✌

  8. "1*ΛσϓϩΠͨ͠ͱ͜Ζ&MBTUJDTFBSDIͳͲͷ༷ʑͳ1PE͕&WJDUFEͨ͠ޙʹ ৽͘͠1PEΛׂΓ౰ͯΔॲཧͰ1FOEJOHʹͳΓӬԕʹϊʔυʹׂΓ౰ͯΒΕͳ͍ɻ *TTVF 

  9. ਤʹ͢Δͱ͜Μͳײ͡ɻ *TTVF  "1* &4 &4 ᶃ"QQMZ ᶄ&WJDUFE ᶅ1FOEJOH º

    ࠓճ͸ɺ͜ͷ໰୊Λ௥͍ͬͯ͘͜ͱͰ,VCFSOFUTΛΑΓਂ͘஌͍ͬͯ͜͏ʂʂ
  10. Investigation \

  11. ·ͣ͸ᶅͷ1FOEJOH͞Ε͍ͯͨཧ༝͔Βݟ͍ͯ͘ 8IZ1FOEJOH   "1* &4 &4 ᶃ"QQMZ ᶄ&WJDUFE ᶅ1FOEJOH

    º
  12. 1FOEJOHʹͳ͍ͬͯΔ&MBTUJDTFBSDIͷ1PEΛΈΔͱ"⒏OJUZ$POqJDU͕ൃੜ͍ͯ͠Δ 8IZ1FOEJOH  

  13. /PEFͷ"⒏OJUZ͸5BJOU5PMFSBUJPOͰ࣮ݱ͞ΕΔɻ1PEΛεέδϡʔϧͤ͞ͳ͍ͨΊͷػೳͱͯ͠ར ༻Ͱ͖Δɻ/PEFʹ5BJOU ԚΕ Λઃఆ͠ɺͦΕΛڐ༰͢Δ1PEʹ͸5PMFSBUJPO ڐ༰ Λઃఆ͢Δ /PEF"⒏OJUZ  1PE UZQF$/P4DIFEVMF

    UZQF"/P4DIFEVMF UZQF#/P4DIFEVMF /PEF ˓ º º
  14. ਖ਼ৗ࣌Λ֬ೝ͢Δͱࣗ෼͸/PEFʹ͸5BJOUͳΜͯઃఆ͍ͯ͠ͳ͍ɻɻͳͥ/PEF"⒏OJUZ$POqJDU͕ʁʁʁ /PEF"⒏OJUZ 

  15. /PEF͕͋Δ৚݅Λຬͨ͢ͱɺ/PEF$POUSPMMFS͕৚݅ʹԠͯ͡ӈͷ5BJOUΛࣗಈͰ෇༩͢Δɻࠓճ͸ ࣗಈͰʮOPEFLVCFSOFUFTJPEJTLQSFTTVSFʯ͕෇༩͞Ε͍ͯͨɻ͜ΕͰ&MBTUJDTFBSDIͷ1PEׂ͕ Γ౰ͯΒΕͣ1FOEJOH͞Ε͍ͯͨʂ IUUQTLVCFSOFUFTJPEPDTDPODFQUTTDIFEVMJOHFWJDUJPOUBJOUBOEUPMFSBUJPOUBJOUCBTFEFWJDUJPOT 5BJOUCBTFE&WJDUJPO

  16. ͪͳΈʹ&MBTUJDTFBSDI͸&$,Ͱઃఆ͓ͯ͠Γɺ5PMFSBUJPO͸ԼهͷΑ͏ʹઃఆ͞Ε͍ͯͨɻ 5BJOUCBTFE&WJDUJPO

  17. 1FOEJOH͞Ε͍ͯͨཧ༝͸Θ͔ͬͨɻͰ͸ͳͥ1PE͕&WJDUFEʹͳ͔ͬͨʁ 1FOEJOHͷௐࠪͰ͢ͰʹσΟεΫ࢖༻཰ͷ͍ͤͰ͋Δ͜ͱ͸໌നɻ 8IZ&WJDUFE "1* &4 &4 ᶃ"QQMZ ᶄ&WJDUFE ᶅ1FOEJOH º

  18. &WJDUJPOʹ͸࣮͸ύλʔϯ͋Γɺ4PGU&WJDUJPOͱ)BSE &WJDUJPO͕͋Δɻ 4PGU&WJDUJPO &WJDUJPOॲཧ·Ͱͷ༛༧ظ͕ؒ༩͑ΒΕΔɻ )BSE&WJDUJPO ༛༧ظؒͳ͠ʂʂ IUUQTLVCFSOFUFTJPEPDTUBTLTBENJOJTUFSDMVTUFSPVUPG SFTPVSDFFWJDUJPOUISFTIPMET 4PGU&WJDUJPO)BSE&WJDUJPO

  19. 4PGU&WJDUJPO5ISFTIPMETΛ௒͑Δͱɺ1SFTTVSF͕͔͔͍ͬͯΔ͜ͱΛࣔ͢ϑϥά͕ΞΫςΟϒʹ ͳΔɻ͜Ε͸༛༧ظؒʹؔ܎ͳ͘ߦΘΕΔɻEJTDSJCFαϒίϚϯυͰ֬ೝͰ͖Δɻ &WJDUBUJPO1SPDFTT  ͜ΕΒͷϑϥάʹΑΓɺϊʔυ಺ͷ৽ׂ͍͠Γ౰͕ͯϒϩοΫ͞ΕɺFWJDUJPOϓϩηε͕։࢝͞ΕΔɻ IUUQTLVCFSOFUFTJPEPDTUBTLTBENJOJTUFSDMVTUFSPVUPGSFTPVSDFOPEFDPOEJUJPOT

  20. LVCFMFU͕Ϧιʔεͷ࢖༻ྔ͕࠶ͼFWJDUJPOͷ͖͍͠஋ΛԼճΔ·Ͱɺ1PEΛ࡟আ͍ͯ͘͠ϓϩηεͰ͢ɻ ͪͳΈʹࠓճ͸σϓϩΠ࣌ʹ)BSE&WJDUJPOͷᮢ஋ͷʮOPEFGTBWBJMBCMFʯʹҾ͔͔͍ͬͬͯͨɻ IUUQTLVCFSOFUFTJPEPDTUBTLTBENJOJTUFSDMVTUFSPVUPGSFTPVSDFIBSEFWJDUJPOUISFTIPMET &WJDUBUJPO1SPDFTT 

  21. LVCFMFU͸ɺෆཁͳϙουΛ࡟আ΍ະ࢖༻ͷΠϝʔδΛ࡟আ͍͖ͯ͠·͕͢ɺͦΕͰ΋ෆे෼ͳ৔߹͸ɺ 1PEʹઃఆ͞Εͨ2PT 2VBMJUZPG4FSWJD ͷॱংͰ1PEΛ࡟আ͍ͯ͘͠ɻ2PT͸1PEͷ3FRVFTUT-JNJUT Ͱ൑அ͞ΕΔɻ #FTU&⒎PSU #VSTUBCMF (VBSBOUFFE 2P4 

  22. ԼهίϚϯυͰ2PT͕֬ೝͰ͖Δɻ"1*ΑΓୀආͤ͞ΒΕ΍͍͢&MBTUJDTFBSDI স  ࠓճ͸ͦͷதͰ΋࠷΋σΟεΫΛফඅ͍ͯ͠Δ&MBTUJDTFBSDI͕ର৅ʹͳ͍ͬͯͨɻ 2P4 

  23. ࠷ऴతʹݪҼΛԼهͷΑ͏ʹಛఆͰ͖ͨɻ *TTVF "1* &4 &4 ᶃ "QQMZ3PMMJOH6QEBUFͰߋͳΔσΟεΫͷཁٻ ᶄ&WJDUFEσΟεΫѹഭʹΑΔ)BSE&WJDUJPO ᶅ1FOEJOHσΟεΫѹഭʹΑΔ5BJOU º

  24. σΟεΫαΠζ͸े෼ʹͱ͓ͬͯ͘ ҰܸͰୀආ͞Εͳ͍Α͏ʹ5FSNJOBUJPO(SBDF1FSJPE4FDPOET΋ઃఆ͠ͱ͘ͱྑ͍ &WJDU͞Εͨ͘ͳ͍1PEͷ2P4Λ(VBSBOUFFE·ͰͪΌΜͱ࣋ͬͯ͘ɻ 1SJPSJUZ$MBTT΋ઃఆ͓ͯ͘͠ ͦ΋ͦ΋ϊʔυࣗମͷσΟεΫ༰ྔ͸ͪΌΜͱ؂ࢹ͢Δɻ ςετ؀ڥ͕༻ҙͰ͖ΔͳΒϦιʔεྔ΋ಉ͡ʹ͓ͯ͘͠ɻ 4PMVUJPO