Upgrade to Pro — share decks privately, control downloads, hide ads and more …

デプロイすると色々なPodが死ぬk8sで何が起きていたのか

po3rin
July 27, 2020

 デプロイすると色々なPodが死ぬk8sで何が起きていたのか

Taint/Toleration/Node Affinity/Taint based Eviction/Soft Eviction/Hard Eviction/QoS

po3rin

July 27, 2020
Tweet

More Decks by po3rin

Other Decks in Programming

Transcript

  1. σϓϩΠ͢Δͱ৭ʑͳ1PE͕ࢮ͵LTͰ
    Կ͕ى͖͍ͯͨͷ͔
    ,VCFSOFUFT/PWJDF5PLZP
    +VMZ
    CZQPSJO
    8IBUIBQQFOTXJUILTXIFSFWBSJPVTQPETEJFXIFOEFQMPZFE

    View full-size slide

  2. pon ( @po3rin )

    View full-size slide

  3. ɾ&WJDUBUJPOͱ͸
    ɾ/PEF"⒏OJUZ
    ɾ3PMMJOH6QEBUF
    ɾ4DIFEVMFSͷ͜ͱ গ͠

    ֶ΂Δ͜ͱ

    View full-size slide

  4. Introduction
    \

    View full-size slide

  5. ࣾ಺Ͱ&,4΁ͷҠߦϓϩδΣΫτ͕࢝·ͬͨͷͰษڧͷͨΊʹԿ͔Λ
    LTͰϚωʔδυͰӡ༻ͯ͠Έ͍ͨ✊ ʂʂ

    View full-size slide

  6. &,4Ͱӡ༻ͯ͠Δϒϩά✌
    QPSJODPN

    View full-size slide

  7. ϒϩάͷΞʔΩςΫνϟ✌

    View full-size slide

  8. "1*ΛσϓϩΠͨ͠ͱ͜Ζ&MBTUJDTFBSDIͳͲͷ༷ʑͳ1PE͕&WJDUFEͨ͠ޙʹ
    ৽͘͠1PEΛׂΓ౰ͯΔॲཧͰ1FOEJOHʹͳΓӬԕʹϊʔυʹׂΓ౰ͯΒΕͳ͍ɻ
    *TTVF

    View full-size slide

  9. ਤʹ͢Δͱ͜Μͳײ͡ɻ
    *TTVF

    "1* &4
    &4
    ᶃ"QQMZ
    ᶄ&WJDUFE
    ᶅ1FOEJOH
    º
    ࠓճ͸ɺ͜ͷ໰୊Λ௥͍ͬͯ͘͜ͱͰ,VCFSOFUTΛΑΓਂ͘஌͍ͬͯ͜͏ʂʂ

    View full-size slide

  10. Investigation
    \

    View full-size slide

  11. ·ͣ͸ᶅͷ1FOEJOH͞Ε͍ͯͨཧ༝͔Βݟ͍ͯ͘
    8IZ1FOEJOH

    "1* &4
    &4
    ᶃ"QQMZ
    ᶄ&WJDUFE
    ᶅ1FOEJOH
    º

    View full-size slide

  12. 1FOEJOHʹͳ͍ͬͯΔ&MBTUJDTFBSDIͷ1PEΛΈΔͱ"⒏OJUZ$POqJDU͕ൃੜ͍ͯ͠Δ
    8IZ1FOEJOH

    View full-size slide

  13. /PEFͷ"⒏OJUZ͸5BJOU5PMFSBUJPOͰ࣮ݱ͞ΕΔɻ1PEΛεέδϡʔϧͤ͞ͳ͍ͨΊͷػೳͱͯ͠ར
    ༻Ͱ͖Δɻ/PEFʹ5BJOU ԚΕ
    Λઃఆ͠ɺͦΕΛڐ༰͢Δ1PEʹ͸5PMFSBUJPO ڐ༰
    Λઃఆ͢Δ
    /PEF"⒏OJUZ

    1PE
    UZQF$/P4DIFEVMF
    UZQF"/P4DIFEVMF
    UZQF#/P4DIFEVMF
    /PEF
    ˓
    º
    º

    View full-size slide

  14. ਖ਼ৗ࣌Λ֬ೝ͢Δͱࣗ෼͸/PEFʹ͸5BJOUͳΜͯઃఆ͍ͯ͠ͳ͍ɻɻͳͥ/PEF"⒏OJUZ$POqJDU͕ʁʁʁ
    /PEF"⒏OJUZ

    View full-size slide

  15. /PEF͕͋Δ৚݅Λຬͨ͢ͱɺ/PEF$POUSPMMFS͕৚݅ʹԠͯ͡ӈͷ5BJOUΛࣗಈͰ෇༩͢Δɻࠓճ͸
    ࣗಈͰʮOPEFLVCFSOFUFTJPEJTLQSFTTVSFʯ͕෇༩͞Ε͍ͯͨɻ͜ΕͰ&MBTUJDTFBSDIͷ1PEׂ͕
    Γ౰ͯΒΕͣ1FOEJOH͞Ε͍ͯͨʂ
    IUUQTLVCFSOFUFTJPEPDTDPODFQUTTDIFEVMJOHFWJDUJPOUBJOUBOEUPMFSBUJPOUBJOUCBTFEFWJDUJPOT
    5BJOUCBTFE&WJDUJPO

    View full-size slide

  16. ͪͳΈʹ&MBTUJDTFBSDI͸&$,Ͱઃఆ͓ͯ͠Γɺ5PMFSBUJPO͸ԼهͷΑ͏ʹઃఆ͞Ε͍ͯͨɻ
    5BJOUCBTFE&WJDUJPO

    View full-size slide

  17. 1FOEJOH͞Ε͍ͯͨཧ༝͸Θ͔ͬͨɻͰ͸ͳͥ1PE͕&WJDUFEʹͳ͔ͬͨʁ
    1FOEJOHͷௐࠪͰ͢ͰʹσΟεΫ࢖༻཰ͷ͍ͤͰ͋Δ͜ͱ͸໌നɻ
    8IZ&WJDUFE
    "1* &4
    &4
    ᶃ"QQMZ
    ᶄ&WJDUFE
    ᶅ1FOEJOH
    º

    View full-size slide

  18. &WJDUJPOʹ͸࣮͸ύλʔϯ͋Γɺ4PGU&WJDUJPOͱ)BSE
    &WJDUJPO͕͋Δɻ
    4PGU&WJDUJPO
    &WJDUJPOॲཧ·Ͱͷ༛༧ظ͕ؒ༩͑ΒΕΔɻ
    )BSE&WJDUJPO
    ༛༧ظؒͳ͠ʂʂ
    IUUQTLVCFSOFUFTJPEPDTUBTLTBENJOJTUFSDMVTUFSPVUPG
    SFTPVSDFFWJDUJPOUISFTIPMET
    4PGU&WJDUJPO)BSE&WJDUJPO

    View full-size slide

  19. 4PGU&WJDUJPO5ISFTIPMETΛ௒͑Δͱɺ1SFTTVSF͕͔͔͍ͬͯΔ͜ͱΛࣔ͢ϑϥά͕ΞΫςΟϒʹ
    ͳΔɻ͜Ε͸༛༧ظؒʹؔ܎ͳ͘ߦΘΕΔɻEJTDSJCFαϒίϚϯυͰ֬ೝͰ͖Δɻ
    &WJDUBUJPO1SPDFTT

    ͜ΕΒͷϑϥάʹΑΓɺϊʔυ಺ͷ৽ׂ͍͠Γ౰͕ͯϒϩοΫ͞ΕɺFWJDUJPOϓϩηε͕։࢝͞ΕΔɻ
    IUUQTLVCFSOFUFTJPEPDTUBTLTBENJOJTUFSDMVTUFSPVUPGSFTPVSDFOPEFDPOEJUJPOT

    View full-size slide

  20. LVCFMFU͕Ϧιʔεͷ࢖༻ྔ͕࠶ͼFWJDUJPOͷ͖͍͠஋ΛԼճΔ·Ͱɺ1PEΛ࡟আ͍ͯ͘͠ϓϩηεͰ͢ɻ
    ͪͳΈʹࠓճ͸σϓϩΠ࣌ʹ)BSE&WJDUJPOͷᮢ஋ͷʮOPEFGTBWBJMBCMFʯʹҾ͔͔͍ͬͬͯͨɻ
    IUUQTLVCFSOFUFTJPEPDTUBTLTBENJOJTUFSDMVTUFSPVUPGSFTPVSDFIBSEFWJDUJPOUISFTIPMET
    &WJDUBUJPO1SPDFTT

    View full-size slide

  21. LVCFMFU͸ɺෆཁͳϙουΛ࡟আ΍ະ࢖༻ͷΠϝʔδΛ࡟আ͍͖ͯ͠·͕͢ɺͦΕͰ΋ෆे෼ͳ৔߹͸ɺ
    1PEʹઃఆ͞Εͨ2PT 2VBMJUZPG4FSWJD
    ͷॱংͰ1PEΛ࡟আ͍ͯ͘͠ɻ2PT͸1PEͷ3FRVFTUT-JNJUT
    Ͱ൑அ͞ΕΔɻ
    #FTU&⒎PSU
    #VSTUBCMF
    (VBSBOUFFE
    2P4

    View full-size slide

  22. ԼهίϚϯυͰ2PT͕֬ೝͰ͖Δɻ"1*ΑΓୀආͤ͞ΒΕ΍͍͢&MBTUJDTFBSDI স

    ࠓճ͸ͦͷதͰ΋࠷΋σΟεΫΛফඅ͍ͯ͠Δ&MBTUJDTFBSDI͕ର৅ʹͳ͍ͬͯͨɻ
    2P4

    View full-size slide

  23. ࠷ऴతʹݪҼΛԼهͷΑ͏ʹಛఆͰ͖ͨɻ
    *TTVF
    "1* &4
    &4
    ᶃ "QQMZ3PMMJOH6QEBUFͰߋͳΔσΟεΫͷཁٻ
    ᶄ&WJDUFEσΟεΫѹഭʹΑΔ)BSE&WJDUJPO
    ᶅ1FOEJOHσΟεΫѹഭʹΑΔ5BJOU
    º

    View full-size slide

  24. σΟεΫαΠζ͸े෼ʹͱ͓ͬͯ͘
    ҰܸͰୀආ͞Εͳ͍Α͏ʹ5FSNJOBUJPO(SBDF1FSJPE4FDPOET΋ઃఆ͠ͱ͘ͱྑ͍
    &WJDU͞Εͨ͘ͳ͍1PEͷ2P4Λ(VBSBOUFFE·ͰͪΌΜͱ࣋ͬͯ͘ɻ
    1SJPSJUZ$MBTT΋ઃఆ͓ͯ͘͠
    ͦ΋ͦ΋ϊʔυࣗମͷσΟεΫ༰ྔ͸ͪΌΜͱ؂ࢹ͢Δɻ
    ςετ؀ڥ͕༻ҙͰ͖ΔͳΒϦιʔεྔ΋ಉ͡ʹ͓ͯ͘͠ɻ
    4PMVUJPO

    View full-size slide