Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How I've Operated Kubernetes on Prod for Half a Year (FJCT Meetup on Kubernetes #14)

lanocci
March 27, 2019

How I've Operated Kubernetes on Prod for Half a Year (FJCT Meetup on Kubernetes #14)

lanocci

March 27, 2019
Tweet

More Decks by lanocci

Other Decks in Technology

Transcript

  1. ,VCFSOFUFTΛ
    ൒೥ؒຊ൪ӡ༻͖ͯͯ͠
    ϋϚͬͨ͜ͱ
    !MBOPDDJ
    1

    View Slide

  2. ,PJDIJ6SBOP!MBOPDDJ
    w $ZCFS"HFOUΞυςΫελδΦॴଐ
    w $"%Z7& %41
    ʢࣗশʣ43&
    w $,"

    Koichi Urano
    April 1, 2018 CKA-1800-0366-0100
    1 / 1
    2

    View Slide

  3. "HFOEB
    όοΫάϥ΢ϯυ঺հ
    ,VCFSOFUFTͰϋϚͬͨ͜ͱʢ͍͍ҙຯͰʣ
    ,VCFSOFUFTͰϋϚͬͨ͜ͱʢѱ͍ҙຯͰʣͱͦͷରॲ
    ·ͱΊ
    3

    View Slide

  4. (PBM
    ΞϓϦέʔγϣϯج൫ͱͯ͠൒೥ؒӡ༻͖ͯͨ͠චऀͷܦݧ͔Βɺ๛෋ͳ
    ػೳΛඋ͑Δ,VCFSOFUFTͷ࢖͍ํΛ۩ମతʹΠϝʔδͯ͠΋Β͍ɺগ͠Ͱ
    ΋໾ཱ͍ͯͯͨͩ͘͜ͱ͕Ͱ͖ͨΒخ͍͠Ͱ͢
    4

    View Slide

  5. ஫ҙࣄ߲
    ͳΔ΂͘ಛఆͷΫϥ΢υϓϥοτϑΥʔϜʹภΒͳ͍Α͏ɺҰൠԽͨ͠ܗ
    Ͱ͝આ໌͢ΔΑ͏ʹ͍ͯ͠·͕͢ɺ($1ʹߏஙͨ͠γεςϜ͕࿩ͷݩʹͳͬ
    ͓ͯΓ·͢ͷͰɺҰ෦($1ಛ༗ͷ࿩୊͕ग़ͯ͘Δ͔΋͠Ε·ͤΜ͕ɺԿଔ
    ͝༰͍ࣻͩ͘͞N @@
    N
    5

    View Slide

  6. IUUQTBEUFDIDZCFSBHFOUJPBCPVUVT
    όοΫάϥ΢ϯυͷ͝঺հ
    6

    View Slide

  7. w bαΠόʔΤʔδΣϯτάϧʔϓͷΞυςΫϊϩδʔ෼໺ʹ͓͚Δ
    αʔϏε։ൃɾఏڙΛߦ͏ԣஅ૊৫`
    w ऑͷϓϩμΫτ͕͋ΓɺશମͰ໊Ҏ্ͷΤϯδχΞ͕ॴଐ
    w Πϯϑϥઐ໳ͷԣ࣠૊৫΋͋ΓɺϓϥΠϕʔτΫϥ΢υ؀ڥ͕੔උ͞Ε
    ͍ͯΔ
    7

    View Slide

  8. ΞυςΫʁ
    IUUQTBEUFDIDZCFSBHFOUJPQSBSDIJWFT 8

    View Slide

  9. w Πϯλʔωοτ޿ࠂϓϩμΫτʢ%41ʣ
    w %41ͱ͸
    w ޿ࠂओͱϝσΟΞΛͭͳ͙޿ࠂϓϥοτϑΥʔϜͷ͏ͪɺ޿ࠂओͷͨΊͷαʔ
    ϏεΛఏڙ͢Δ΋ͷ
    w 441ͱݺ͹ΕΔϝσΟΞଆͷͨΊͷϓϥοτϑΥʔϜ͔Β໰͍߹ΘͤΛड͚ɺ
    ද͍ࣔͨ͠޿ࠂΛฦ͢
    w େن໛ͳ%41ͩͱඵؒສϦΫΤετΛͦΕͧΕNTҎ಺ʹฦ͢
    9

    View Slide

  10. ͲΜͳγεςϜʁ
    Mark
    Bid
    Ad
    SSP
    དྷ๚௨஌
    ޿ࠂओ)1
    ϒϩάهࣄ౳
    ܭଌ
    ޿ࠂϦΫΤετ
    σʔλ
    ूܭɾՃ޻
    10

    View Slide

  11. ࢀߟɿલఏ஌ࣝ
    w .BTUFS
    w ,VCFSFOUFT$MVTUFSͷ؅ཧશൠΛߦ͏
    w /PEF
    w 1PE͕഑ஔ͞ΕΔϚγϯʢ7.෺ཧϚγϯʣ
    w LVCFDUM
    w ,VCFSOFUFTΛૢ࡞͢ΔͨΊͷίϚϯυϥΠϯπʔϧ
    https://kubernetes.io/docs/tutorials/kubernetes-basics/create-cluster/cluster-intro/
    11

    View Slide

  12. ࢀߟɿલఏ஌ࣝ
    w 1PE
    w ΞϓϦέʔγϣϯͷ࣮ߦʹඞཁͳίϯςφ΍ϦιʔεΛ·ͱΊͨ΋ͷ
    w %FQMPZNFOU
    w 1PEͷ࡞੒ɺߋ৽Λߦ͏୯Ґ
    w ΞϓϦέʔγϣϯछผ͝ͱʹ࡞੒͓͚ͯ͠͹ɺΞϓϦέʔγϣϯͷߋ৽΍૿ઃ
    Λ؆୯ʹͰ͖Δ
    w 4FSWJDF
    w 1PE΁ͷΞΫηεΛந৅Խ͢Δ΋ͷɻϩʔυόϥϯαʹ૬౰͢ΔϦιʔε
    12

    View Slide

  13. https://kubernetes.io/docs/tutorials/kubernetes-basics/expose-intro/
    13
    ࢀߟɿલఏ஌ࣝ

    View Slide

  14. ,VCFSOFUFTͰϋϚͬͨ͜ͱ
    ʢ͍͍ҙຯͰʣ
    14

    View Slide

  15. ,VCFSOFUFTͰΑ͔ͬͨ͜ͱ
    w ϦϦʔεָ͕ʢ͔͠΋҆શʣ
    w 7.ʹൺ΂ͯγεςϜӡ༻ָ͕
    15

    View Slide

  16. w ϦϦʔεָ͕ʢ͔͠΋҆શʣ
    w 7.ʹൺ΂ͯγεςϜӡ༻ָ͕
    16
    ,VCFSOFUFTͰΑ͔ͬͨ͜ͱ

    View Slide

  17. ϦϦʔεָ͕
    w ҙࣝ͠ͳͯ͘΋ϩʔϦϯάΞοϓσʔτͯ͘͠ΕΔ
    w ແఀࢭͰΞϓϦέʔγϣϯΛߋ৽Ͱ͖Δ
    w (JU0QTͱͷ਌࿨ੑ͕ߴ͍
    w (JUͷঢ়ଶͱΞϓϦέʔγϣϯঢ়ଶΛಉظ
    w ΧφϦΞϦϦʔεͱͷ਌࿨ੑ͕ߴ͍
    w ৽͍͠όʔδϣϯΛগ਺͚ͩग़͠ɺෆ۩߹͕ग़ͳ͍͜ͱΛ֬ೝ͔ͯ͠ΒશମΛߋ৽
    17

    View Slide

  18. ϦϦʔεָ͕
    w (JU0QTʁ
    18

    View Slide

  19. ϦϦʔεָ͕
    w (JU0QT
    w XFBWFXPSLT͕ࣾఏএ
    w IUUQTXXXXFBWFXPSLTUFDIOPMPHJFTHJUPQT
    w (JUΛA4JOHMF4PVSDFPG5SVUIA །Ұͷਅ࣮ͷ৘ใݯ
    ͱ͢Δ
    w ΞϓϦέʔγϣϯίʔυ΋Πϯϑϥͷઃఆ΋શͯ(JUͰ؅ཧ͢Δ
    w (JUͷঢ়ଶͱಈ࡞͍ͯ͠ΔΞϓϦέʔγϣϯͷঢ়ଶΛಉظ͢Δ
    w (JUͷΑ͏ʹ࢖͍׳ΕͨπʔϧΛ$*$%πʔϧͷத৺ʹਾ͑Δ͜ͱͰɺΤϯδχΞɹɹ
    ΈΜͳ͕։ൃʹूதͰ͖Δ͠ɺมߋཤྺΛશͯ࢒͢͜ͱ͕Ͱ͖Δ
    19

    View Slide

  20. ϦϦʔεָ͕
    20
    NFSHF

    SFMFBTFCSBODI
    ΞϓϦέʔγϣϯίʔυ
    ϦϙδτϦ
    ,TNBOJGFTU
    ϦϙδτϦ
    $MPVE#VJME $POUBJOFS3FHJTUSZ
    ,T&OHJOF
    NFSHFIPPL
    NFSHFIPPL
    NFSHF

    SFMFBTFCSBODI
    13UP
    SFMFBTFCSBODI
    QVTI
    QVMM
    EFQMPZ
    ˞؇Ίͷ(JU0QT

    View Slide

  21. ϦϦʔεָ͕
    21
    NFSHF

    SFMFBTFCSBODI
    NFSHF

    SFMFBTFCSBODI
    QVTI
    QVMM
    EFQMPZ
    ਓ͕հࡏ͢Δͷ͸
    (JU)VCͷૢ࡞ NFSHF
    ͚ͩ
    ΞϓϦέʔγϣϯίʔυ
    ϦϙδτϦ
    ,TNBOJGFTU
    ϦϙδτϦ
    ϥΫ
    ࣦഊ͠ͳ͍

    View Slide

  22. w ϦϦʔεָ͕ʢ͔͠΋҆શʣ
    w 7.ʹൺ΂ͯγεςϜӡ༻ָ͕
    22
    ,VCFSOFUFTͰΑ͔ͬͨ͜ͱ

    View Slide

  23. Ϧιʔεͷ࠷దԽָ͕
    w ΦʔτεέʔϦϯά
    w )PSJ[POUBM1PE"VUPTDBMFS,VCFSOFUFTͷػೳ
    w ໨ඪͱ͢ΔࢦඪΛઃఆ͓͚ͯ͠͹ɺͦΕʹ߹ΘͤͯࣗಈͰεέʔϧΞ΢τɺ
    εέʔϧΠϯͯ͘͠ΕΔ
    w $16ϝϞϦͷλʔήοτΛ࢖͏͜ͱ͕Ͱ͖Δͷ͸΋ͪΖΜɺϞχλϦϯά
    πʔϧ͔ΒࢦඪΛͱ͖ͬͯͯΧελϜࢦඪͰઃఆ͢Δ͜ͱ΋Ͱ͖Δ
    w FH
    ϦΫΤετ਺ͱ͔
    23
    UBSHFU$16
    $16ฏۉ

    View Slide

  24. Τϥʔରॲ͕ݮͬͨ
    w ϔϧενΣοΫ
    w MJWFOFTT1SPCF,VCFSOFUFTͷػೳ %FQMPZNFOU

    w 1PE͕݈߁Ͱ͋ΔͱΈͳ͢৚݅Λఆٛͯ͠ɺͦͷ৚͔݅Β֎ΕͨΒ
    ࣗ཯తʹ࠶࡞੒ͯ͘͠ΕΔ
    w FH
    +BWBϓϩηε͕ࢮΜͩΒউखʹ1PEΛ࡞Γ௚ͯ͘͠ΕΔ
    24

    View Slide

  25. ੑೳ΋͍ͨͬͯྑ޷
    w ,VCFSOFUFTʹࡌ͔ͤͨΒͱݴͬͯ໨ཱͬͯੑೳ͕ѱ͍ͱ͍͏͜ͱ΋ͳ͍
    w ޿ࠂϦΫΤετͷ͏ͪɺҎ্ΛNTҎ಺ʹฦ͍ͤͯΔ
    w ˞΄ͱΜͲ Ҏ্
    ͸NTҎ಺
    25

    View Slide

  26. ,VCFSOFUFTͰϋϚͬͨ͜ͱ
    ѱ͍ҙຯͰʣ
    26

    View Slide

  27. 3PMMJOH6QEBUFͯ͠Δ͸͕ͣɺɺɺ
    w ϦϦʔεதʹϨεϙϯεฦͤͳ͘ͳͬͯΔ͕࣌ؒ͋Δ
    w Ұؾʹ1PE࡞Γ௚͗͢͠ʁ
    27

    View Slide

  28. 3PMMJOH6QEBUFͯ͠Δ͸͕ͣɺɺɺ
    w 3PMMJOH6QEBUF4USBUFHZ
    w NBY4VSHF,VCFSOFUFTͷػೳ %FQMPZNFOU

    w 3PMMJOH6QEBUFதͷ1PE਺্ݶΛࢦఆ
    w NBY6OBWBJMBCMF,VCFSOFUFTͷػೳ %FQMPZNFOU

    w 3PMMJOH6QEBUFதͷՔಇ1PE਺ͷԼݶΛࢦఆ
    28

    View Slide

  29. 29
    w NBY4VSHFNBY6OBWBJMBCMF
    w NBY4VSHFNBY6OBWBJMBCMFʹͯ͠΋௚Βͣɺɺɺ
    3PMMJOH6QEBUFͯ͠Δ͸͕ͣɺɺɺ
    old
    old
    old old old
    new
    Քಇத
    ४උத
    SFQMJDBT NBY4VSHFNBY6OBWBJMBCMFͷ৔߹
    શମ·Ͱ૿͑ͯ0,Քಇத·Ͱݮͬͯ0,

    old new new
    new
    new new
    new
    new
    new new
    old old
    ʜ

    View Slide

  30. 3PMMJOH6QEBUFͯ͠Δ͸͕ͣɺɺɺ
    w ϦϦʔεதʹϨεϙϯεฦͤͳ͘ͳͬͯΔ͕࣌ؒ͋Δ
    w ࣮͸1PE͕ϦΫΤετड͚ΒΕΔঢ়ଶʹͳΔલʹτϥϑΟοΫɹɹɹ
    ৼΒΕͪΌͬͯΔʁ
    30

    View Slide

  31. 3PMMJOH6QEBUFͯ͠Δ͸͕ͣɺɺɺ
    w Πϝʔδ
    31
    1PE࡞੒։࢝
    1PE಺શͯͷ
    ίϯςφ͕ىಈ
    ࡞੒ͨ͠1PEʹ
    τϥϑΟοΫ͕
    ৼΒΕ࢝ΊΔ +7.ىಈ׬ྃ
    ʜ
    ͜ͷؒ
    ͣͬͱΤϥʔ

    View Slide

  32. 3PMMJOH6QEBUFͯ͠Δ͸͕ͣɺɺɺ
    w 3FBEJOFTT1SPCF,VCFSOFUFTͷػೳ %FQMPZNFOU

    w 3FBEJOFTT1SPCFΛࢦఆ͠ͳ͍ঢ়ଶͰ͸ɺ,VCFSOFUFT͸1PE͕ىಈͨ͠ͱಉ࣌ʹ
    -# 4FSWJDF
    ʹ૊ΈೖΕΔ
    w +BWBΞϓϦ౳Ͱ͸ɺ+7.ͷॳظԽ΍ΩϟογϡͷॳճಡΈࠐΈ͕૸Γɺ1PEͷىಈ
    ͱΞϓϦέʔγϣϯ͕3FBEZͱͳΔλΠϛϯάʹ͸݁ߏϥά͕͋Δ
    w ͦ͜Ͱɺ3FBEJOFTT1SPCFͰΞϓϦέʔγϣϯͷ3FBEZͱͳΔ৚݅ͱͯ͠+BWB
    ΞϓϦέʔγϣϯ͕ىಈ͢Δ1PSU΁ͷϋʔτϏʔτΛࢦఆ͓͚ͯ͠͹ɺΞϓϦͷ
    ىಈલʹΞΫηε͕དྷΔ͜ͱΛ๷͛Δʁ
    32

    View Slide

  33. 3PMMJOH6QEBUFͯ͠Δ͸͕ͣɺɺɺ
    w 3FBEJOFTT1SPCF
    33
    1PE࡞੒։࢝
    1PE಺શͯͷ
    ίϯςφ͕ىಈ
    ࡞੒ͨ͠1PEʹ
    τϥϑΟοΫ͕
    ৼΒΕ࢝ΊΔ
    +7.ىಈ׬ྃ
    ʜ
    3FBEJOFTT1SPCF0,
    3FBEJOFTT1SPCF
    /(
    ղܾ

    View Slide

  34. ݹ͘ͳΓΏ͘,Tͷόʔδϣϯ
    w ύϒϦοΫΫϥ΢υ࢖ͬͯΔͱɺ͍ͭͷؒʹ͔࢖ͬͯΔ,Tͷόʔδϣϯ
    ͕αϙʔτର৅֎ʹ
    w /PEFͷόʔδϣϯΛ্͛Δʹ͸/PEFΛ࠶ىಈ͠ͳ͍ͱ͍͚ͳ͍
    w ࠶ىಈͰҰؾʹ1PE͕ࢮΜͩΒࠔΔ
    34

    View Slide

  35. ݹ͘ͳΓΏ͘,Tͷόʔδϣϯ
    w Πϝʔδ
    w /PEFΛ࡟আ͔ͯ͠Β%FQMPZNFOU͕1PEΛ࡞Γ௚͢
    w
    35
    https://kubernetes.io/docs/tasks/run-application/configure-pdb/#specifying-a-poddisruptionbudget
    %FQMPZNFOU"
    %FQMPZNFOU#
    ෛՙूத

    View Slide

  36. ݹ͘ͳΓΏ͘,Tͷόʔδϣϯ
    w 1PE%JTSVQUJPO#VEHFUNBY6OBWBJMBCMF
    w SFQMJDB਺ʹରͯ͠Քಇதͷ1PEΛ͔ͭ͠ݮΒͯ͠͸͍͚ͳ͍
    w
    36
    https://kubernetes.io/docs/tasks/run-application/configure-pdb/#specifying-a-poddisruptionbudget
    %FQMPZNFOU"
    %FQMPZNFOU#
    ղܾ

    View Slide

  37. ݹ͘ͳΓΏ͘,Tͷόʔδϣϯ
    w 1PE%JTSVQUJPO#VEHFU,VCFSOFUFTͷػೳ
    w %FQMPZNFOU͝ͱʹɺ͍ͭ͘·ͰͳΒ1PEΛམͱͯ͠΋͍͍͔ΛఆٛͰ͖
    Δ
    w ͜ͷ৚݅Λຬͨͤͳ͍Α͏ͳૢ࡞Λߦ͓͏ͱ͢Δͱɺผͷ/PEFʹ৽͍͠
    SFBEZঢ়ଶͷ1PE͕Ͱ͖Δ·Ͱૢ࡞ΛਐΊͳ͍Α͏੍ݶͯ͘͠ΕΔ
    w όʔδϣϯΞοϓճͯ͠Δ͚Ͳ3FBEJOFTT1SPCFͱͷ૊Έ߹ΘͤͰ໰
    ୊ͳ͍
    37

    View Slide

  38. 1PE͕ภͬͯΔɻɻɻ
    w ͋Δछྨͷ1PE͕ಛఆͷ/PEFʹภͬͯ͠·ͬͨ
    w ΋͠ͳΜΒ͔ͷཧ༝Ͱ/PEF͕མͪͨΒαʔϏε͕མͪΔ
    38

    View Slide

  39. w Πϝʔδ
    w
    1PE͕ภͬͯΔɻɻɻ
    39
    %FQMPZNFOU"
    %FQMPZNFOU#
    %FQMPZNFOU$

    View Slide

  40. 1PE͕ภͬͯΔɻɻɻ
    w *OUFSQPEBOUJB⒏OJUZ,VCFSOFUFTͷػೳ %FQMPZNFOU

    w ಛఆͷ1PE͕ಈ͍͍ͯΔ/PEFʹରͯ͠৽͍͠1PEΛεέδϡʔϦϯά
    ͠ͳ͍ઃఆ͕Ͱ͖Δɹ
    w ͜ΕΛ࢖ͬͯɺಉҰ%FQMPZNFOUͷ1PE͕͢Ͱʹଘࡏ͍ͯ͠Δ/PEF
    ʹεέδϡʔϦϯά͠ͳ͍Α͏ʹઃఆͰ͖Δ
    40

    View Slide

  41. w *OUFSQPEBOUJB⒏OJUZͷઃఆ͋ΓʢಉҰ1PE͕ଘࡏ͢Δϊʔυʹ͸εέδϡʔϦϯά͠ͳ͍ʣ
    w
    1PE͕ภͬͯΔɻɻɻ
    41
    %FQMPZNFOU"
    %FQMPZNFOU#
    %FQMPZNFOU$

    View Slide

  42. γεςϜܥ1PEམͪͯΔ
    w LVCFQSPYZͱ͍͏ɺϦόʔεϓϩΩγͷΑ͏ͳ΋ͷ͕མ͍ͪͯͯɺ1PE΁ͷΞΫη
    εʹࣦഊ͢Δ
    w γεςϜܥͷ1PE΋શͯ%BFNPO4FU΋͘͠͸%FQMPZNFOUͳͷͰʢLVCFQSPYZ͸
    %BFNPO4FUʣɺ໰୊ͷ͋Δ1PEΛ࡟আ͢Ε͹ࣗಈతʹ࠶࡞੒͞Εͯ෮چ͸Մೳ
    w γεςϜܥ1PE΋མͪͳ͍อূ͸ͳ͍͠ɺো֐࣌ͷ੾Γ෼͚͕େมͳͷͰɺɹɹɹ
    ͪΌΜͱ؂ࢹ͸͓͍ͯͨ͠΄͏͕Αͦ͞͏
    w (,&ͳΒ૊ΈࠐΈͰTUBDLESJWFS͕ར༻Մೳɺ΋ͪΖΜ%BUBEPH౳Ͱ΋؂ࢹՄೳ
    w TUBDLESJWFSͰ֤ϊʔυͭҎ্LVCFQSPYZ্͕͕͍ͬͯͳ͚Ε͹Ξϥʔτ
    42

    View Slide

  43. ೔ʑؾΛ͚͍ͭͯΔ͜ͱ
    w νʔϜͰษڧձΛߦͬͨΓυΩϡϝϯτΛڞ༗ͨ͠Γͯ͠஌ࣝͷఈ্͛
    w υΩϡϝϯτ؅ཧʹ͸4DSBQCPYΛ࢖͍ͬͯ·͢ IUUQTTDSBQCPYJP

    w ฤू͠΍͘͢ɺݕࡧ͠΍͍͢
    w λά΍ϦϯΫΛ࢖ͬͯυΩϡϝϯτಉ࢜Λؔ࿈͚ͮͯɺ஌ࣝΛܨ͍͚͛ͯΔ
    w ࣗಈԽ
    w LVCFDUMΛͳΔ΂͘ਓ͕ؒୟ͔ͳ͍Α͏ʹ͍ͯ͘͠
    w HJU0QT
    w +FOLJOTΛ࢖ͬͨΦϖϨʔγϣϯࣗಈԽ
    43

    View Slide

  44. ·ͱΊ
    w ϦϦʔε࡞ۀ͕؆୯Ͱ҆શˠߴ଎ϦϦʔε
    w ࣗ཯తʹਖ਼ৗͳঢ়ଶΛҡ࣋ͯ͘͠ΕΔˠলྗӡ༻
    w ৭ʑ໰୊͸ग़ͯ͘Δ͚Ͳɺ,VCFSOFUFTͷ࢓૊ΈͷதͰղܾͰ͖ͦ͏
    w ୯ʹ,VCFSOFUFTΛ࢖͏͚ͩͰͦͷ··ӡ༻ָ͕ʹͳΔΘ͚Ͱ͸ͳ͍
    w ,VCFSOFUFTͷػೳΛཧղ্ͨ͠Ͱࡌͤସ͑Ε͹ӡ༻Λָʹͯ͘͠ΕΔ
    44

    View Slide

  45. ,VCFSOFUFTɺ࠷ߴ
    45

    View Slide

  46. 46
    ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠

    View Slide