How I've Operated Kubernetes on Prod for Half a Year (FJCT Meetup on Kubernetes #14)

Transcript

1. ,VCFSOFUFTΛ
   ൒೥ؒຊ൪ӡ༻͖ͯͯ͠
   ϋϚͬͨ͜ͱ
   !MBOPDDJ
   1
   

   View Slide

2. ,PJDIJ6SBOP!MBOPDDJ
   w $ZCFS"HFOUΞυςΫελδΦॴଐ
   w $"%Z7& %41
   ʢࣗশʣ43&
   w $,"
   
   Koichi Urano
   April 1, 2018 CKA-1800-0366-0100
   1 / 1
   2
   

   View Slide

3. "HFOEB
   όοΫάϥ΢ϯυ঺հ
   ,VCFSOFUFTͰϋϚͬͨ͜ͱʢ͍͍ҙຯͰʣ
   ,VCFSOFUFTͰϋϚͬͨ͜ͱʢѱ͍ҙຯͰʣͱͦͷରॲ
   ·ͱΊ
   3
   

   View Slide

4. (PBM
   ΞϓϦέʔγϣϯج൫ͱͯ͠൒೥ؒӡ༻͖ͯͨ͠චऀͷܦݧ͔Βɺ๛෋ͳ
   ػೳΛඋ͑Δ,VCFSOFUFTͷ࢖͍ํΛ۩ମతʹΠϝʔδͯ͠΋Β͍ɺগ͠Ͱ
   ΋໾ཱ͍ͯͯͨͩ͘͜ͱ͕Ͱ͖ͨΒخ͍͠Ͱ͢
   4
   

   View Slide

5. ஫ҙࣄ߲
   ͳΔ΂͘ಛఆͷΫϥ΢υϓϥοτϑΥʔϜʹภΒͳ͍Α͏ɺҰൠԽͨ͠ܗ
   Ͱ͝આ໌͢ΔΑ͏ʹ͍ͯ͠·͕͢ɺ($1ʹߏஙͨ͠γεςϜ͕࿩ͷݩʹͳͬ
   ͓ͯΓ·͢ͷͰɺҰ෦($1ಛ༗ͷ࿩୊͕ग़ͯ͘Δ͔΋͠Ε·ͤΜ͕ɺԿଔ
   ͝༰͍ࣻͩ͘͞N @@
   N
   5
   

   View Slide

6. IUUQTBEUFDIDZCFSBHFOUJPBCPVUVT
   όοΫάϥ΢ϯυͷ͝঺հ
   6
   

   View Slide

7. w bαΠόʔΤʔδΣϯτάϧʔϓͷΞυςΫϊϩδʔ෼໺ʹ͓͚Δ
   αʔϏε։ൃɾఏڙΛߦ͏ԣஅ૊৫`
   w ऑͷϓϩμΫτ͕͋ΓɺશମͰ໊Ҏ্ͷΤϯδχΞ͕ॴଐ
   w Πϯϑϥઐ໳ͷԣ࣠૊৫΋͋ΓɺϓϥΠϕʔτΫϥ΢υ؀ڥ͕੔උ͞Ε
   ͍ͯΔ
   7
   

   View Slide

8. ΞυςΫʁ
   IUUQTBEUFDIDZCFSBHFOUJPQSBSDIJWFT 8
   

   View Slide

9. w Πϯλʔωοτ޿ࠂϓϩμΫτʢ%41ʣ
   w %41ͱ͸
   w ޿ࠂओͱϝσΟΞΛͭͳ͙޿ࠂϓϥοτϑΥʔϜͷ͏ͪɺ޿ࠂओͷͨΊͷαʔ
   ϏεΛఏڙ͢Δ΋ͷ
   w 441ͱݺ͹ΕΔϝσΟΞଆͷͨΊͷϓϥοτϑΥʔϜ͔Β໰͍߹ΘͤΛड͚ɺ
   ද͍ࣔͨ͠޿ࠂΛฦ͢
   w େن໛ͳ%41ͩͱඵؒສϦΫΤετΛͦΕͧΕNTҎ಺ʹฦ͢
   9
   

   View Slide

10. ͲΜͳγεςϜʁ
    Mark
    Bid
    Ad
    SSP
    དྷ๚௨஌
    ޿ࠂओ)1
    ϒϩάهࣄ౳
    ܭଌ
    ޿ࠂϦΫΤετ
    σʔλ
    ूܭɾՃ޻
    10
    

    View Slide

11. ࢀߟɿલఏ஌ࣝ
    w .BTUFS
    w ,VCFSFOUFT$MVTUFSͷ؅ཧશൠΛߦ͏
    w /PEF
    w 1PE͕഑ஔ͞ΕΔϚγϯʢ7.෺ཧϚγϯʣ
    w LVCFDUM
    w ,VCFSOFUFTΛૢ࡞͢ΔͨΊͷίϚϯυϥΠϯπʔϧ
    https://kubernetes.io/docs/tutorials/kubernetes-basics/create-cluster/cluster-intro/
    11
    

    View Slide

12. ࢀߟɿલఏ஌ࣝ
    w 1PE
    w ΞϓϦέʔγϣϯͷ࣮ߦʹඞཁͳίϯςφ΍ϦιʔεΛ·ͱΊͨ΋ͷ
    w %FQMPZNFOU
    w 1PEͷ࡞੒ɺߋ৽Λߦ͏୯Ґ
    w ΞϓϦέʔγϣϯछผ͝ͱʹ࡞੒͓͚ͯ͠͹ɺΞϓϦέʔγϣϯͷߋ৽΍૿ઃ
    Λ؆୯ʹͰ͖Δ
    w 4FSWJDF
    w 1PE΁ͷΞΫηεΛந৅Խ͢Δ΋ͷɻϩʔυόϥϯαʹ૬౰͢ΔϦιʔε
    12
    

    View Slide

13. https://kubernetes.io/docs/tutorials/kubernetes-basics/expose-intro/
    13
    ࢀߟɿલఏ஌ࣝ
    

    View Slide

14. ,VCFSOFUFTͰϋϚͬͨ͜ͱ
    ʢ͍͍ҙຯͰʣ
    14
    

    View Slide

15. ,VCFSOFUFTͰΑ͔ͬͨ͜ͱ
    w ϦϦʔεָ͕ʢ͔͠΋҆શʣ
    w 7.ʹൺ΂ͯγεςϜӡ༻ָ͕
    15
    

    View Slide

16. w ϦϦʔεָ͕ʢ͔͠΋҆શʣ
    w 7.ʹൺ΂ͯγεςϜӡ༻ָ͕
    16
    ,VCFSOFUFTͰΑ͔ͬͨ͜ͱ
    

    View Slide

17. ϦϦʔεָ͕
    w ҙࣝ͠ͳͯ͘΋ϩʔϦϯάΞοϓσʔτͯ͘͠ΕΔ
    w ແఀࢭͰΞϓϦέʔγϣϯΛߋ৽Ͱ͖Δ
    w (JU0QTͱͷ਌࿨ੑ͕ߴ͍
    w (JUͷঢ়ଶͱΞϓϦέʔγϣϯঢ়ଶΛಉظ
    w ΧφϦΞϦϦʔεͱͷ਌࿨ੑ͕ߴ͍
    w ৽͍͠όʔδϣϯΛগ਺͚ͩग़͠ɺෆ۩߹͕ग़ͳ͍͜ͱΛ֬ೝ͔ͯ͠ΒશମΛߋ৽
    17
    

    View Slide

18. ϦϦʔεָ͕
    w (JU0QTʁ
    18
    

    View Slide

19. ϦϦʔεָ͕
    w (JU0QT
    w XFBWFXPSLT͕ࣾఏএ
    w IUUQTXXXXFBWFXPSLTUFDIOPMPHJFTHJUPQT
    w (JUΛA4JOHMF4PVSDFPG5SVUIA །Ұͷਅ࣮ͷ৘ใݯ
    ͱ͢Δ
    w ΞϓϦέʔγϣϯίʔυ΋Πϯϑϥͷઃఆ΋શͯ(JUͰ؅ཧ͢Δ
    w (JUͷঢ়ଶͱಈ࡞͍ͯ͠ΔΞϓϦέʔγϣϯͷঢ়ଶΛಉظ͢Δ
    w (JUͷΑ͏ʹ࢖͍׳ΕͨπʔϧΛ$*$%πʔϧͷத৺ʹਾ͑Δ͜ͱͰɺΤϯδχΞɹɹ
    ΈΜͳ͕։ൃʹूதͰ͖Δ͠ɺมߋཤྺΛશͯ࢒͢͜ͱ͕Ͱ͖Δ
    19
    

    View Slide

20. ϦϦʔεָ͕
    20
    NFSHF

    SFMFBTFCSBODI
    ΞϓϦέʔγϣϯίʔυ
    ϦϙδτϦ
    ,TNBOJGFTU
    ϦϙδτϦ
    $MPVE#VJME $POUBJOFS3FHJTUSZ
    ,T&OHJOF
    NFSHFIPPL
    NFSHFIPPL
    NFSHF

    SFMFBTFCSBODI
    13UP
    SFMFBTFCSBODI
    QVTI
    QVMM
    EFQMPZ
    ˞؇Ίͷ(JU0QT
    

    View Slide

21. ϦϦʔεָ͕
    21
    NFSHF

    SFMFBTFCSBODI
    NFSHF

    SFMFBTFCSBODI
    QVTI
    QVMM
    EFQMPZ
    ਓ͕հࡏ͢Δͷ͸
    (JU)VCͷૢ࡞ NFSHF
    ͚ͩ
    ΞϓϦέʔγϣϯίʔυ
    ϦϙδτϦ
    ,TNBOJGFTU
    ϦϙδτϦ
    ϥΫ
    ࣦഊ͠ͳ͍
    

    View Slide

22. w ϦϦʔεָ͕ʢ͔͠΋҆શʣ
    w 7.ʹൺ΂ͯγεςϜӡ༻ָ͕
    22
    ,VCFSOFUFTͰΑ͔ͬͨ͜ͱ
    

    View Slide

23. Ϧιʔεͷ࠷దԽָ͕
    w ΦʔτεέʔϦϯά
    w )PSJ[POUBM1PE"VUPTDBMFS,VCFSOFUFTͷػೳ
    w ໨ඪͱ͢ΔࢦඪΛઃఆ͓͚ͯ͠͹ɺͦΕʹ߹ΘͤͯࣗಈͰεέʔϧΞ΢τɺ
    εέʔϧΠϯͯ͘͠ΕΔ
    w $16ϝϞϦͷλʔήοτΛ࢖͏͜ͱ͕Ͱ͖Δͷ͸΋ͪΖΜɺϞχλϦϯά
    πʔϧ͔ΒࢦඪΛͱ͖ͬͯͯΧελϜࢦඪͰઃఆ͢Δ͜ͱ΋Ͱ͖Δ
    w FH
    ϦΫΤετ਺ͱ͔
    23
    UBSHFU$16
    $16ฏۉ
    

    View Slide

24. Τϥʔରॲ͕ݮͬͨ
    w ϔϧενΣοΫ
    w MJWFOFTT1SPCF,VCFSOFUFTͷػೳ %FQMPZNFOU
    
    w 1PE͕݈߁Ͱ͋ΔͱΈͳ͢৚݅Λఆٛͯ͠ɺͦͷ৚͔݅Β֎ΕͨΒ
    ࣗ཯తʹ࠶࡞੒ͯ͘͠ΕΔ
    w FH
    +BWBϓϩηε͕ࢮΜͩΒউखʹ1PEΛ࡞Γ௚ͯ͘͠ΕΔ
    24
    

    View Slide

25. ੑೳ΋͍ͨͬͯྑ޷
    w ,VCFSOFUFTʹࡌ͔ͤͨΒͱݴͬͯ໨ཱͬͯੑೳ͕ѱ͍ͱ͍͏͜ͱ΋ͳ͍
    w ޿ࠂϦΫΤετͷ͏ͪɺҎ্ΛNTҎ಺ʹฦ͍ͤͯΔ
    w ˞΄ͱΜͲ Ҏ্
    ͸NTҎ಺
    25
    

    View Slide

26. ,VCFSOFUFTͰϋϚͬͨ͜ͱ
    ѱ͍ҙຯͰʣ
    26
    

    View Slide

27. 3PMMJOH6QEBUFͯ͠Δ͸͕ͣɺɺɺ
    w ϦϦʔεதʹϨεϙϯεฦͤͳ͘ͳͬͯΔ͕࣌ؒ͋Δ
    w Ұؾʹ1PE࡞Γ௚͗͢͠ʁ
    27
    

    View Slide

28. 3PMMJOH6QEBUFͯ͠Δ͸͕ͣɺɺɺ
    w 3PMMJOH6QEBUF4USBUFHZ
    w NBY4VSHF,VCFSOFUFTͷػೳ %FQMPZNFOU
    
    w 3PMMJOH6QEBUFதͷ1PE਺্ݶΛࢦఆ
    w NBY6OBWBJMBCMF,VCFSOFUFTͷػೳ %FQMPZNFOU
    
    w 3PMMJOH6QEBUFதͷՔಇ1PE਺ͷԼݶΛࢦఆ
    28
    

    View Slide

29. 29
    w NBY4VSHFNBY6OBWBJMBCMF
    w NBY4VSHFNBY6OBWBJMBCMFʹͯ͠΋௚Βͣɺɺɺ
    3PMMJOH6QEBUFͯ͠Δ͸͕ͣɺɺɺ
    old
    old
    old old old
    new
    Քಇத
    ४උத
    SFQMJDBT NBY4VSHFNBY6OBWBJMBCMFͷ৔߹
    શମ·Ͱ૿͑ͯ0,Քಇத·Ͱݮͬͯ0,
    
    old new new
    new
    new new
    new
    new
    new new
    old old
    ʜ
    

    View Slide

30. 3PMMJOH6QEBUFͯ͠Δ͸͕ͣɺɺɺ
    w ϦϦʔεதʹϨεϙϯεฦͤͳ͘ͳͬͯΔ͕࣌ؒ͋Δ
    w ࣮͸1PE͕ϦΫΤετड͚ΒΕΔঢ়ଶʹͳΔલʹτϥϑΟοΫɹɹɹ
    ৼΒΕͪΌͬͯΔʁ
    30
    

    View Slide

31. 3PMMJOH6QEBUFͯ͠Δ͸͕ͣɺɺɺ
    w Πϝʔδ
    31
    1PE࡞੒։࢝
    1PE಺શͯͷ
    ίϯςφ͕ىಈ
    ࡞੒ͨ͠1PEʹ
    τϥϑΟοΫ͕
    ৼΒΕ࢝ΊΔ +7.ىಈ׬ྃ
    ʜ
    ͜ͷؒ
    ͣͬͱΤϥʔ
    

    View Slide

32. 3PMMJOH6QEBUFͯ͠Δ͸͕ͣɺɺɺ
    w 3FBEJOFTT1SPCF,VCFSOFUFTͷػೳ %FQMPZNFOU
    
    w 3FBEJOFTT1SPCFΛࢦఆ͠ͳ͍ঢ়ଶͰ͸ɺ,VCFSOFUFT͸1PE͕ىಈͨ͠ͱಉ࣌ʹ
    -# 4FSWJDF
    ʹ૊ΈೖΕΔ
    w +BWBΞϓϦ౳Ͱ͸ɺ+7.ͷॳظԽ΍ΩϟογϡͷॳճಡΈࠐΈ͕૸Γɺ1PEͷىಈ
    ͱΞϓϦέʔγϣϯ͕3FBEZͱͳΔλΠϛϯάʹ͸݁ߏϥά͕͋Δ
    w ͦ͜Ͱɺ3FBEJOFTT1SPCFͰΞϓϦέʔγϣϯͷ3FBEZͱͳΔ৚݅ͱͯ͠+BWB
    ΞϓϦέʔγϣϯ͕ىಈ͢Δ1PSU΁ͷϋʔτϏʔτΛࢦఆ͓͚ͯ͠͹ɺΞϓϦͷ
    ىಈલʹΞΫηε͕དྷΔ͜ͱΛ๷͛Δʁ
    32
    

    View Slide

33. 3PMMJOH6QEBUFͯ͠Δ͸͕ͣɺɺɺ
    w 3FBEJOFTT1SPCF
    33
    1PE࡞੒։࢝
    1PE಺શͯͷ
    ίϯςφ͕ىಈ
    ࡞੒ͨ͠1PEʹ
    τϥϑΟοΫ͕
    ৼΒΕ࢝ΊΔ
    +7.ىಈ׬ྃ
    ʜ
    3FBEJOFTT1SPCF0,
    3FBEJOFTT1SPCF
    /(
    ղܾ
    

    View Slide

34. ݹ͘ͳΓΏ͘,Tͷόʔδϣϯ
    w ύϒϦοΫΫϥ΢υ࢖ͬͯΔͱɺ͍ͭͷؒʹ͔࢖ͬͯΔ,Tͷόʔδϣϯ
    ͕αϙʔτର৅֎ʹ
    w /PEFͷόʔδϣϯΛ্͛Δʹ͸/PEFΛ࠶ىಈ͠ͳ͍ͱ͍͚ͳ͍
    w ࠶ىಈͰҰؾʹ1PE͕ࢮΜͩΒࠔΔ
    34
    

    View Slide

35. ݹ͘ͳΓΏ͘,Tͷόʔδϣϯ
    w Πϝʔδ
    w /PEFΛ࡟আ͔ͯ͠Β%FQMPZNFOU͕1PEΛ࡞Γ௚͢
    w
    35
    https://kubernetes.io/docs/tasks/run-application/configure-pdb/#specifying-a-poddisruptionbudget
    %FQMPZNFOU"
    %FQMPZNFOU#
    ෛՙूத
    

    View Slide

36. ݹ͘ͳΓΏ͘,Tͷόʔδϣϯ
    w 1PE%JTSVQUJPO#VEHFUNBY6OBWBJMBCMF
    w SFQMJDB਺ʹରͯ͠Քಇதͷ1PEΛ͔ͭ͠ݮΒͯ͠͸͍͚ͳ͍
    w
    36
    https://kubernetes.io/docs/tasks/run-application/configure-pdb/#specifying-a-poddisruptionbudget
    %FQMPZNFOU"
    %FQMPZNFOU#
    ղܾ
    

    View Slide

37. ݹ͘ͳΓΏ͘,Tͷόʔδϣϯ
    w 1PE%JTSVQUJPO#VEHFU,VCFSOFUFTͷػೳ
    w %FQMPZNFOU͝ͱʹɺ͍ͭ͘·ͰͳΒ1PEΛམͱͯ͠΋͍͍͔ΛఆٛͰ͖
    Δ
    w ͜ͷ৚݅Λຬͨͤͳ͍Α͏ͳૢ࡞Λߦ͓͏ͱ͢Δͱɺผͷ/PEFʹ৽͍͠
    SFBEZঢ়ଶͷ1PE͕Ͱ͖Δ·Ͱૢ࡞ΛਐΊͳ͍Α͏੍ݶͯ͘͠ΕΔ
    w όʔδϣϯΞοϓճͯ͠Δ͚Ͳ3FBEJOFTT1SPCFͱͷ૊Έ߹ΘͤͰ໰
    ୊ͳ͍
    37
    

    View Slide

38. 1PE͕ภͬͯΔɻɻɻ
    w ͋Δछྨͷ1PE͕ಛఆͷ/PEFʹภͬͯ͠·ͬͨ
    w ΋͠ͳΜΒ͔ͷཧ༝Ͱ/PEF͕མͪͨΒαʔϏε͕མͪΔ
    38
    

    View Slide

39. w Πϝʔδ
    w
    1PE͕ภͬͯΔɻɻɻ
    39
    %FQMPZNFOU"
    %FQMPZNFOU#
    %FQMPZNFOU$
    

    View Slide

40. 1PE͕ภͬͯΔɻɻɻ
    w *OUFSQPEBOUJB⒏OJUZ,VCFSOFUFTͷػೳ %FQMPZNFOU
    
    w ಛఆͷ1PE͕ಈ͍͍ͯΔ/PEFʹରͯ͠৽͍͠1PEΛεέδϡʔϦϯά
    ͠ͳ͍ઃఆ͕Ͱ͖Δɹ
    w ͜ΕΛ࢖ͬͯɺಉҰ%FQMPZNFOUͷ1PE͕͢Ͱʹଘࡏ͍ͯ͠Δ/PEF
    ʹεέδϡʔϦϯά͠ͳ͍Α͏ʹઃఆͰ͖Δ
    40
    

    View Slide

41. w *OUFSQPEBOUJB⒏OJUZͷઃఆ͋ΓʢಉҰ1PE͕ଘࡏ͢Δϊʔυʹ͸εέδϡʔϦϯά͠ͳ͍ʣ
    w
    1PE͕ภͬͯΔɻɻɻ
    41
    %FQMPZNFOU"
    %FQMPZNFOU#
    %FQMPZNFOU$
    

    View Slide

42. γεςϜܥ1PEམͪͯΔ
    w LVCFQSPYZͱ͍͏ɺϦόʔεϓϩΩγͷΑ͏ͳ΋ͷ͕མ͍ͪͯͯɺ1PE΁ͷΞΫη
    εʹࣦഊ͢Δ
    w γεςϜܥͷ1PE΋શͯ%BFNPO4FU΋͘͠͸%FQMPZNFOUͳͷͰʢLVCFQSPYZ͸
    %BFNPO4FUʣɺ໰୊ͷ͋Δ1PEΛ࡟আ͢Ε͹ࣗಈతʹ࠶࡞੒͞Εͯ෮چ͸Մೳ
    w γεςϜܥ1PE΋མͪͳ͍อূ͸ͳ͍͠ɺো֐࣌ͷ੾Γ෼͚͕େมͳͷͰɺɹɹɹ
    ͪΌΜͱ؂ࢹ͸͓͍ͯͨ͠΄͏͕Αͦ͞͏
    w (,&ͳΒ૊ΈࠐΈͰTUBDLESJWFS͕ར༻Մೳɺ΋ͪΖΜ%BUBEPH౳Ͱ΋؂ࢹՄೳ
    w TUBDLESJWFSͰ֤ϊʔυͭҎ্LVCFQSPYZ্͕͕͍ͬͯͳ͚Ε͹Ξϥʔτ
    42
    

    View Slide

43. ೔ʑؾΛ͚͍ͭͯΔ͜ͱ
    w νʔϜͰษڧձΛߦͬͨΓυΩϡϝϯτΛڞ༗ͨ͠Γͯ͠஌ࣝͷఈ্͛
    w υΩϡϝϯτ؅ཧʹ͸4DSBQCPYΛ࢖͍ͬͯ·͢ IUUQTTDSBQCPYJP
    
    w ฤू͠΍͘͢ɺݕࡧ͠΍͍͢
    w λά΍ϦϯΫΛ࢖ͬͯυΩϡϝϯτಉ࢜Λؔ࿈͚ͮͯɺ஌ࣝΛܨ͍͚͛ͯΔ
    w ࣗಈԽ
    w LVCFDUMΛͳΔ΂͘ਓ͕ؒୟ͔ͳ͍Α͏ʹ͍ͯ͘͠
    w HJU0QT
    w +FOLJOTΛ࢖ͬͨΦϖϨʔγϣϯࣗಈԽ
    43
    

    View Slide

44. ·ͱΊ
    w ϦϦʔε࡞ۀ͕؆୯Ͱ҆શˠߴ଎ϦϦʔε
    w ࣗ཯తʹਖ਼ৗͳঢ়ଶΛҡ࣋ͯ͘͠ΕΔˠলྗӡ༻
    w ৭ʑ໰୊͸ग़ͯ͘Δ͚Ͳɺ,VCFSOFUFTͷ࢓૊ΈͷதͰղܾͰ͖ͦ͏
    w ୯ʹ,VCFSOFUFTΛ࢖͏͚ͩͰͦͷ··ӡ༻ָ͕ʹͳΔΘ͚Ͱ͸ͳ͍
    w ,VCFSOFUFTͷػೳΛཧղ্ͨ͠Ͱࡌͤସ͑Ε͹ӡ༻Λָʹͯ͘͠ΕΔ
    44
    

    View Slide

45. ,VCFSOFUFTɺ࠷ߴ
    45
    

    View Slide

46. 46
    ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
    

    View Slide