コンテナたちを計測すること ~ ロリポップ！マネージドクラウドの裏側 ~ /wip-how-to-get-container-metrics

Cookpad Tech Kitchen #12

Transcript

1. dϩϦϙοϓʂϚωʔδυΫϥ΢υͷཪଆd
   ۙ౻Ӊஐ࿕(.01FQBCP *OD
   $PPLQBE5FDI,JUDIFO
   ίϯςφͨͪΛܭଌ͢Δ͜ͱ
   

   View Slide

2. ΤϯδχΞ
   ۙ౻Ӊஐ࿕!VE[VSB
   ٕज़෦ٕज़ج൫νʔϜ
   ޷͖ͳϥʔϝϯ͸ΒʔΊΜ޻๪ཾ
   (.0ϖύϘॴଐ
   ΍ͬͯΔ෼໺3VCZɺNSVCZɺίϯςφ
   ΍Γ͍ͨ෼໺γεςϜϓϩάϥϛϯάɺ-JOVY֦ு
   

   View Slide

3. ؂ࢹɾܭଌʹ͍ͭͯ͸ɺࠓ΋
   ࢼߦࡨޡ͕ଓ͍͍ͯ·͕͢
   ࠓ೔͸ݕূͷաఔΛڞ༗͠·͢
   

   View Slide

4. ϩϦϙοϓʂϚωʔδυΫϥ΢υ
   IUUQTNDMPMJQPQKQ
   ΞϧϑΝςετத
   

   View Slide

5. IUUQTNDMPMJQPQKQ
   ΞϧϑΝςετத
   ˞͝ڵຯͷ͋Δํ͸ձ৔Ͱ੠Λֻ͓͚Լ͍͞
   

   View Slide

6. ؂ࢹͱ͸
   

   View Slide

7. *5Πϯϑϥͷ؂ࢹͱ͸ɺຊॻͰ͸࣍ͷ௨Γఆ
   ٛ͠·͢ɻ
   ো֐ൃੜͷ༗ແΛҰఆظؒͰ଺Γͳ֬͘ೝ͢
   Δ͜ͱɻ
   ʰ*5Πϯϑϥ؂ࢹ<࣮ફ>ೖ໳ʱΑΓ
   

   View Slide

8. ؂ࢹɾϝτϦΫεͷ໨త
   wӡ༻ͱͯ͠
   wαʔϏεμ΢ϯʹ͙͢ʹؾ෇͖͍ͨɻ·ͨɺ෮چΛૉૣ͍ͨ͘͠
   wαʔϏεͷػೳͱͯ͠
   wίϯςφͷෛՙঢ়گ΍ɺো֐ঢ়ଶʹͳ͍ͬͯΔ͔൱͔ͳͲɻ·ͨͦͷ
   ཤྺΛͪΌΜͱՄࢹԽͯ͠ఏڙͰ͖Δͱخ͍͠
   wˠͦͷͨΊʹɺϝτϦΫεΛܭଌɺ؂ࢹɺෆਖ਼ͳঢ়ଶͷ࣌͸Ξϥʔ
   τͳͲΛ͋͛ΒΕΔΑ͏ʹ͠ͳ͚Ε͹ͳΒͳ͍ɻ
   

   View Slide

9. ؂ࢹͷͨΊʹ
   ·ͣ͸ܭଌ͠ͳ͍ͱ͍͚ͳ͍
   

   View Slide

10. ࠓɺࠔ͍ͬͯΔ͜ͱ
    wίϯςφ͸ී௨ͷ7.ͷ؂ࢹͷΑ͏ʹ͍͔ͳ͍ͱ͜Ζ΋͋Δ
    wˠϓϩηεʢ܈ʣͳͷͰ-"ͷΑ͏ͳఆ൪ͷ஋͕औΕͳ͔ͬͨΓͳͲ
    wίϯςφϗεςΟϯάͰ͋Δ͜ͱɺ·ͨϚΠΫϩαʔϏεతͰ͋Δ͜
    ͱʹΑΔɺͭͷʮ਺ʯͷ໰୊͕͋Δ
    

    View Slide

11. ʮίϯςφ਺ʯͷ໰୊
    wϚωʔδυΫϥ΢υ͸ɺίϯςφΛར༻ͨ͠ߴूੵϓϥοτϑΥʔϜ
    Ͱ͋Δɻ͢ͳΘͪɺίϯςφ͕ͱͯ΋ଟ͍ɻ
    w͔͠΋ɺ'BTU$POUBJOFSΛ࠾༻͍ͯ͠ΔͳͲɺίϯςφ͸ىಈ͠ଓ͚
    ΔΘ͚Ͱ͸ͳ͍ɻ·ͨɺҰͭ࣌ͷυϝΠϯʹඥͮ͘ίϯςφͷ਺΋ม
    ΘΔɻ
    wͨ͘͞Μ͋ΔͷͰɺूΊΔ໰୊ͱɺ෼͔Γ΍͘͢ݟͤΔ໰୊͕͋Δ
    

    View Slide

12. ʮίϯϙʔωϯτ਺ʯͷ໰୊
    wԿ౓͔ର֎ൃද͍ͯ͠Δ௨ΓɺϚωʔδυΫϥ΢υ͸ίϯϙʔωϯτ
    ͕ଟ͘ɺϚΠΫϩαʔϏεతʹ૊Έ߹Θ͍ͬͯ͞Δ
    wʮαΠτ͕ݟ͑ͳ͍ʯͷΑ͏ͳ৔߹ʹɺݪҼͱߟ͑ΒΕΔՕॴͷಛఆ
    ͕೉͘͠ɺΤϥʔτϥοΩϯάʹ՝୊͕͋Δɻ
    wϘτϧωοΫͷಛఆͳͲ΋ಉ༷
    

    View Slide

13. গͣͭ͠ɺ֎ງΛຒΊΔ
    ͦΜͳઓ͍ͷաఔΛ࿩͠·͢
    

    View Slide

14. ͲΜͳ஋͕औΕΔ͔ʁ
    

    View Slide

15. ܭଌͷର৅
    w$16ϝϞϦར༻཰ͷ؂ࢹɾܭଌ
    wωοτϫʔΫͷ؂ࢹɾܭଌ
    w*0΍σΟεΫͷ؂ࢹɾܭଌ
    wϦΫΤετͷτϨʔγϯά
    wίϯςφ಺ϛυϧ΢ΣΞͷϩάʢ͜Ε͸ϩάύʔεͰૉ௚ʹूΊΔʣ
    

    View Slide

16. ࢀߟ
    wͦʔ͍ͩ͞ΜͷࢿྉΑΓɺجຊతͳγεςϜʹؔΘΔϝτϦοΫ
    IUUQTPVEBJIBUFOBCMPHDPNFOUSZ
    

    View Slide

17. DHSPVQͷTUBUͷར༻
    wDQVDQVBDDUαϒγεςϜ
    wNFNPSZαϒγεςϜ
    w ଞɺCMLJP WͷJP
    
    wςΩετσʔλͰɺར༻ঢ়گ͕Θ͔Δˠύʔεͯ͠׆༻͢Δ
    DHSPVQࣗମʹ͍ͭͯ͸ʮDHSPVQVE[VSBʯ·ͨ͸

    ʮDHSPVQ5FO'PSXBSEʯͰݕࡧ͢Δͱ೔ຊޠͷղઆ͕ɻɻɻ
    root@compute-1:~# cat /sys/fs/cgroup/cpu/$CONTAINER/cpu.stat
    nr_periods 140373
    nr_throttled 25
    throttled_time 1112602844
    

    View Slide

18. ωοτϫʔΫʹؔ͢ΔϝτϦΫε
    wݪଇɺίϯςφ͸ࣗ෼ͷ/FUXPSL/BNFTQBDFʹॴଐ͢Δ
    w/FUXPSL/BNFTQBDFͲ͏͠ʢࢠ͔Β਌ʣ͸ɺWFUIσόΠεΛܦ༝
    ͯ͠௨৴͢Δ͜ͱʹͳΔ
    wͦͷWFUIσόΠεʹؔ͢Δ஋͸ز͔ͭͱΔखஈ͕͋Δ
    wJGUPQͷΑ͏ͳίϚϯυɺQSPDOFUEFWϑΝΠϧ
    wϓϩηε୯ҐͷωοτϫʔΫ৘ใQSPD1*%OFU\EFW UDQ VEQ ^
    

    View Slide

19. ':*ίϯςφωοτϫʔΫͷτϙϩδ
    wCSJEHFઃఆͷྫɻ%PDLFSͷσϑΥϧτઃఆ΋͜Εɻ
    FUI
    EPDLFS
    WFUIIPTU
    WFUIHVFTU
    WFUIIPTU
    WFUIHVFTU
    WFUIIPTU
    WFUIHVFTU
    FH
    FH
    
    ඞཁʹԠ͡/"5ͳͲ
    /FUXPSL
    /BNFTQBDF
    ͻͱͭͷ)PTU
    CSJEHF
    ͜ΕΒΛ؂ࢹ͢Δʂ
    

    View Slide

20. *0ΞΧ΢ϯςΟϯά
    wDHSPVQͷCMLJPJP͸ɺσόΠε୯Ґͷܭଌ΍εϩοτϧͳͷͰɺ/'4
    ͷΑ͏ʹσόΠε൪߸͕ৼΒΕͳ͍ϑΝΠϧγεςϜʹ͸೉͍͠
    wҰํɺϓϩηε୯ҐͷSFBEʢ଒ʣɺXSJUFʢ଒ʣͷݺͼग़͠ճ਺΍ɺ
    ಡΈࠐΈॻ͖ࠐΈʹ੒ޭͨ͠όΠτ਺ͷྦྷܭ͸ɺΧʔωϧ͕ूܭͯ͠
    ͍ͯར༻Ͱ͖Δɻ
    wQSPD1*%JPϑΝΠϧܦ༝Ͱ؆୯ʹ֬ೝՄೳ
    wྫ͑͹͜ΕΛίϯςφʹॴଐ͢Δϓϩηε͝ͱʹूܭ͢Δ
    

    View Slide

21. ϦΫΤεττϥοΩϯά
    w͋ΔαʔόΛ௨ա͢Δ͝ͱʹಛఆͷϔομʔʹ࣌ؒΛ௥ه͍ͯ͘͠
    wDG9'PSXBSEFE'PSϔομʔ
    w3BDL౳ͷൃ૝ΛਅࣅͯɺϚʔΧʔΛωετ͍ͯ͘͠ͷ͸Ͳ͏͔ʁ
    wίϯϙωϯτ͝ͱͷܦա͕࣌ؒΘ͔ΔΑ͏ʹͳΓɺϘτϧωοΫͳͲ
    ͕Θ͔ΔɻͦΕͦ͜(SBGBOBͳͲͰՄࢹԽͯ͠΋ྑ͍
    wΤϥʔτϥοΩϯάʹ΋Ԡ༻Ͱ͖Δʁ
    IUUQTQZMPOTXFCGSBNFXPSLSFBEUIFEPDTJPFOWSDDPODFQUTIUNM
    

    View Slide

22. DGޚࣾͷ෼ࢄτϨʔγϯάج൫
    wؾʹͳΓ·͢
    IUUQUFDIMJGFDPPLQBEDPNFOUSZ
    

    View Slide

23. σʔλΛཷΊࠐΉ
    

    View Slide

24. େ·͔ͳΞʔΩςΫνϟ
    wूΊΔˠஷΊΔˠՄࢹԽ͢Δ
    wσʔλͷอଘઌ1SPNFUIFVT
    wՄࢹԽ(SBGBOB
    wσʔλΤʔδΣϯτD"EWJTPS΍NUBJMɺࣗ࡞FYQPSUFS
    

    View Slide

25. 1SPNFUIFVT
    (SBGBOB
    &YQPSUFS
    &YQPSUFS
    &YQPSUFS
    $POUBJOFS
    $POUBJOFS
    $POUBJOFS
    $POUBJOFS
    $POUBJOFS
    "1*޲͚1SPYZ
    ूܭϝτϦΫεͷάϧʔϓผʹ
    FYQPSUFSΛ༻ҙ
    Ϣʔβ༷μογϡϘʔυ

    ʢ։ൃதʣ
    ӡ༻ΤϯδχΞ

    σΟϨΫλʔͳͲ
    "MFSUNBOBHFS
    

    View Slide

26. 1SPNFUIFVTͷ࠾༻
    w(PPHMFͷ#PSHNPOʹӨڹΛड͚ͨ؂ࢹπʔϧ
    w#PSHNPOͷ঺հ͸43&ຊʹ΋͋Δͦ͏
    wΠϯετʔϧʢجຊతʹ͸ϫϯόΠφϦΛ഑උ͢Ε͹0,ʣ΍ɺ

    ӡ༻ʢجຊతʹΦϯϝϞϦʴϩʔΧϧετϨʔδʣ͕༰қ
    wॊೈͳσʔλߏ଄ͱߴػೳͳΫΤϦ
    w"MFSU࿈ܞ͕༰қ
    w(SBGBOBͳͲՄࢹԽπʔϧ΍֤छFYQPSUFS͕ἧ͍ͬͯΔ
    

    View Slide

27. 1SPNFUIFVTͱ(SBGBOBͷ࿈ܞ
    w(SBGBOB͸΋͸΍આ໌ͷඞཁ΋ͳ͍΄Ͳͷఆ൪μογϡϘʔυ
    wެࣜʹ΋υΩϡϝϯτ͕͋Δ
    wIUUQTQSPNFUIFVTJPEPDTWJTVBMJ[BUJPOHSBGBOB
    w1SPNFUIFVTͷΫΤϦΛͦͷ·Μ·ॻ͚ͨΓͯ͠ศརײ͕ߴ͍
    

    View Slide

28. Մ༻ੑͷߟ͑ํ
    w1SPNFUIFVT͸֤ϊʔυͷFYQPSUFS͔ΒσʔλΛQVMM͍ͯ͘͠ελΠ
    ϧΛͱ͍ͬͯΔʢDG/BHJPT/31&4BB4ܕ؂ࢹʣ
    wͦͷͨΊ"DUJWF"DUJWFߏ੒ΛऔΕ͹ಉ͡σʔλʹͳΓɺ৑௕ߏ੒ͱ
    ͍͏͜ͱʹͳΔɻ
    wSFGIUUQTHJUIVCDPNQSPNFUIFVTQSPNFUIFVTJTTVFT
    

    View Slide

29. 4FSWJDF%JTDPWFSZ
    wϊʔυ͔ΒQVMM͢ΔͷͰɺ1SPNFUIFVT͸ϊʔυ͕Ͳ͜ʹ͋Δͷ͔ͱ
    ͍͏৘ใΛ͍࣋ͬͯͳ͍ͱ͍͚ͳ͍ɻ
    w4FSWJDF%JTDPWFSZͷͨΊͷํ๏ࣗମ͸͍͔ͭ͋͘Δɻ
    wϑΝΠϧϕʔεɺ%/4ϕʔεɺ"840QFO4UBDLͳͲ%ZOBNJD*OGSB
    ޲͚ઃఆɺ͋Δ͍͸.BSBUIPO΍$POTVMͱ΋࿈ܞՄೳ
    

    View Slide

30. FYQPSUFSͷ֓೦ʹ͍ͭͯ
    wσʔλΛऔΓ͍ͨϊʔυʹɺFYQPSUFSͱݴ͏ܾ·ͬͨϑΥʔϚοτͰ
    σʔλΛు͖ग़͢ਓΛৗற͓ͤͯ͘͞ʢ࣮ଶ͸IUUQ"1*ʣɻ
    w1SPNFUIFVT͸ɺ4FSWJDF%JTDPWFSZΛͯ͠ɺͦͷFYQPSUFSʹΞΫη
    ε͠ɺఆظతʹσʔλΛಘΔ
    wFYQPSUFS͸ɺطʹ࣮૷͞Ε͍ͯΔ΋ͷ΋ͨ͘͞Μ͋Δ͠ɺ

    ࣗ෼Ͱ࣮૷͢Δ͜ͱ΋Մೳɻ
    

    View Slide

31. FYQPSUFSͷྫ
    wOPEF@FYQPSUFSl&YQPSUFSGPSNBDIJOFNFUSJDTz
    w$16ɺϝϞϦɺ-"ɺωοτϫʔΫͳͲϚγϯ୯ҐͰ৭ʑͱूܭ
    w(SBGBOBͳͲͰ͍͍ײ͡ʹάϥϑԽͯ͠׆༻
    wD"EWJTPS(PPHMFࣾ಺޲͚ίϯςφ؂ࢹج൫Λϕʔεʹɺ%PDLFSͳ
    ͲʹରԠͨ͠΋ͷɻͪͳΈʹTZTUFNEͳͲͷ؂ࢹͰ΋ศར
    wͦͷଞɺଟ͘ͷ࣮૷͕͋Δɻී௨͸ࣗ࡞͠ͳ͍͍ͯ͘͸ͣ

    IUUQTHJUIVCDPNQSPNFUIFVTEPDTCMPCNBTUFSDPOUFOUEPDTJOTUSVNFOUJOHFYQPSUFSTNE
    

    View Slide

32. D"EWJTPSͰ)BDPOJXBͷϝτϦΫεΛऔΔ
    

    View Slide

33. D"EWJTPSͰ)BDPOJXBͷϝτϦΫεΛऔΔ
    

    View Slide

34. NUBJMIUUQTHJUIVCDPNHPPHMFNUBJM
    wઌड़ͨ͠/HJOYͳͲͷϩάΛύʔε্ͨ͠ͰɺFYQPSUFSʹ΋ͳͬͯ͘
    ΕΔπʔϧ
    w%4-ʹΑΓϧʔϧΛهड़
    wԿ͔طࢹײͷ͋Δྗڧ͞

    DG'MVFOUEͷJO@UBJM
    
    w/HJOYͷྫˠ
    IUUQTHJUIVCDPNOEJB[HOHJOYQSPNFUIFVTFYQPSUFSCMPCNBTUFSpMFTQSPHTOHJOYNUBJMɹ
    

    View Slide

35. ܭଌͷର৅
    w$16ϝϞϦར༻཰ͷ؂ࢹɾܭଌ
    wωοτϫʔΫͷ؂ࢹɾܭଌ
    w*0΍σΟεΫͷ؂ࢹɾܭଌ
    wϦΫΤετͷτϨʔγϯά
    wίϯςφ಺ϛυϧ΢ΣΞͷϩά
    (PPHMFπʔϧ܈ͷΤίγεςϜ͕ศརɻ
    ˡD"EWJTFSͰऔΕΔ
    ˡD"EWJTFSͰ޻෉͢Ε͹औΕΔ
    ˡ·ͩཁݕ౼
    ˡNUBJMͰָͰ͖Δ
    ˡώοʢOHY@NSVCZͳͲͰؤுΕͦ͏ʣ
    

    View Slide

36. FYQPSUFSࣗ࡞
    wͱ͸͍͑ɺࣗ࡞΋Ұͭͷख

    
    (PͳͲͳΒ1SPNFUIFVTެࣜͷϥΠϒϥϦ͕࢖͑Δ

    
    طଘ࣮૷ׂ͕ͱ๛෋

    
    ͦ΋ͦ΋ɺຬͨ͢΂͖ϑΥʔϚοτ͕୯७
    wIUUQTQSPNFUIFVTJPEPDTJOTUSVNFOUJOHXSJUJOH@FYQPSUFST

    ΨΠυΛݟͳ͕Βɺ͋ͱطଘ࣮૷ΛோΊͯॻ͘ͱྑͦ͞͏
    

    View Slide

37. FYQPSUFSͱ1SPNFUIFVTͷ࿈ܞ
    w4FSWJDF%JTDPWFSZͷதͰ΋$POTVMͱͷ૬ੑ͸ྑ͍ͱ͍͏ࢲݟ
    wFYQPSUFSଆͰɺ$POTVMͷTFSWJDFDIFDLΛ௥ՃˠಛఆͷαʔϏεΛ
    1SPNFUIFVT͔ΒσΟεΧόϦɺ͕ඇৗʹ༰қʹͳ͍ͬͯΔ
    wྫ͑͹Ұ୆ʹෳ਺FYQPSUFSͳͲ΋؆୯ͩ͠ɺFYQPSUFSͷ௥Ճ࡟আɺ
    ࢮ׆؂ࢹ΋$POTVMʹ৐Δ͜ͱͰߟ͑ͳͯ͘ࡁΉ
    

    View Slide

38. ͜ΜͳTFSWJDFఆٛΛॻ͚͹
    wεοͱ+0*/͢Δ
    wศར
    {
    "service": {
    "name": "node_exporter",
    "tags": [
    "development"
    ],
    "port": 9100,
    "checks": [
    {
    "id": "ping",
    "tcp": "localhost:9100",
    "interval": "15s",
    "timeout": "1s"
    }
    ]
    }
    }
    

    View Slide

39. ͜ͷষ·ͱΊ
    wʮσʔλͷஷΊͲ͜Ζʯʹ1SPNFUIFVTΛਅΜதʹਾ͑Δ
    w1SPNFUIFVTͷΤίγεςϜ͸ڧ͍ɻ
    w͍͟ͱͳΕ͹ཉ͍͠σʔλ͸ࣗ෼ͰऔΓʹߦ͘
    w͜ΕΒͷͭͳ͗ʹ$POTVM͕ศར
    

    View Slide

40. ࠓޙ΍Γ͍ͨ͜ͱ
    

    View Slide

41. ࠓޙ΍Γ͍ͨ͜ͱ
    wूΊΔͱ͜Ζ·Ͱ͸Կͱ͔ͳΓͦ͏
    wཷΊࠐΜͩσʔλͷ׆༻ʹ͍ͭͯ
    w"MFSUNBOBHFSͷ֓೦ʢࠓ೔͸লུʣ
    wҟৗݕ஌ͷࣗಈԽɺՄࢹԽͳͲ
    

    View Slide

42. ҟৗݕ஌ͷࣗಈԽɺՄࢹԽ
    wίϯςφʹؔ͢ΔϝτϦοΫε͸ɺ$.%#ܦ༝Ͱू໿Ͱ͖Δঢ়ଶ
    wϓϩδΣΫτ͝ͱͷσʔλΛूΊͯԿ͔Ͱ͖ͳ͍͔
    1SPKFDUT
    EPNBJO
    $POUBJOFST
    $*% DPOUBJOFSJE
    
    $POUBJOFST
    $*% DPOUBJOFSJE
    
    .FUSJDT\EBUBlBBBz ^999
    .FUSJDT\EBUBlCCCz ^:::
    .FUSJDT\EBUBlDDDz ^;;;
    .FUSJDT\EBUBlEEEz ^888
    

    View Slide

43. ूΊͨσʔλΛͲ͏͢Δ͔
    w͜Ε͕ɺࠓޙͷ՝୊
    w୯७ʹάϥϑԽͯ͠Έͯ΋ɺͦΕͳΓʹঢ়ଶ͕Θ͔Δؾ΋͢Δ͠ɺٸ
    ʹෛՙ͕ߴ͍ɺ௨৴Ͱ͖ͳ͘ͳͬͯΔɺͳͲ͸໨ࢹͰ͖ͦ͏Ͱ͸͋Δ
    w·ͣ͸ӡ༻ͯ͠Έ͍ͨͱ͜Ζ
    

    View Slide

44. ՄࢹԽɾࣗಈ൑ఆͷख๏
    wϓϩδΣΫτʹඥ෇͍ͨσʔλ͸ɺجຊతʹଟ࣍ݩͷσʔλͰ͋Δͷ
    Ͱɺ࣍ݩѹॖΛͯ͠ɺʮ͜Ε͸ਖ਼ৗʯʮ͜Ε͸ো֐ঢ়ଶʯͷΑ͏ʹ෼
    ྨ͕Ͱ͖ΔΜͰ͸ͳ͍͔ͱߟ͍͑ͯΔ͕ɺࠓޙͷ՝୊
    wେྔͷίϯςφϓϩδΣΫτΛɺࣗݾ૊৫ԽϚοϓͳͲͰࣗಈͰ

    ෼ྨͯ͠ɺͦΕΛՄࢹԽͰ͖Δͱ໘നͦ͏ʁ
    w͜ͷลΓʹɺʮ਺ͷ໰୊ʯͷղܾࡦ͕ݟ͑Δؾ͕͍ͯ͠Δ
    

    View Slide

45. 5P#F$POUJOVFE
    

    View Slide

46. ·ͩ·ͩख୳Γͷ
    ཁૉ΋ଟ͍Ͱ͕͢
    ΍Γ͕͍͕͋Γ·͢ʂ
    

    View Slide

47. View Slide

48. ϚωʔδυΫϥ΢υΛҰॹʹ࡞Γ·͠ΐ͏
    ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU
    QIPUPIUUQTUFDIQFQBCPDPNFOHJOFFSUSBJOJOHWPM
    

    View Slide