コンテナたちを計測すること ~ ロリポップ！マネージドクラウドの裏側 ~ /wip-how-to-get-container-metrics

Cookpad Tech Kitchen #12

Transcript

1. d
   ϩϦϙοϓʂϚωʔδυΫϥ΢υͷཪଆ
   d ۙ౻Ӊஐ࿕
   
   (.0
   1FQBCP
   *OD
   
   $PPLQBE
   5FDI
   ,JUDIFO
    ίϯςφͨͪΛܭଌ͢Δ͜ͱ
2. ΤϯδχΞ ۙ౻Ӊஐ࿕
   !VE[VSB ٕज़෦ٕज़ج൫νʔϜ ޷͖ͳϥʔϝϯ͸ΒʔΊΜ޻๪ཾ (.0ϖύϘॴଐ ΍ͬͯΔ෼໺
   3VCZɺNSVCZɺίϯςφ ΍Γ͍ͨ෼໺
   γεςϜϓϩάϥϛϯάɺ-JOVY֦ு
3. ؂ࢹɾܭଌʹ͍ͭͯ͸ɺࠓ΋
   ࢼߦࡨޡ͕ଓ͍͍ͯ·͕͢
   ࠓ೔͸ݕূͷաఔΛڞ༗͠·͢
4. ϩϦϙοϓʂϚωʔδυΫϥ΢υ IUUQTNDMPMJQPQKQ
   ΞϧϑΝςετத
5. IUUQTNDMPMJQPQKQ
   ΞϧϑΝςετத ˞͝ڵຯͷ͋Δํ͸ձ৔Ͱ੠Λֻ͓͚Լ͍͞
6. ؂ࢹͱ͸
7. *5Πϯϑϥͷ؂ࢹͱ͸ɺຊॻͰ͸࣍ͷ௨Γఆ ٛ͠·͢ɻ
   ো֐ൃੜͷ༗ແΛҰఆظؒͰ଺Γͳ֬͘ೝ͢ Δ͜ͱɻ ʰ*5Πϯϑϥ؂ࢹ<࣮ફ>ೖ໳ʱΑΓ
8. ؂ࢹɾϝτϦΫεͷ໨త wӡ༻ͱͯ͠
   wαʔϏεμ΢ϯʹ͙͢ʹؾ෇͖͍ͨɻ·ͨɺ෮چΛૉૣ͍ͨ͘͠
   wαʔϏεͷػೳͱͯ͠
   wίϯςφͷෛՙঢ়گ΍ɺো֐ঢ়ଶʹͳ͍ͬͯΔ͔൱͔ͳͲɻ·ͨͦͷ ཤྺΛͪΌΜͱՄࢹԽͯ͠ఏڙͰ͖Δͱخ͍͠
   wˠ
   ͦͷͨΊʹɺϝτϦΫεΛܭଌɺ؂ࢹɺෆਖ਼ͳঢ়ଶͷ࣌͸Ξϥʔ τͳͲΛ͋͛ΒΕΔΑ͏ʹ͠ͳ͚Ε͹ͳΒͳ͍ɻ
9. ؂ࢹͷͨΊʹ
   ·ͣ͸ܭଌ͠ͳ͍ͱ͍͚ͳ͍
10. ࠓɺࠔ͍ͬͯΔ͜ͱ wίϯςφ͸ී௨ͷ7.ͷ؂ࢹͷΑ͏ʹ͍͔ͳ͍ͱ͜Ζ΋͋Δ
    wˠ
    ϓϩηεʢ܈ʣͳͷͰ-"ͷΑ͏ͳఆ൪ͷ஋͕औΕͳ͔ͬͨΓͳͲ
    wίϯςφϗεςΟϯάͰ͋Δ͜ͱɺ·ͨϚΠΫϩαʔϏεతͰ͋Δ͜ ͱʹΑΔɺͭͷʮ਺ʯͷ໰୊͕͋Δ
11. ʮίϯςφ਺ʯͷ໰୊ wϚωʔδυΫϥ΢υ͸ɺίϯςφΛར༻ͨ͠ߴूੵϓϥοτϑΥʔϜ Ͱ͋Δɻ͢ͳΘͪɺίϯςφ͕ͱͯ΋ଟ͍ɻ
    w͔͠΋ɺ'BTU$POUBJOFSΛ࠾༻͍ͯ͠ΔͳͲɺίϯςφ͸ىಈ͠ଓ͚ ΔΘ͚Ͱ͸ͳ͍ɻ·ͨɺҰͭ࣌ͷυϝΠϯʹඥͮ͘ίϯςφͷ਺΋ม ΘΔɻ
    wͨ͘͞Μ͋ΔͷͰɺूΊΔ໰୊ͱɺ෼͔Γ΍͘͢ݟͤΔ໰୊͕͋Δ
12. ʮίϯϙʔωϯτ਺ʯͷ໰୊ wԿ౓͔ର֎ൃද͍ͯ͠Δ௨ΓɺϚωʔδυΫϥ΢υ͸ίϯϙʔωϯτ ͕ଟ͘ɺϚΠΫϩαʔϏεతʹ૊Έ߹Θ͍ͬͯ͞Δ
    wʮαΠτ͕ݟ͑ͳ͍ʯͷΑ͏ͳ৔߹ʹɺݪҼͱߟ͑ΒΕΔՕॴͷಛఆ ͕೉͘͠ɺΤϥʔτϥοΩϯάʹ՝୊͕͋Δɻ
    wϘτϧωοΫͷಛఆͳͲ΋ಉ༷
13. গͣͭ͠ɺ֎ງΛຒΊΔ
    ͦΜͳઓ͍ͷաఔΛ࿩͠·͢
14. ͲΜͳ஋͕औΕΔ͔ʁ
15. ܭଌͷର৅ w$16
    
    ϝϞϦར༻཰ͷ؂ࢹɾܭଌ
    wωοτϫʔΫͷ؂ࢹɾܭଌ
    w*0΍σΟεΫͷ؂ࢹɾܭଌ
    wϦΫΤετͷτϨʔγϯά
    wίϯςφ಺ϛυϧ΢ΣΞͷϩάʢ͜Ε͸ϩάύʔεͰૉ௚ʹूΊΔʣ
16. ࢀߟ wͦʔ͍ͩ͞ΜͷࢿྉΑΓɺجຊతͳγεςϜʹؔΘΔϝτϦοΫ IUUQTPVEBJIBUFOBCMPHDPNFOUSZ
    
17. DHSPVQͷTUBUͷར༻ wDQVDQVBDDU
    αϒγεςϜ
    wNFNPSZ
    αϒγεςϜ
    w ଞɺCMLJP
    WͷJP
    
    wςΩετσʔλͰɺར༻ঢ়گ͕Θ͔Δˠύʔεͯ͠׆༻͢Δ
    DHSPVQࣗମʹ͍ͭͯ͸ʮDHSPVQ
    VE[VSBʯ·ͨ͸
 ʮDHSPVQ
    5FO'PSXBSEʯͰݕࡧ͢Δͱ೔ຊޠͷղઆ͕ɻɻɻ root@compute-1:~# cat /sys/fs/cgroup/cpu/$CONTAINER/cpu.stat nr_periods 140373 nr_throttled 25 throttled_time 1112602844
18. ωοτϫʔΫʹؔ͢ΔϝτϦΫε wݪଇɺίϯςφ͸ࣗ෼ͷ/FUXPSL
    /BNFTQBDFʹॴଐ͢Δ
    w/FUXPSL
    /BNFTQBDF
    Ͳ͏͠ʢࢠ͔Β਌ʣ͸ɺWFUIσόΠεΛܦ༝ ͯ͠௨৴͢Δ͜ͱʹͳΔ
    wͦͷWFUIσόΠεʹؔ͢Δ஋͸ز͔ͭͱΔखஈ͕͋Δ
    wJGUPQͷΑ͏ͳίϚϯυɺ
    QSPDOFUEFW
    ϑΝΠϧ
    wϓϩηε୯ҐͷωοτϫʔΫ৘ใ
    QSPD1*%OFU\EFW UDQ VEQ ^
19. ':*
    ίϯςφωοτϫʔΫͷτϙϩδ wCSJEHFઃఆͷྫɻ%PDLFSͷσϑΥϧτઃఆ΋͜Εɻ FUI EPDLFS WFUIIPTU WFUIHVFTU WFUIIPTU WFUIHVFTU WFUIIPTU WFUIHVFTU FH
     FH
        ඞཁʹԠ͡/"5ͳͲ /FUXPSL
    /BNFTQBDF ͻͱͭͷ)PTU CSJEHF ͜ΕΒΛ؂ࢹ͢Δʂ
20. *0
    ΞΧ΢ϯςΟϯά wDHSPVQͷCMLJPJP͸ɺσόΠε୯Ґͷܭଌ΍εϩοτϧͳͷͰɺ/'4 ͷΑ͏ʹσόΠε൪߸͕ৼΒΕͳ͍ϑΝΠϧγεςϜʹ͸೉͍͠
    wҰํɺϓϩηε୯ҐͷSFBEʢ଒ʣɺXSJUFʢ଒ʣͷݺͼग़͠ճ਺΍ɺ ಡΈࠐΈॻ͖ࠐΈʹ੒ޭͨ͠όΠτ਺ͷྦྷܭ͸ɺΧʔωϧ͕ूܭͯ͠ ͍ͯར༻Ͱ͖Δɻ
    wQSPD1*%JP
    ϑΝΠϧܦ༝Ͱ؆୯ʹ֬ೝՄೳ
    wྫ͑͹͜ΕΛίϯςφʹॴଐ͢Δϓϩηε͝ͱʹूܭ͢Δ
21. ϦΫΤεττϥοΩϯά w͋ΔαʔόΛ௨ա͢Δ͝ͱʹಛఆͷϔομʔʹ࣌ؒΛ௥ه͍ͯ͘͠
    wDG
    9'PSXBSEFE'PS
    ϔομʔ
    w3BDL
    ౳ͷൃ૝ΛਅࣅͯɺϚʔΧʔΛωετ͍ͯ͘͠ͷ͸Ͳ͏͔ʁ
    wίϯϙωϯτ͝ͱͷܦա͕࣌ؒΘ͔ΔΑ͏ʹͳΓɺϘτϧωοΫͳͲ ͕Θ͔ΔɻͦΕͦ͜(SBGBOBͳͲͰՄࢹԽͯ͠΋ྑ͍
    wΤϥʔτϥοΩϯάʹ΋Ԡ༻Ͱ͖Δʁ IUUQTQZMPOTXFCGSBNFXPSLSFBEUIFEPDTJPFOWSDDPODFQUTIUNM
    
22. DG
    ޚࣾͷ෼ࢄτϨʔγϯάج൫ wؾʹͳΓ·͢ IUUQUFDIMJGFDPPLQBEDPNFOUSZ
    
23. σʔλΛཷΊࠐΉ
24. େ·͔ͳΞʔΩςΫνϟ wूΊΔˠஷΊΔˠՄࢹԽ͢Δ
    wσʔλͷอଘઌ
    1SPNFUIFVT
    wՄࢹԽ
    (SBGBOB
    wσʔλΤʔδΣϯτ
    D"EWJTPS΍NUBJMɺࣗ࡞FYQPSUFS
25. 1SPNFUIFVT
    (SBGBOB &YQPSUFS &YQPSUFS &YQPSUFS $POUBJOFS $POUBJOFS $POUBJOFS $POUBJOFS $POUBJOFS "1*޲͚1SPYZ ूܭϝτϦΫεͷάϧʔϓผʹ
    FYQPSUFSΛ༻ҙ Ϣʔβ༷μογϡϘʔυ
 ʢ։ൃதʣ ӡ༻ΤϯδχΞ
 σΟϨΫλʔͳͲ "MFSUNBOBHFS
26. 1SPNFUIFVTͷ࠾༻ w(PPHMF
    ͷ
    #PSHNPO
    ʹӨڹΛड͚ͨ؂ࢹπʔϧ
    w#PSHNPOͷ঺հ͸43&ຊʹ΋͋Δͦ͏
    wΠϯετʔϧʢجຊతʹ͸ϫϯόΠφϦΛ഑උ͢Ε͹0,ʣ΍ɺ
 ӡ༻ʢجຊతʹΦϯϝϞϦʴϩʔΧϧετϨʔδʣ͕༰қ
    wॊೈͳσʔλߏ଄ͱߴػೳͳΫΤϦ
    w"MFSU࿈ܞ͕༰қ
    w(SBGBOBͳͲՄࢹԽπʔϧ΍֤छFYQPSUFS͕ἧ͍ͬͯΔ
27. 1SPNFUIFVTͱ(SBGBOBͷ࿈ܞ w(SBGBOB
    ͸΋͸΍આ໌ͷඞཁ΋ͳ͍΄Ͳͷఆ൪μογϡϘʔυ
    wެࣜʹ΋υΩϡϝϯτ͕͋Δ
    wIUUQTQSPNFUIFVTJPEPDTWJTVBMJ[BUJPOHSBGBOB
    
    w1SPNFUIFVTͷΫΤϦΛͦͷ·Μ·ॻ͚ͨΓͯ͠ศརײ͕ߴ͍
28. Մ༻ੑͷߟ͑ํ w1SPNFUIFVT͸֤ϊʔυͷFYQPSUFS͔ΒσʔλΛQVMM͍ͯ͘͠ελΠ ϧΛͱ͍ͬͯΔʢDG
    /BHJPT /31&
    
    4BB4ܕ؂ࢹʣ
    wͦͷͨΊ"DUJWF"DUJWFߏ੒ΛऔΕ͹ಉ͡σʔλʹͳΓɺ৑௕ߏ੒ͱ ͍͏͜ͱʹͳΔɻ
    wSFG
    IUUQTHJUIVCDPNQSPNFUIFVTQSPNFUIFVTJTTVFT
29. 4FSWJDF
    %JTDPWFSZ wϊʔυ͔ΒQVMM͢ΔͷͰɺ1SPNFUIFVT͸ϊʔυ͕Ͳ͜ʹ͋Δͷ͔ͱ ͍͏৘ใΛ͍࣋ͬͯͳ͍ͱ͍͚ͳ͍ɻ
    w4FSWJDF
    %JTDPWFSZ
    ͷͨΊͷํ๏ࣗମ͸͍͔ͭ͋͘Δɻ
    wϑΝΠϧϕʔεɺ%/4ϕʔεɺ"840QFO4UBDLͳͲ%ZOBNJD
    *OGSB ޲͚ઃఆɺ͋Δ͍͸.BSBUIPO΍$POTVMͱ΋࿈ܞՄೳ
30. FYQPSUFSͷ֓೦ʹ͍ͭͯ wσʔλΛऔΓ͍ͨϊʔυʹɺFYQPSUFSͱݴ͏ܾ·ͬͨϑΥʔϚοτͰ σʔλΛు͖ग़͢ਓΛৗற͓ͤͯ͘͞ʢ࣮ଶ͸IUUQ
    "1*ʣɻ
    w1SPNFUIFVT͸ɺ4FSWJDF
    %JTDPWFSZΛͯ͠ɺͦͷFYQPSUFSʹΞΫη ε͠ɺఆظతʹσʔλΛಘΔ
    wFYQPSUFS͸ɺطʹ࣮૷͞Ε͍ͯΔ΋ͷ΋ͨ͘͞Μ͋Δ͠ɺ
 ࣗ෼Ͱ࣮૷͢Δ͜ͱ΋Մೳɻ
31. FYQPSUFSͷྫ wOPEF@FYQPSUFS
    
    l&YQPSUFS
    GPS
    NBDIJOF
    NFUSJDTz
    w$16ɺϝϞϦɺ-"ɺωοτϫʔΫͳͲϚγϯ୯ҐͰ৭ʑͱूܭ
    w(SBGBOBͳͲͰ͍͍ײ͡ʹάϥϑԽͯ͠׆༻
    wD"EWJTPS
    
    (PPHMFࣾ಺޲͚ίϯςφ؂ࢹج൫Λϕʔεʹɺ%PDLFSͳ ͲʹରԠͨ͠΋ͷɻͪͳΈʹTZTUFNEͳͲͷ؂ࢹͰ΋ศར
    wͦͷଞɺଟ͘ͷ࣮૷͕͋Δɻී௨͸ࣗ࡞͠ͳ͍͍ͯ͘͸ͣ
 IUUQTHJUIVCDPNQSPNFUIFVTEPDTCMPCNBTUFSDPOUFOUEPDTJOTUSVNFOUJOHFYQPSUFSTNE
32. D"EWJTPSͰ)BDPOJXBͷϝτϦΫεΛऔΔ
33. D"EWJTPSͰ)BDPOJXBͷϝτϦΫεΛऔΔ
34. NUBJM
    IUUQTHJUIVCDPNHPPHMFNUBJM wઌड़ͨ͠/HJOYͳͲͷϩάΛύʔε্ͨ͠ͰɺFYQPSUFSʹ΋ͳͬͯ͘ ΕΔπʔϧ
    w%4-ʹΑΓϧʔϧΛهड़
    wԿ͔طࢹײͷ͋Δྗڧ͞
 DG
    'MVFOUEͷJO@UBJM
    w/HJOYͷྫ
    ˠ IUUQTHJUIVCDPNOEJB[HOHJOYQSPNFUIFVTFYQPSUFSCMPCNBTUFSpMFTQSPHTOHJOYNUBJMɹ
35. ܭଌͷର৅ w$16
    
    ϝϞϦར༻཰ͷ؂ࢹɾܭଌ
    wωοτϫʔΫͷ؂ࢹɾܭଌ
    w*0΍σΟεΫͷ؂ࢹɾܭଌ
    wϦΫΤετͷτϨʔγϯά
    wίϯςφ಺ϛυϧ΢ΣΞͷϩά (PPHMFπʔϧ܈ͷΤίγεςϜ͕ศརɻ ˡ
    D"EWJTFS
    ͰऔΕΔ ˡ
    D"EWJTFS
    Ͱ޻෉͢Ε͹औΕΔ ˡ
    ·ͩཁݕ౼ ˡ
    NUBJM
    ͰָͰ͖Δ
    ˡ
    ώοʢOHY@NSVCZͳͲͰؤுΕͦ͏ʣ
36. FYQPSUFSࣗ࡞ wͱ͸͍͑ɺࣗ࡞΋Ұͭͷख
 
    (PͳͲͳΒ1SPNFUIFVTެࣜͷϥΠϒϥϦ͕࢖͑Δ
 
    طଘ࣮૷ׂ͕ͱ๛෋
 
    ͦ΋ͦ΋ɺຬͨ͢΂͖ϑΥʔϚοτ͕୯७
    wIUUQTQSPNFUIFVTJPEPDTJOTUSVNFOUJOHXSJUJOH@FYQPSUFST
 ΨΠυΛݟͳ͕Βɺ͋ͱطଘ࣮૷ΛோΊͯॻ͘ͱྑͦ͞͏
37. FYQPSUFSͱ1SPNFUIFVTͷ࿈ܞ w4FSWJDF
    %JTDPWFSZͷதͰ΋
    $POTVM
    ͱͷ૬ੑ͸ྑ͍ͱ͍͏ࢲݟ
    wFYQPSUFSଆͰɺ$POTVMͷTFSWJDF
    DIFDLΛ௥ՃˠಛఆͷαʔϏεΛ 1SPNFUIFVT͔ΒσΟεΧόϦɺ͕ඇৗʹ༰қʹͳ͍ͬͯΔ
    wྫ͑͹Ұ୆ʹෳ਺FYQPSUFSͳͲ΋؆୯ͩ͠ɺFYQPSUFSͷ௥Ճ࡟আɺ ࢮ׆؂ࢹ΋$POTVMʹ৐Δ͜ͱͰߟ͑ͳͯ͘ࡁΉ
38. ͜ΜͳTFSWJDFఆٛΛॻ͚͹ wεοͱ+0*/͢Δ
    wศར { "service": { "name": "node_exporter", "tags": [ "development" ], "port": 9100, "checks": [ { "id": "ping", "tcp": "localhost:9100", "interval": "15s", "timeout": "1s" } ] } }
39. ͜ͷষ·ͱΊ wʮσʔλͷஷΊͲ͜Ζʯʹ1SPNFUIFVTΛਅΜதʹਾ͑Δ
    w1SPNFUIFVTͷΤίγεςϜ͸ڧ͍ɻ
    w͍͟ͱͳΕ͹ཉ͍͠σʔλ͸ࣗ෼ͰऔΓʹߦ͘
    w͜ΕΒͷͭͳ͗ʹ
    $POTVM
    ͕ศར
40. ࠓޙ΍Γ͍ͨ͜ͱ
41. ࠓޙ΍Γ͍ͨ͜ͱ wूΊΔͱ͜Ζ·Ͱ͸Կͱ͔ͳΓͦ͏
    wཷΊࠐΜͩσʔλͷ׆༻ʹ͍ͭͯ
    w"MFSUNBOBHFSͷ֓೦ʢࠓ೔͸লུʣ
    wҟৗݕ஌ͷࣗಈԽɺՄࢹԽͳͲ
42. ҟৗݕ஌ͷࣗಈԽɺՄࢹԽ wίϯςφʹؔ͢ΔϝτϦοΫε͸ɺ$.%#ܦ༝Ͱू໿Ͱ͖Δঢ়ଶ
    wϓϩδΣΫτ͝ͱͷσʔλΛूΊͯԿ͔Ͱ͖ͳ͍͔ 1SPKFDUT
    
    EPNBJO $POUBJOFST
    
    $*% DPOUBJOFS
    JE $POUBJOFST
    
    $*% DPOUBJOFS
    JE .FUSJDT\EBUBlBBBz
    ^
    999 .FUSJDT\EBUBlCCCz
    ^
    ::: .FUSJDT\EBUBlDDDz
    ^
    ;;; .FUSJDT\EBUBlEEEz
    ^
    888
43. ूΊͨσʔλΛͲ͏͢Δ͔ w͜Ε͕ɺࠓޙͷ՝୊
    w୯७ʹάϥϑԽͯ͠Έͯ΋ɺͦΕͳΓʹঢ়ଶ͕Θ͔Δؾ΋͢Δ͠ɺٸ ʹෛՙ͕ߴ͍ɺ௨৴Ͱ͖ͳ͘ͳͬͯΔɺͳͲ͸໨ࢹͰ͖ͦ͏Ͱ͸͋Δ
    w·ͣ͸ӡ༻ͯ͠Έ͍ͨͱ͜Ζ
44. ՄࢹԽɾࣗಈ൑ఆͷख๏ wϓϩδΣΫτʹඥ෇͍ͨσʔλ͸ɺجຊతʹଟ࣍ݩͷσʔλͰ͋Δͷ Ͱɺ࣍ݩѹॖΛͯ͠ɺʮ͜Ε͸ਖ਼ৗʯʮ͜Ε͸ো֐ঢ়ଶʯͷΑ͏ʹ෼ ྨ͕Ͱ͖ΔΜͰ͸ͳ͍͔ͱߟ͍͑ͯΔ͕ɺࠓޙͷ՝୊
    wେྔͷίϯςφϓϩδΣΫτΛɺࣗݾ૊৫ԽϚοϓͳͲͰࣗಈͰ
 ෼ྨͯ͠ɺͦΕΛՄࢹԽͰ͖Δͱ໘നͦ͏ʁ
    w͜ͷลΓʹɺʮ਺ͷ໰୊ʯͷղܾࡦ͕ݟ͑Δؾ͕͍ͯ͠Δ
45. 5P
    #F
    $POUJOVFE
46. ·ͩ·ͩख୳Γͷ
    ཁૉ΋ଟ͍Ͱ͕͢
    ΍Γ͕͍͕͋Γ·͢ʂ
48. ϚωʔδυΫϥ΢υΛҰॹʹ࡞Γ·͠ΐ͏ ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU QIPUP
    IUUQTUFDIQFQBCPDPNFOHJOFFSUSBJOJOHWPM