コンテナたちを計測すること ~ ロリポップ！マネージドクラウドの裏側 ~ /wip-how-to-get-container-metrics

dϩϦϙοϓʂϚωʔδυΫϥ΢υͷཪଆd ۙ౻Ӊஐ࿕(.01FQBCP *OD $PPLQBE5FDI,JUDIFO ίϯςφͨͪΛܭଌ͢Δ͜ͱ

ΤϯδχΞ ۙ౻Ӊஐ࿕!VE[VSB ٕज़෦ٕज़ج൫νʔϜ ޷͖ͳϥʔϝϯ͸ΒʔΊΜ޻๪ཾ (.0ϖύϘॴଐ ΍ͬͯΔ෼໺3VCZɺNSVCZɺίϯςφ ΍Γ͍ͨ෼໺γεςϜϓϩάϥϛϯάɺ-JOVY֦ு

؂ࢹɾܭଌʹ͍ͭͯ͸ɺࠓ΋ ࢼߦࡨޡ͕ଓ͍͍ͯ·͕͢ ࠓ೔͸ݕূͷաఔΛڞ༗͠·͢

ϩϦϙοϓʂϚωʔδυΫϥ΢υ IUUQTNDMPMJQPQKQ ΞϧϑΝςετத

IUUQTNDMPMJQPQKQ ΞϧϑΝςετத ˞͝ڵຯͷ͋Δํ͸ձ৔Ͱ੠Λֻ͓͚Լ͍͞

؂ࢹͱ͸

*5Πϯϑϥͷ؂ࢹͱ͸ɺຊॻͰ͸࣍ͷ௨Γఆ ٛ͠·͢ɻ ো֐ൃੜͷ༗ແΛҰఆظؒͰ଺Γͳ֬͘ೝ͢ Δ͜ͱɻ ʰ*5Πϯϑϥ؂ࢹ<࣮ફ>ೖ໳ʱΑΓ

؂ࢹɾϝτϦΫεͷ໨త wӡ༻ͱͯ͠ wαʔϏεμ΢ϯʹ͙͢ʹؾ෇͖͍ͨɻ·ͨɺ෮چΛૉૣ͍ͨ͘͠ wαʔϏεͷػೳͱͯ͠ wίϯςφͷෛՙঢ়گ΍ɺো֐ঢ়ଶʹͳ͍ͬͯΔ͔൱͔ͳͲɻ·ͨͦͷ ཤྺΛͪΌΜͱՄࢹԽͯ͠ఏڙͰ͖Δͱخ͍͠ wˠͦͷͨΊʹɺϝτϦΫεΛܭଌɺ؂ࢹɺෆਖ਼ͳঢ়ଶͷ࣌͸Ξϥʔ τͳͲΛ͋͛ΒΕΔΑ͏ʹ͠ͳ͚Ε͹ͳΒͳ͍ɻ

؂ࢹͷͨΊʹ ·ͣ͸ܭଌ͠ͳ͍ͱ͍͚ͳ͍

ࠓɺࠔ͍ͬͯΔ͜ͱ wίϯςφ͸ී௨ͷ7.ͷ؂ࢹͷΑ͏ʹ͍͔ͳ͍ͱ͜Ζ΋͋Δ wˠϓϩηεʢ܈ʣͳͷͰ-"ͷΑ͏ͳఆ൪ͷ஋͕औΕͳ͔ͬͨΓͳͲ wίϯςφϗεςΟϯάͰ͋Δ͜ͱɺ·ͨϚΠΫϩαʔϏεతͰ͋Δ͜ ͱʹΑΔɺͭͷʮ਺ʯͷ໰୊͕͋Δ

ʮίϯςφ਺ʯͷ໰୊ wϚωʔδυΫϥ΢υ͸ɺίϯςφΛར༻ͨ͠ߴूੵϓϥοτϑΥʔϜ Ͱ͋Δɻ͢ͳΘͪɺίϯςφ͕ͱͯ΋ଟ͍ɻ w͔͠΋ɺ'BTU$POUBJOFSΛ࠾༻͍ͯ͠ΔͳͲɺίϯςφ͸ىಈ͠ଓ͚ ΔΘ͚Ͱ͸ͳ͍ɻ·ͨɺҰͭ࣌ͷυϝΠϯʹඥͮ͘ίϯςφͷ਺΋ม ΘΔɻ wͨ͘͞Μ͋ΔͷͰɺूΊΔ໰୊ͱɺ෼͔Γ΍͘͢ݟͤΔ໰୊͕͋Δ

ʮίϯϙʔωϯτ਺ʯͷ໰୊ wԿ౓͔ର֎ൃද͍ͯ͠Δ௨ΓɺϚωʔδυΫϥ΢υ͸ίϯϙʔωϯτ ͕ଟ͘ɺϚΠΫϩαʔϏεతʹ૊Έ߹Θ͍ͬͯ͞Δ wʮαΠτ͕ݟ͑ͳ͍ʯͷΑ͏ͳ৔߹ʹɺݪҼͱߟ͑ΒΕΔՕॴͷಛఆ ͕೉͘͠ɺΤϥʔτϥοΩϯάʹ՝୊͕͋Δɻ wϘτϧωοΫͷಛఆͳͲ΋ಉ༷

গͣͭ͠ɺ֎ງΛຒΊΔ ͦΜͳઓ͍ͷաఔΛ࿩͠·͢

ͲΜͳ஋͕औΕΔ͔ʁ

ܭଌͷର৅ w$16ϝϞϦར༻཰ͷ؂ࢹɾܭଌ wωοτϫʔΫͷ؂ࢹɾܭଌ w*0΍σΟεΫͷ؂ࢹɾܭଌ wϦΫΤετͷτϨʔγϯά wίϯςφ಺ϛυϧ΢ΣΞͷϩάʢ͜Ε͸ϩάύʔεͰૉ௚ʹूΊΔʣ

ࢀߟ wͦʔ͍ͩ͞ΜͷࢿྉΑΓɺجຊతͳγεςϜʹؔΘΔϝτϦοΫ IUUQTPVEBJIBUFOBCMPHDPNFOUSZ

DHSPVQͷTUBUͷར༻ wDQVDQVBDDUαϒγεςϜ wNFNPSZαϒγεςϜ w ଞɺCMLJP WͷJP wςΩετσʔλͰɺར༻ঢ়گ͕Θ͔Δˠύʔεͯ͠׆༻͢Δ DHSPVQࣗମʹ͍ͭͯ͸ʮDHSPVQVE[VSBʯ·ͨ͸ 
ʮDHSPVQ5FO'PSXBSEʯͰݕࡧ͢Δͱ೔ຊޠͷղઆ͕ɻɻɻ root@compute-1:~# cat /sys/fs/cgroup/cpu/$CONTAINER/cpu.stat nr_periods 140373 nr_throttled 25 throttled_time 1112602844

ωοτϫʔΫʹؔ͢ΔϝτϦΫε wݪଇɺίϯςφ͸ࣗ෼ͷ/FUXPSL/BNFTQBDFʹॴଐ͢Δ w/FUXPSL/BNFTQBDFͲ͏͠ʢࢠ͔Β਌ʣ͸ɺWFUIσόΠεΛܦ༝ ͯ͠௨৴͢Δ͜ͱʹͳΔ wͦͷWFUIσόΠεʹؔ͢Δ஋͸ز͔ͭͱΔखஈ͕͋Δ wJGUPQͷΑ͏ͳίϚϯυɺQSPDOFUEFWϑΝΠϧ wϓϩηε୯ҐͷωοτϫʔΫ৘ใQSPD1*%OFU\EFW UDQ VEQ ^

':*ίϯςφωοτϫʔΫͷτϙϩδ wCSJEHFઃఆͷྫɻ%PDLFSͷσϑΥϧτઃఆ΋͜Εɻ FUI EPDLFS WFUIIPTU WFUIHVFTU WFUIIPTU WFUIHVFTU WFUIIPTU WFUIHVFTU
FH FH ඞཁʹԠ͡/"5ͳͲ /FUXPSL /BNFTQBDF ͻͱͭͷ)PTU CSJEHF ͜ΕΒΛ؂ࢹ͢Δʂ

*0ΞΧ΢ϯςΟϯά wDHSPVQͷCMLJPJP͸ɺσόΠε୯Ґͷܭଌ΍εϩοτϧͳͷͰɺ/'4 ͷΑ͏ʹσόΠε൪߸͕ৼΒΕͳ͍ϑΝΠϧγεςϜʹ͸೉͍͠ wҰํɺϓϩηε୯ҐͷSFBEʢ଒ʣɺXSJUFʢ଒ʣͷݺͼग़͠ճ਺΍ɺ ಡΈࠐΈॻ͖ࠐΈʹ੒ޭͨ͠όΠτ਺ͷྦྷܭ͸ɺΧʔωϧ͕ूܭͯ͠ ͍ͯར༻Ͱ͖Δɻ wQSPD1*%JPϑΝΠϧܦ༝Ͱ؆୯ʹ֬ೝՄೳ wྫ͑͹͜ΕΛίϯςφʹॴଐ͢Δϓϩηε͝ͱʹूܭ͢Δ

ϦΫΤεττϥοΩϯά w͋ΔαʔόΛ௨ա͢Δ͝ͱʹಛఆͷϔομʔʹ࣌ؒΛ௥ه͍ͯ͘͠ wDG9'PSXBSEFE'PSϔομʔ w3BDL౳ͷൃ૝ΛਅࣅͯɺϚʔΧʔΛωετ͍ͯ͘͠ͷ͸Ͳ͏͔ʁ wίϯϙωϯτ͝ͱͷܦա͕࣌ؒΘ͔ΔΑ͏ʹͳΓɺϘτϧωοΫͳͲ ͕Θ͔ΔɻͦΕͦ͜(SBGBOBͳͲͰՄࢹԽͯ͠΋ྑ͍ wΤϥʔτϥοΩϯάʹ΋Ԡ༻Ͱ͖Δʁ IUUQTQZMPOTXFCGSBNFXPSLSFBEUIFEPDTJPFOWSDDPODFQUTIUNM

DGޚࣾͷ෼ࢄτϨʔγϯάج൫ wؾʹͳΓ·͢ IUUQUFDIMJGFDPPLQBEDPNFOUSZ

σʔλΛཷΊࠐΉ

େ·͔ͳΞʔΩςΫνϟ wूΊΔˠஷΊΔˠՄࢹԽ͢Δ wσʔλͷอଘઌ1SPNFUIFVT wՄࢹԽ(SBGBOB wσʔλΤʔδΣϯτD"EWJTPS΍NUBJMɺࣗ࡞FYQPSUFS

1SPNFUIFVT (SBGBOB &YQPSUFS &YQPSUFS &YQPSUFS $POUBJOFS $POUBJOFS $POUBJOFS $POUBJOFS $POUBJOFS
"1*޲͚1SPYZ ूܭϝτϦΫεͷάϧʔϓผʹ FYQPSUFSΛ༻ҙ Ϣʔβ༷μογϡϘʔυ  ʢ։ൃதʣ ӡ༻ΤϯδχΞ  σΟϨΫλʔͳͲ "MFSUNBOBHFS

1SPNFUIFVTͷ࠾༻ w(PPHMFͷ#PSHNPOʹӨڹΛड͚ͨ؂ࢹπʔϧ w#PSHNPOͷ঺հ͸43&ຊʹ΋͋Δͦ͏ wΠϯετʔϧʢجຊతʹ͸ϫϯόΠφϦΛ഑උ͢Ε͹0,ʣ΍ɺ  ӡ༻ʢجຊతʹΦϯϝϞϦʴϩʔΧϧετϨʔδʣ͕༰қ wॊೈͳσʔλߏ଄ͱߴػೳͳΫΤϦ w"MFSU࿈ܞ͕༰қ w(SBGBOBͳͲՄࢹԽπʔϧ΍֤छFYQPSUFS͕ἧ͍ͬͯΔ

1SPNFUIFVTͱ(SBGBOBͷ࿈ܞ w(SBGBOB͸΋͸΍આ໌ͷඞཁ΋ͳ͍΄Ͳͷఆ൪μογϡϘʔυ wެࣜʹ΋υΩϡϝϯτ͕͋Δ wIUUQTQSPNFUIFVTJPEPDTWJTVBMJ[BUJPOHSBGBOB w1SPNFUIFVTͷΫΤϦΛͦͷ·Μ·ॻ͚ͨΓͯ͠ศརײ͕ߴ͍

Մ༻ੑͷߟ͑ํ w1SPNFUIFVT͸֤ϊʔυͷFYQPSUFS͔ΒσʔλΛQVMM͍ͯ͘͠ελΠ ϧΛͱ͍ͬͯΔʢDG/BHJPT /31&4BB4ܕ؂ࢹʣ wͦͷͨΊ"DUJWF"DUJWFߏ੒ΛऔΕ͹ಉ͡σʔλʹͳΓɺ৑௕ߏ੒ͱ ͍͏͜ͱʹͳΔɻ wSFGIUUQTHJUIVCDPNQSPNFUIFVTQSPNFUIFVTJTTVFT

4FSWJDF%JTDPWFSZ wϊʔυ͔ΒQVMM͢ΔͷͰɺ1SPNFUIFVT͸ϊʔυ͕Ͳ͜ʹ͋Δͷ͔ͱ ͍͏৘ใΛ͍࣋ͬͯͳ͍ͱ͍͚ͳ͍ɻ w4FSWJDF%JTDPWFSZͷͨΊͷํ๏ࣗମ͸͍͔ͭ͋͘Δɻ wϑΝΠϧϕʔεɺ%/4ϕʔεɺ"840QFO4UBDLͳͲ%ZOBNJD*OGSB ޲͚ઃఆɺ͋Δ͍͸.BSBUIPO΍$POTVMͱ΋࿈ܞՄೳ

FYQPSUFSͷ֓೦ʹ͍ͭͯ wσʔλΛऔΓ͍ͨϊʔυʹɺFYQPSUFSͱݴ͏ܾ·ͬͨϑΥʔϚοτͰ σʔλΛు͖ग़͢ਓΛৗற͓ͤͯ͘͞ʢ࣮ଶ͸IUUQ"1*ʣɻ w1SPNFUIFVT͸ɺ4FSWJDF%JTDPWFSZΛͯ͠ɺͦͷFYQPSUFSʹΞΫη ε͠ɺఆظతʹσʔλΛಘΔ wFYQPSUFS͸ɺطʹ࣮૷͞Ε͍ͯΔ΋ͷ΋ͨ͘͞Μ͋Δ͠ɺ  ࣗ෼Ͱ࣮૷͢Δ͜ͱ΋Մೳɻ

FYQPSUFSͷྫ wOPEF@FYQPSUFSl&YQPSUFSGPSNBDIJOFNFUSJDTz w$16ɺϝϞϦɺ-"ɺωοτϫʔΫͳͲϚγϯ୯ҐͰ৭ʑͱूܭ w(SBGBOBͳͲͰ͍͍ײ͡ʹάϥϑԽͯ͠׆༻ wD"EWJTPS(PPHMFࣾ಺޲͚ίϯςφ؂ࢹج൫Λϕʔεʹɺ%PDLFSͳ ͲʹରԠͨ͠΋ͷɻͪͳΈʹTZTUFNEͳͲͷ؂ࢹͰ΋ศར wͦͷଞɺଟ͘ͷ࣮૷͕͋Δɻී௨͸ࣗ࡞͠ͳ͍͍ͯ͘͸ͣ  IUUQTHJUIVCDPNQSPNFUIFVTEPDTCMPCNBTUFSDPOUFOUEPDTJOTUSVNFOUJOHFYQPSUFSTNE

D"EWJTPSͰ)BDPOJXBͷϝτϦΫεΛऔΔ

NUBJMIUUQTHJUIVCDPNHPPHMFNUBJM wઌड़ͨ͠/HJOYͳͲͷϩάΛύʔε্ͨ͠ͰɺFYQPSUFSʹ΋ͳͬͯ͘ ΕΔπʔϧ w%4-ʹΑΓϧʔϧΛهड़ wԿ͔طࢹײͷ͋Δྗڧ͞  DG'MVFOUEͷJO@UBJM w/HJOYͷྫˠ IUUQTHJUIVCDPNOEJB[HOHJOYQSPNFUIFVTFYQPSUFSCMPCNBTUFSpMFTQSPHTOHJOYNUBJMɹ

ܭଌͷର৅ w$16ϝϞϦར༻཰ͷ؂ࢹɾܭଌ wωοτϫʔΫͷ؂ࢹɾܭଌ w*0΍σΟεΫͷ؂ࢹɾܭଌ wϦΫΤετͷτϨʔγϯά wίϯςφ಺ϛυϧ΢ΣΞͷϩά (PPHMFπʔϧ܈ͷΤίγεςϜ͕ศརɻ ˡD"EWJTFSͰऔΕΔ ˡD"EWJTFSͰ޻෉͢Ε͹औΕΔ ˡ·ͩཁݕ౼
ˡNUBJMͰָͰ͖Δ ˡώοʢOHY@NSVCZͳͲͰؤுΕͦ͏ʣ

FYQPSUFSࣗ࡞ wͱ͸͍͑ɺࣗ࡞΋Ұͭͷख  (PͳͲͳΒ1SPNFUIFVTެࣜͷϥΠϒϥϦ͕࢖͑Δ  طଘ࣮૷ׂ͕ͱ๛෋  ͦ΋ͦ΋ɺຬͨ͢΂͖ϑΥʔϚοτ͕୯७ wIUUQTQSPNFUIFVTJPEPDTJOTUSVNFOUJOHXSJUJOH@FYQPSUFST  ΨΠυΛݟͳ͕Βɺ͋ͱطଘ࣮૷ΛோΊͯॻ͘ͱྑͦ͞͏

FYQPSUFSͱ1SPNFUIFVTͷ࿈ܞ w4FSWJDF%JTDPWFSZͷதͰ΋$POTVMͱͷ૬ੑ͸ྑ͍ͱ͍͏ࢲݟ wFYQPSUFSଆͰɺ$POTVMͷTFSWJDFDIFDLΛ௥ՃˠಛఆͷαʔϏεΛ 1SPNFUIFVT͔ΒσΟεΧόϦɺ͕ඇৗʹ༰қʹͳ͍ͬͯΔ wྫ͑͹Ұ୆ʹෳ਺FYQPSUFSͳͲ΋؆୯ͩ͠ɺFYQPSUFSͷ௥Ճ࡟আɺ ࢮ׆؂ࢹ΋$POTVMʹ৐Δ͜ͱͰߟ͑ͳͯ͘ࡁΉ

͜ΜͳTFSWJDFఆٛΛॻ͚͹ wεοͱ+0*/͢Δ wศར { "service": { "name": "node_exporter", "tags": [
"development" ], "port": 9100, "checks": [ { "id": "ping", "tcp": "localhost:9100", "interval": "15s", "timeout": "1s" } ] } }

͜ͷষ·ͱΊ wʮσʔλͷஷΊͲ͜Ζʯʹ1SPNFUIFVTΛਅΜதʹਾ͑Δ w1SPNFUIFVTͷΤίγεςϜ͸ڧ͍ɻ w͍͟ͱͳΕ͹ཉ͍͠σʔλ͸ࣗ෼ͰऔΓʹߦ͘ w͜ΕΒͷͭͳ͗ʹ$POTVM͕ศར

ࠓޙ΍Γ͍ͨ͜ͱ

ࠓޙ΍Γ͍ͨ͜ͱ wूΊΔͱ͜Ζ·Ͱ͸Կͱ͔ͳΓͦ͏ wཷΊࠐΜͩσʔλͷ׆༻ʹ͍ͭͯ w"MFSUNBOBHFSͷ֓೦ʢࠓ೔͸লུʣ wҟৗݕ஌ͷࣗಈԽɺՄࢹԽͳͲ

ҟৗݕ஌ͷࣗಈԽɺՄࢹԽ wίϯςφʹؔ͢ΔϝτϦοΫε͸ɺ$.%#ܦ༝Ͱू໿Ͱ͖Δঢ়ଶ wϓϩδΣΫτ͝ͱͷσʔλΛूΊͯԿ͔Ͱ͖ͳ͍͔ 1SPKFDUT EPNBJO $POUBJOFST $*% DPOUBJOFSJE $POUBJOFST $*%
DPOUBJOFSJE .FUSJDT\EBUBlBBBz ^999 .FUSJDT\EBUBlCCCz ^::: .FUSJDT\EBUBlDDDz ^;;; .FUSJDT\EBUBlEEEz ^888

ूΊͨσʔλΛͲ͏͢Δ͔ w͜Ε͕ɺࠓޙͷ՝୊ w୯७ʹάϥϑԽͯ͠Έͯ΋ɺͦΕͳΓʹঢ়ଶ͕Θ͔Δؾ΋͢Δ͠ɺٸ ʹෛՙ͕ߴ͍ɺ௨৴Ͱ͖ͳ͘ͳͬͯΔɺͳͲ͸໨ࢹͰ͖ͦ͏Ͱ͸͋Δ w·ͣ͸ӡ༻ͯ͠Έ͍ͨͱ͜Ζ

ՄࢹԽɾࣗಈ൑ఆͷख๏ wϓϩδΣΫτʹඥ෇͍ͨσʔλ͸ɺجຊతʹଟ࣍ݩͷσʔλͰ͋Δͷ Ͱɺ࣍ݩѹॖΛͯ͠ɺʮ͜Ε͸ਖ਼ৗʯʮ͜Ε͸ো֐ঢ়ଶʯͷΑ͏ʹ෼ ྨ͕Ͱ͖ΔΜͰ͸ͳ͍͔ͱߟ͍͑ͯΔ͕ɺࠓޙͷ՝୊ wେྔͷίϯςφϓϩδΣΫτΛɺࣗݾ૊৫ԽϚοϓͳͲͰࣗಈͰ  ෼ྨͯ͠ɺͦΕΛՄࢹԽͰ͖Δͱ໘നͦ͏ʁ w͜ͷลΓʹɺʮ਺ͷ໰୊ʯͷղܾࡦ͕ݟ͑Δؾ͕͍ͯ͠Δ

5P#F$POUJOVFE

·ͩ·ͩख୳Γͷ ཁૉ΋ଟ͍Ͱ͕͢ ΍Γ͕͍͕͋Γ·͢ʂ

ϚωʔδυΫϥ΢υΛҰॹʹ࡞Γ·͠ΐ͏ ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU QIPUPIUUQTUFDIQFQBCPDPNFOHJOFFSUSBJOJOHWPM

コンテナたちを計測すること ~ ロリポップ！マネージドクラウドの裏側 ~ /wip-how-to-get-container-metrics

コンテナたちを計測すること ~ ロリポップ！マネージドクラウドの裏側 ~ /wip-how-to-get-container-metrics

More Decks by KONDO Uchio

Other Decks in Technology

Featured

Transcript