コンテナたちを計測すること ~ ロリポップ！マネージドクラウドの裏側 ~ /wip-how-to-get-container-metrics

by KONDO Uchio

Slide 1

Slide 1 text

dϩϦϙοϓʂϚωʔδυΫϥ΢υͷཪଆd ۙ౻Ӊஐ࿕(.01FQBCP *OD $PPLQBE5FDI,JUDIFO ίϯςφͨͪΛܭଌ͢Δ͜ͱ

Slide 2

Slide 2 text

ΤϯδχΞ ۙ౻Ӊஐ࿕!VE[VSB ٕज़෦ٕज़ج൫νʔϜ ޷͖ͳϥʔϝϯ͸ΒʔΊΜ޻๪ཾ (.0ϖύϘॴଐ ΍ͬͯΔ෼໺3VCZɺNSVCZɺίϯςφ ΍Γ͍ͨ෼໺γεςϜϓϩάϥϛϯάɺ-JOVY֦ு

Slide 3

Slide 3 text

؂ࢹɾܭଌʹ͍ͭͯ͸ɺࠓ΋ ࢼߦࡨޡ͕ଓ͍͍ͯ·͕͢ ࠓ೔͸ݕূͷաఔΛڞ༗͠·͢

Slide 4

Slide 4 text

ϩϦϙοϓʂϚωʔδυΫϥ΢υ IUUQTNDMPMJQPQKQ ΞϧϑΝςετத

Slide 5

Slide 5 text

IUUQTNDMPMJQPQKQ ΞϧϑΝςετத ˞͝ڵຯͷ͋Δํ͸ձ৔Ͱ੠Λֻ͓͚Լ͍͞

Slide 6

Slide 6 text

؂ࢹͱ͸

Slide 7

Slide 7 text

*5Πϯϑϥͷ؂ࢹͱ͸ɺຊॻͰ͸࣍ͷ௨Γఆ ٛ͠·͢ɻ ো֐ൃੜͷ༗ແΛҰఆظؒͰ଺Γͳ֬͘ೝ͢ Δ͜ͱɻ ʰ*5Πϯϑϥ؂ࢹ<࣮ફ>ೖ໳ʱΑΓ

Slide 8

Slide 8 text

؂ࢹɾϝτϦΫεͷ໨త wӡ༻ͱͯ͠ wαʔϏεμ΢ϯʹ͙͢ʹؾ෇͖͍ͨɻ·ͨɺ෮چΛૉૣ͍ͨ͘͠ wαʔϏεͷػೳͱͯ͠ wίϯςφͷෛՙঢ়گ΍ɺো֐ঢ়ଶʹͳ͍ͬͯΔ͔൱͔ͳͲɻ·ͨͦͷ ཤྺΛͪΌΜͱՄࢹԽͯ͠ఏڙͰ͖Δͱخ͍͠ wˠͦͷͨΊʹɺϝτϦΫεΛܭଌɺ؂ࢹɺෆਖ਼ͳঢ়ଶͷ࣌͸Ξϥʔ τͳͲΛ͋͛ΒΕΔΑ͏ʹ͠ͳ͚Ε͹ͳΒͳ͍ɻ

Slide 9

Slide 9 text

؂ࢹͷͨΊʹ ·ͣ͸ܭଌ͠ͳ͍ͱ͍͚ͳ͍

Slide 10

Slide 10 text

ࠓɺࠔ͍ͬͯΔ͜ͱ wίϯςφ͸ී௨ͷ7.ͷ؂ࢹͷΑ͏ʹ͍͔ͳ͍ͱ͜Ζ΋͋Δ wˠϓϩηεʢ܈ʣͳͷͰ-"ͷΑ͏ͳఆ൪ͷ஋͕औΕͳ͔ͬͨΓͳͲ wίϯςφϗεςΟϯάͰ͋Δ͜ͱɺ·ͨϚΠΫϩαʔϏεతͰ͋Δ͜ ͱʹΑΔɺͭͷʮ਺ʯͷ໰୊͕͋Δ

Slide 11

Slide 11 text

ʮίϯςφ਺ʯͷ໰୊ wϚωʔδυΫϥ΢υ͸ɺίϯςφΛར༻ͨ͠ߴूੵϓϥοτϑΥʔϜ Ͱ͋Δɻ͢ͳΘͪɺίϯςφ͕ͱͯ΋ଟ͍ɻ w͔͠΋ɺ'BTU$POUBJOFSΛ࠾༻͍ͯ͠ΔͳͲɺίϯςφ͸ىಈ͠ଓ͚ ΔΘ͚Ͱ͸ͳ͍ɻ·ͨɺҰͭ࣌ͷυϝΠϯʹඥͮ͘ίϯςφͷ਺΋ม ΘΔɻ wͨ͘͞Μ͋ΔͷͰɺूΊΔ໰୊ͱɺ෼͔Γ΍͘͢ݟͤΔ໰୊͕͋Δ

Slide 12

Slide 12 text

ʮίϯϙʔωϯτ਺ʯͷ໰୊ wԿ౓͔ର֎ൃද͍ͯ͠Δ௨ΓɺϚωʔδυΫϥ΢υ͸ίϯϙʔωϯτ ͕ଟ͘ɺϚΠΫϩαʔϏεతʹ૊Έ߹Θ͍ͬͯ͞Δ wʮαΠτ͕ݟ͑ͳ͍ʯͷΑ͏ͳ৔߹ʹɺݪҼͱߟ͑ΒΕΔՕॴͷಛఆ ͕೉͘͠ɺΤϥʔτϥοΩϯάʹ՝୊͕͋Δɻ wϘτϧωοΫͷಛఆͳͲ΋ಉ༷

Slide 13

Slide 13 text

গͣͭ͠ɺ֎ງΛຒΊΔ ͦΜͳઓ͍ͷաఔΛ࿩͠·͢

Slide 14

Slide 14 text

ͲΜͳ஋͕औΕΔ͔ʁ

Slide 15

Slide 15 text

ܭଌͷର৅ w$16ϝϞϦར༻཰ͷ؂ࢹɾܭଌ wωοτϫʔΫͷ؂ࢹɾܭଌ w*0΍σΟεΫͷ؂ࢹɾܭଌ wϦΫΤετͷτϨʔγϯά wίϯςφ಺ϛυϧ΢ΣΞͷϩάʢ͜Ε͸ϩάύʔεͰૉ௚ʹूΊΔʣ

Slide 16

Slide 16 text

ࢀߟ wͦʔ͍ͩ͞ΜͷࢿྉΑΓɺجຊతͳγεςϜʹؔΘΔϝτϦοΫ IUUQTPVEBJIBUFOBCMPHDPNFOUSZ

Slide 17

Slide 17 text

DHSPVQͷTUBUͷར༻ wDQVDQVBDDUαϒγεςϜ wNFNPSZαϒγεςϜ w ଞɺCMLJP WͷJP wςΩετσʔλͰɺར༻ঢ়گ͕Θ͔Δˠύʔεͯ͠׆༻͢Δ DHSPVQࣗମʹ͍ͭͯ͸ʮDHSPVQVE[VSBʯ·ͨ͸  ʮDHSPVQ5FO'PSXBSEʯͰݕࡧ͢Δͱ೔ຊޠͷղઆ͕ɻɻɻ root@compute-1:~# cat /sys/fs/cgroup/cpu/$CONTAINER/cpu.stat nr_periods 140373 nr_throttled 25 throttled_time 1112602844

Slide 18

Slide 18 text

ωοτϫʔΫʹؔ͢ΔϝτϦΫε wݪଇɺίϯςφ͸ࣗ෼ͷ/FUXPSL/BNFTQBDFʹॴଐ͢Δ w/FUXPSL/BNFTQBDFͲ͏͠ʢࢠ͔Β਌ʣ͸ɺWFUIσόΠεΛܦ༝ ͯ͠௨৴͢Δ͜ͱʹͳΔ wͦͷWFUIσόΠεʹؔ͢Δ஋͸ز͔ͭͱΔखஈ͕͋Δ wJGUPQͷΑ͏ͳίϚϯυɺQSPDOFUEFWϑΝΠϧ wϓϩηε୯ҐͷωοτϫʔΫ৘ใQSPD1*%OFU\EFW UDQ VEQ ^

Slide 19

Slide 19 text

':*ίϯςφωοτϫʔΫͷτϙϩδ wCSJEHFઃఆͷྫɻ%PDLFSͷσϑΥϧτઃఆ΋͜Εɻ FUI EPDLFS WFUIIPTU WFUIHVFTU WFUIIPTU WFUIHVFTU WFUIIPTU WFUIHVFTU FH FH ඞཁʹԠ͡/"5ͳͲ /FUXPSL /BNFTQBDF ͻͱͭͷ)PTU CSJEHF ͜ΕΒΛ؂ࢹ͢Δʂ

Slide 20

Slide 20 text

*0ΞΧ΢ϯςΟϯά wDHSPVQͷCMLJPJP͸ɺσόΠε୯Ґͷܭଌ΍εϩοτϧͳͷͰɺ/'4 ͷΑ͏ʹσόΠε൪߸͕ৼΒΕͳ͍ϑΝΠϧγεςϜʹ͸೉͍͠ wҰํɺϓϩηε୯ҐͷSFBEʢ଒ʣɺXSJUFʢ଒ʣͷݺͼग़͠ճ਺΍ɺ ಡΈࠐΈॻ͖ࠐΈʹ੒ޭͨ͠όΠτ਺ͷྦྷܭ͸ɺΧʔωϧ͕ूܭͯ͠ ͍ͯར༻Ͱ͖Δɻ wQSPD1*%JPϑΝΠϧܦ༝Ͱ؆୯ʹ֬ೝՄೳ wྫ͑͹͜ΕΛίϯςφʹॴଐ͢Δϓϩηε͝ͱʹूܭ͢Δ

Slide 21

Slide 21 text

ϦΫΤεττϥοΩϯά w͋ΔαʔόΛ௨ա͢Δ͝ͱʹಛఆͷϔομʔʹ࣌ؒΛ௥ه͍ͯ͘͠ wDG9'PSXBSEFE'PSϔομʔ w3BDL౳ͷൃ૝ΛਅࣅͯɺϚʔΧʔΛωετ͍ͯ͘͠ͷ͸Ͳ͏͔ʁ wίϯϙωϯτ͝ͱͷܦա͕࣌ؒΘ͔ΔΑ͏ʹͳΓɺϘτϧωοΫͳͲ ͕Θ͔ΔɻͦΕͦ͜(SBGBOBͳͲͰՄࢹԽͯ͠΋ྑ͍ wΤϥʔτϥοΩϯάʹ΋Ԡ༻Ͱ͖Δʁ IUUQTQZMPOTXFCGSBNFXPSLSFBEUIFEPDTJPFOWSDDPODFQUTIUNM

Slide 22

Slide 22 text

DGޚࣾͷ෼ࢄτϨʔγϯάج൫ wؾʹͳΓ·͢ IUUQUFDIMJGFDPPLQBEDPNFOUSZ

Slide 23

Slide 23 text

σʔλΛཷΊࠐΉ

Slide 24

Slide 24 text

େ·͔ͳΞʔΩςΫνϟ wूΊΔˠஷΊΔˠՄࢹԽ͢Δ wσʔλͷอଘઌ1SPNFUIFVT wՄࢹԽ(SBGBOB wσʔλΤʔδΣϯτD"EWJTPS΍NUBJMɺࣗ࡞FYQPSUFS

Slide 25

Slide 25 text

1SPNFUIFVT (SBGBOB &YQPSUFS &YQPSUFS &YQPSUFS $POUBJOFS $POUBJOFS $POUBJOFS $POUBJOFS $POUBJOFS "1*޲͚1SPYZ ूܭϝτϦΫεͷάϧʔϓผʹ FYQPSUFSΛ༻ҙ Ϣʔβ༷μογϡϘʔυ  ʢ։ൃதʣ ӡ༻ΤϯδχΞ  σΟϨΫλʔͳͲ "MFSUNBOBHFS

Slide 26

Slide 26 text

1SPNFUIFVTͷ࠾༻ w(PPHMFͷ#PSHNPOʹӨڹΛड͚ͨ؂ࢹπʔϧ w#PSHNPOͷ঺հ͸43&ຊʹ΋͋Δͦ͏ wΠϯετʔϧʢجຊతʹ͸ϫϯόΠφϦΛ഑උ͢Ε͹0,ʣ΍ɺ  ӡ༻ʢجຊతʹΦϯϝϞϦʴϩʔΧϧετϨʔδʣ͕༰қ wॊೈͳσʔλߏ଄ͱߴػೳͳΫΤϦ w"MFSU࿈ܞ͕༰қ w(SBGBOBͳͲՄࢹԽπʔϧ΍֤छFYQPSUFS͕ἧ͍ͬͯΔ

Slide 27

Slide 27 text

1SPNFUIFVTͱ(SBGBOBͷ࿈ܞ w(SBGBOB͸΋͸΍આ໌ͷඞཁ΋ͳ͍΄Ͳͷఆ൪μογϡϘʔυ wެࣜʹ΋υΩϡϝϯτ͕͋Δ wIUUQTQSPNFUIFVTJPEPDTWJTVBMJ[BUJPOHSBGBOB w1SPNFUIFVTͷΫΤϦΛͦͷ·Μ·ॻ͚ͨΓͯ͠ศརײ͕ߴ͍

Slide 28

Slide 28 text

Մ༻ੑͷߟ͑ํ w1SPNFUIFVT͸֤ϊʔυͷFYQPSUFS͔ΒσʔλΛQVMM͍ͯ͘͠ελΠ ϧΛͱ͍ͬͯΔʢDG/BHJPT/31&4BB4ܕ؂ࢹʣ wͦͷͨΊ"DUJWF"DUJWFߏ੒ΛऔΕ͹ಉ͡σʔλʹͳΓɺ৑௕ߏ੒ͱ ͍͏͜ͱʹͳΔɻ wSFGIUUQTHJUIVCDPNQSPNFUIFVTQSPNFUIFVTJTTVFT

Slide 29

Slide 29 text

4FSWJDF%JTDPWFSZ wϊʔυ͔ΒQVMM͢ΔͷͰɺ1SPNFUIFVT͸ϊʔυ͕Ͳ͜ʹ͋Δͷ͔ͱ ͍͏৘ใΛ͍࣋ͬͯͳ͍ͱ͍͚ͳ͍ɻ w4FSWJDF%JTDPWFSZͷͨΊͷํ๏ࣗମ͸͍͔ͭ͋͘Δɻ wϑΝΠϧϕʔεɺ%/4ϕʔεɺ"840QFO4UBDLͳͲ%ZOBNJD*OGSB ޲͚ઃఆɺ͋Δ͍͸.BSBUIPO΍$POTVMͱ΋࿈ܞՄೳ

Slide 30

Slide 30 text

FYQPSUFSͷ֓೦ʹ͍ͭͯ wσʔλΛऔΓ͍ͨϊʔυʹɺFYQPSUFSͱݴ͏ܾ·ͬͨϑΥʔϚοτͰ σʔλΛు͖ग़͢ਓΛৗற͓ͤͯ͘͞ʢ࣮ଶ͸IUUQ"1*ʣɻ w1SPNFUIFVT͸ɺ4FSWJDF%JTDPWFSZΛͯ͠ɺͦͷFYQPSUFSʹΞΫη ε͠ɺఆظతʹσʔλΛಘΔ wFYQPSUFS͸ɺطʹ࣮૷͞Ε͍ͯΔ΋ͷ΋ͨ͘͞Μ͋Δ͠ɺ  ࣗ෼Ͱ࣮૷͢Δ͜ͱ΋Մೳɻ

Slide 31

Slide 31 text

FYQPSUFSͷྫ wOPEF@FYQPSUFSl&YQPSUFSGPSNBDIJOFNFUSJDTz w$16ɺϝϞϦɺ-"ɺωοτϫʔΫͳͲϚγϯ୯ҐͰ৭ʑͱूܭ w(SBGBOBͳͲͰ͍͍ײ͡ʹάϥϑԽͯ͠׆༻ wD"EWJTPS(PPHMFࣾ಺޲͚ίϯςφ؂ࢹج൫Λϕʔεʹɺ%PDLFSͳ ͲʹରԠͨ͠΋ͷɻͪͳΈʹTZTUFNEͳͲͷ؂ࢹͰ΋ศར wͦͷଞɺଟ͘ͷ࣮૷͕͋Δɻී௨͸ࣗ࡞͠ͳ͍͍ͯ͘͸ͣ  IUUQTHJUIVCDPNQSPNFUIFVTEPDTCMPCNBTUFSDPOUFOUEPDTJOTUSVNFOUJOHFYQPSUFSTNE

Slide 32

Slide 32 text

D"EWJTPSͰ)BDPOJXBͷϝτϦΫεΛऔΔ

Slide 33

Slide 33 text

D"EWJTPSͰ)BDPOJXBͷϝτϦΫεΛऔΔ

Slide 34

Slide 34 text

NUBJMIUUQTHJUIVCDPNHPPHMFNUBJM wઌड़ͨ͠/HJOYͳͲͷϩάΛύʔε্ͨ͠ͰɺFYQPSUFSʹ΋ͳͬͯ͘ ΕΔπʔϧ w%4-ʹΑΓϧʔϧΛهड़ wԿ͔طࢹײͷ͋Δྗڧ͞  DG'MVFOUEͷJO@UBJM w/HJOYͷྫˠ IUUQTHJUIVCDPNOEJB[HOHJOYQSPNFUIFVTFYQPSUFSCMPCNBTUFSpMFTQSPHTOHJOYNUBJMɹ

Slide 35

Slide 35 text

ܭଌͷର৅ w$16ϝϞϦར༻཰ͷ؂ࢹɾܭଌ wωοτϫʔΫͷ؂ࢹɾܭଌ w*0΍σΟεΫͷ؂ࢹɾܭଌ wϦΫΤετͷτϨʔγϯά wίϯςφ಺ϛυϧ΢ΣΞͷϩά (PPHMFπʔϧ܈ͷΤίγεςϜ͕ศརɻ ˡD"EWJTFSͰऔΕΔ ˡD"EWJTFSͰ޻෉͢Ε͹औΕΔ ˡ·ͩཁݕ౼ ˡNUBJMͰָͰ͖Δ ˡώοʢOHY@NSVCZͳͲͰؤுΕͦ͏ʣ

Slide 36

Slide 36 text

FYQPSUFSࣗ࡞ wͱ͸͍͑ɺࣗ࡞΋Ұͭͷख  (PͳͲͳΒ1SPNFUIFVTެࣜͷϥΠϒϥϦ͕࢖͑Δ  طଘ࣮૷ׂ͕ͱ๛෋  ͦ΋ͦ΋ɺຬͨ͢΂͖ϑΥʔϚοτ͕୯७ wIUUQTQSPNFUIFVTJPEPDTJOTUSVNFOUJOHXSJUJOH@FYQPSUFST  ΨΠυΛݟͳ͕Βɺ͋ͱطଘ࣮૷ΛோΊͯॻ͘ͱྑͦ͞͏

Slide 37

Slide 37 text

FYQPSUFSͱ1SPNFUIFVTͷ࿈ܞ w4FSWJDF%JTDPWFSZͷதͰ΋$POTVMͱͷ૬ੑ͸ྑ͍ͱ͍͏ࢲݟ wFYQPSUFSଆͰɺ$POTVMͷTFSWJDFDIFDLΛ௥ՃˠಛఆͷαʔϏεΛ 1SPNFUIFVT͔ΒσΟεΧόϦɺ͕ඇৗʹ༰қʹͳ͍ͬͯΔ wྫ͑͹Ұ୆ʹෳ਺FYQPSUFSͳͲ΋؆୯ͩ͠ɺFYQPSUFSͷ௥Ճ࡟আɺ ࢮ׆؂ࢹ΋$POTVMʹ৐Δ͜ͱͰߟ͑ͳͯ͘ࡁΉ

Slide 38

Slide 38 text

͜ΜͳTFSWJDFఆٛΛॻ͚͹ wεοͱ+0*/͢Δ wศར { "service": { "name": "node_exporter", "tags": [ "development" ], "port": 9100, "checks": [ { "id": "ping", "tcp": "localhost:9100", "interval": "15s", "timeout": "1s" } ] } }

Slide 39

Slide 39 text

͜ͷষ·ͱΊ wʮσʔλͷஷΊͲ͜Ζʯʹ1SPNFUIFVTΛਅΜதʹਾ͑Δ w1SPNFUIFVTͷΤίγεςϜ͸ڧ͍ɻ w͍͟ͱͳΕ͹ཉ͍͠σʔλ͸ࣗ෼ͰऔΓʹߦ͘ w͜ΕΒͷͭͳ͗ʹ$POTVM͕ศར

Slide 40

Slide 40 text

ࠓޙ΍Γ͍ͨ͜ͱ

Slide 41

Slide 41 text

ࠓޙ΍Γ͍ͨ͜ͱ wूΊΔͱ͜Ζ·Ͱ͸Կͱ͔ͳΓͦ͏ wཷΊࠐΜͩσʔλͷ׆༻ʹ͍ͭͯ w"MFSUNBOBHFSͷ֓೦ʢࠓ೔͸লུʣ wҟৗݕ஌ͷࣗಈԽɺՄࢹԽͳͲ

Slide 42

Slide 42 text

ҟৗݕ஌ͷࣗಈԽɺՄࢹԽ wίϯςφʹؔ͢ΔϝτϦοΫε͸ɺ$.%#ܦ༝Ͱू໿Ͱ͖Δঢ়ଶ wϓϩδΣΫτ͝ͱͷσʔλΛूΊͯԿ͔Ͱ͖ͳ͍͔ 1SPKFDUT EPNBJO $POUBJOFST $*% DPOUBJOFSJE $POUBJOFST $*% DPOUBJOFSJE .FUSJDT\EBUBlBBBz ^999 .FUSJDT\EBUBlCCCz ^::: .FUSJDT\EBUBlDDDz ^;;; .FUSJDT\EBUBlEEEz ^888

Slide 43

Slide 43 text

ूΊͨσʔλΛͲ͏͢Δ͔ w͜Ε͕ɺࠓޙͷ՝୊ w୯७ʹάϥϑԽͯ͠Έͯ΋ɺͦΕͳΓʹঢ়ଶ͕Θ͔Δؾ΋͢Δ͠ɺٸ ʹෛՙ͕ߴ͍ɺ௨৴Ͱ͖ͳ͘ͳͬͯΔɺͳͲ͸໨ࢹͰ͖ͦ͏Ͱ͸͋Δ w·ͣ͸ӡ༻ͯ͠Έ͍ͨͱ͜Ζ

Slide 44

Slide 44 text

ՄࢹԽɾࣗಈ൑ఆͷख๏ wϓϩδΣΫτʹඥ෇͍ͨσʔλ͸ɺجຊతʹଟ࣍ݩͷσʔλͰ͋Δͷ Ͱɺ࣍ݩѹॖΛͯ͠ɺʮ͜Ε͸ਖ਼ৗʯʮ͜Ε͸ো֐ঢ়ଶʯͷΑ͏ʹ෼ ྨ͕Ͱ͖ΔΜͰ͸ͳ͍͔ͱߟ͍͑ͯΔ͕ɺࠓޙͷ՝୊ wେྔͷίϯςφϓϩδΣΫτΛɺࣗݾ૊৫ԽϚοϓͳͲͰࣗಈͰ  ෼ྨͯ͠ɺͦΕΛՄࢹԽͰ͖Δͱ໘നͦ͏ʁ w͜ͷลΓʹɺʮ਺ͷ໰୊ʯͷղܾࡦ͕ݟ͑Δؾ͕͍ͯ͠Δ

Slide 45

Slide 45 text

5P#F$POUJOVFE

Slide 46

Slide 46 text

·ͩ·ͩख୳Γͷ ཁૉ΋ଟ͍Ͱ͕͢ ΍Γ͕͍͕͋Γ·͢ʂ

Slide 47

Slide 47 text

Slide 48

Slide 48 text

ϚωʔδυΫϥ΢υΛҰॹʹ࡞Γ·͠ΐ͏ ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU QIPUPIUUQTUFDIQFQBCPDPNFOHJOFFSUSBJOJOHWPM