Save 37% off PRO during our Black Friday Sale! »

コンテナたちを計測すること ~ ロリポップ!マネージドクラウドの裏側 ~ /wip-how-to-get-container-metrics

2cf373725ded741824c50fd571eda6e1?s=47 KONDO Uchio
September 29, 2017

コンテナたちを計測すること ~ ロリポップ!マネージドクラウドの裏側 ~ /wip-how-to-get-container-metrics

Cookpad Tech Kitchen #12

2cf373725ded741824c50fd571eda6e1?s=128

KONDO Uchio

September 29, 2017
Tweet

Transcript

  1. dϩϦϙοϓʂϚωʔδυΫϥ΢υͷཪଆd ۙ౻Ӊஐ࿕(.01FQBCP *OD $PPLQBE5FDI,JUDIFO ίϯςφͨͪΛܭଌ͢Δ͜ͱ

  2. ΤϯδχΞ ۙ౻Ӊஐ࿕!VE[VSB ٕज़෦ٕज़ج൫νʔϜ ޷͖ͳϥʔϝϯ͸ΒʔΊΜ޻๪ཾ (.0ϖύϘॴଐ ΍ͬͯΔ෼໺3VCZɺNSVCZɺίϯςφ ΍Γ͍ͨ෼໺γεςϜϓϩάϥϛϯάɺ-JOVY֦ு

  3. ؂ࢹɾܭଌʹ͍ͭͯ͸ɺࠓ΋ ࢼߦࡨޡ͕ଓ͍͍ͯ·͕͢ ࠓ೔͸ݕূͷաఔΛڞ༗͠·͢

  4. ϩϦϙοϓʂϚωʔδυΫϥ΢υ IUUQTNDMPMJQPQKQ ΞϧϑΝςετத

  5. IUUQTNDMPMJQPQKQ ΞϧϑΝςετத ˞͝ڵຯͷ͋Δํ͸ձ৔Ͱ੠Λֻ͓͚Լ͍͞

  6. ؂ࢹͱ͸

  7. *5Πϯϑϥͷ؂ࢹͱ͸ɺຊॻͰ͸࣍ͷ௨Γఆ ٛ͠·͢ɻ ো֐ൃੜͷ༗ແΛҰఆظؒͰ଺Γͳ֬͘ೝ͢ Δ͜ͱɻ ʰ*5Πϯϑϥ؂ࢹ<࣮ફ>ೖ໳ʱΑΓ

  8. ؂ࢹɾϝτϦΫεͷ໨త wӡ༻ͱͯ͠ wαʔϏεμ΢ϯʹ͙͢ʹؾ෇͖͍ͨɻ·ͨɺ෮چΛૉૣ͍ͨ͘͠ wαʔϏεͷػೳͱͯ͠ wίϯςφͷෛՙঢ়گ΍ɺো֐ঢ়ଶʹͳ͍ͬͯΔ͔൱͔ͳͲɻ·ͨͦͷ ཤྺΛͪΌΜͱՄࢹԽͯ͠ఏڙͰ͖Δͱخ͍͠ wˠͦͷͨΊʹɺϝτϦΫεΛܭଌɺ؂ࢹɺෆਖ਼ͳঢ়ଶͷ࣌͸Ξϥʔ τͳͲΛ͋͛ΒΕΔΑ͏ʹ͠ͳ͚Ε͹ͳΒͳ͍ɻ

  9. ؂ࢹͷͨΊʹ ·ͣ͸ܭଌ͠ͳ͍ͱ͍͚ͳ͍

  10. ࠓɺࠔ͍ͬͯΔ͜ͱ wίϯςφ͸ී௨ͷ7.ͷ؂ࢹͷΑ͏ʹ͍͔ͳ͍ͱ͜Ζ΋͋Δ wˠϓϩηεʢ܈ʣͳͷͰ-"ͷΑ͏ͳఆ൪ͷ஋͕औΕͳ͔ͬͨΓͳͲ wίϯςφϗεςΟϯάͰ͋Δ͜ͱɺ·ͨϚΠΫϩαʔϏεతͰ͋Δ͜ ͱʹΑΔɺͭͷʮ਺ʯͷ໰୊͕͋Δ

  11. ʮίϯςφ਺ʯͷ໰୊ wϚωʔδυΫϥ΢υ͸ɺίϯςφΛར༻ͨ͠ߴूੵϓϥοτϑΥʔϜ Ͱ͋Δɻ͢ͳΘͪɺίϯςφ͕ͱͯ΋ଟ͍ɻ w͔͠΋ɺ'BTU$POUBJOFSΛ࠾༻͍ͯ͠ΔͳͲɺίϯςφ͸ىಈ͠ଓ͚ ΔΘ͚Ͱ͸ͳ͍ɻ·ͨɺҰͭ࣌ͷυϝΠϯʹඥͮ͘ίϯςφͷ਺΋ม ΘΔɻ wͨ͘͞Μ͋ΔͷͰɺूΊΔ໰୊ͱɺ෼͔Γ΍͘͢ݟͤΔ໰୊͕͋Δ

  12. ʮίϯϙʔωϯτ਺ʯͷ໰୊ wԿ౓͔ର֎ൃද͍ͯ͠Δ௨ΓɺϚωʔδυΫϥ΢υ͸ίϯϙʔωϯτ ͕ଟ͘ɺϚΠΫϩαʔϏεతʹ૊Έ߹Θ͍ͬͯ͞Δ wʮαΠτ͕ݟ͑ͳ͍ʯͷΑ͏ͳ৔߹ʹɺݪҼͱߟ͑ΒΕΔՕॴͷಛఆ ͕೉͘͠ɺΤϥʔτϥοΩϯάʹ՝୊͕͋Δɻ wϘτϧωοΫͷಛఆͳͲ΋ಉ༷

  13. গͣͭ͠ɺ֎ງΛຒΊΔ ͦΜͳઓ͍ͷաఔΛ࿩͠·͢

  14. ͲΜͳ஋͕औΕΔ͔ʁ

  15. ܭଌͷର৅ w$16ϝϞϦར༻཰ͷ؂ࢹɾܭଌ wωοτϫʔΫͷ؂ࢹɾܭଌ w*0΍σΟεΫͷ؂ࢹɾܭଌ wϦΫΤετͷτϨʔγϯά wίϯςφ಺ϛυϧ΢ΣΞͷϩάʢ͜Ε͸ϩάύʔεͰૉ௚ʹूΊΔʣ

  16. ࢀߟ wͦʔ͍ͩ͞ΜͷࢿྉΑΓɺجຊతͳγεςϜʹؔΘΔϝτϦοΫ IUUQTPVEBJIBUFOBCMPHDPNFOUSZ

  17. DHSPVQͷTUBUͷར༻ wDQVDQVBDDUαϒγεςϜ wNFNPSZαϒγεςϜ w ଞɺCMLJP WͷJP   wςΩετσʔλͰɺར༻ঢ়گ͕Θ͔Δˠύʔεͯ͠׆༻͢Δ DHSPVQࣗମʹ͍ͭͯ͸ʮDHSPVQVE[VSBʯ·ͨ͸


    ʮDHSPVQ5FO'PSXBSEʯͰݕࡧ͢Δͱ೔ຊޠͷղઆ͕ɻɻɻ root@compute-1:~# cat /sys/fs/cgroup/cpu/$CONTAINER/cpu.stat nr_periods 140373 nr_throttled 25 throttled_time 1112602844
  18. ωοτϫʔΫʹؔ͢ΔϝτϦΫε wݪଇɺίϯςφ͸ࣗ෼ͷ/FUXPSL/BNFTQBDFʹॴଐ͢Δ w/FUXPSL/BNFTQBDFͲ͏͠ʢࢠ͔Β਌ʣ͸ɺWFUIσόΠεΛܦ༝ ͯ͠௨৴͢Δ͜ͱʹͳΔ wͦͷWFUIσόΠεʹؔ͢Δ஋͸ز͔ͭͱΔखஈ͕͋Δ wJGUPQͷΑ͏ͳίϚϯυɺQSPDOFUEFWϑΝΠϧ wϓϩηε୯ҐͷωοτϫʔΫ৘ใQSPD1*%OFU\EFW UDQ VEQ ^

  19. ':*ίϯςφωοτϫʔΫͷτϙϩδ wCSJEHFઃఆͷྫɻ%PDLFSͷσϑΥϧτઃఆ΋͜Εɻ FUI EPDLFS WFUIIPTU WFUIHVFTU WFUIIPTU WFUIHVFTU WFUIIPTU WFUIHVFTU

    FH FH    ඞཁʹԠ͡/"5ͳͲ /FUXPSL /BNFTQBDF ͻͱͭͷ)PTU CSJEHF ͜ΕΒΛ؂ࢹ͢Δʂ
  20. *0ΞΧ΢ϯςΟϯά wDHSPVQͷCMLJPJP͸ɺσόΠε୯Ґͷܭଌ΍εϩοτϧͳͷͰɺ/'4 ͷΑ͏ʹσόΠε൪߸͕ৼΒΕͳ͍ϑΝΠϧγεςϜʹ͸೉͍͠ wҰํɺϓϩηε୯ҐͷSFBEʢ଒ʣɺXSJUFʢ଒ʣͷݺͼग़͠ճ਺΍ɺ ಡΈࠐΈॻ͖ࠐΈʹ੒ޭͨ͠όΠτ਺ͷྦྷܭ͸ɺΧʔωϧ͕ूܭͯ͠ ͍ͯར༻Ͱ͖Δɻ wQSPD1*%JPϑΝΠϧܦ༝Ͱ؆୯ʹ֬ೝՄೳ wྫ͑͹͜ΕΛίϯςφʹॴଐ͢Δϓϩηε͝ͱʹूܭ͢Δ

  21. ϦΫΤεττϥοΩϯά w͋ΔαʔόΛ௨ա͢Δ͝ͱʹಛఆͷϔομʔʹ࣌ؒΛ௥ه͍ͯ͘͠ wDG9'PSXBSEFE'PSϔομʔ w3BDL౳ͷൃ૝ΛਅࣅͯɺϚʔΧʔΛωετ͍ͯ͘͠ͷ͸Ͳ͏͔ʁ wίϯϙωϯτ͝ͱͷܦա͕࣌ؒΘ͔ΔΑ͏ʹͳΓɺϘτϧωοΫͳͲ ͕Θ͔ΔɻͦΕͦ͜(SBGBOBͳͲͰՄࢹԽͯ͠΋ྑ͍ wΤϥʔτϥοΩϯάʹ΋Ԡ༻Ͱ͖Δʁ IUUQTQZMPOTXFCGSBNFXPSLSFBEUIFEPDTJPFOWSDDPODFQUTIUNM

  22. DGޚࣾͷ෼ࢄτϨʔγϯάج൫ wؾʹͳΓ·͢ IUUQUFDIMJGFDPPLQBEDPNFOUSZ

  23. σʔλΛཷΊࠐΉ

  24. େ·͔ͳΞʔΩςΫνϟ wूΊΔˠஷΊΔˠՄࢹԽ͢Δ wσʔλͷอଘઌ1SPNFUIFVT wՄࢹԽ(SBGBOB wσʔλΤʔδΣϯτD"EWJTPS΍NUBJMɺࣗ࡞FYQPSUFS

  25. 1SPNFUIFVT (SBGBOB &YQPSUFS &YQPSUFS &YQPSUFS $POUBJOFS $POUBJOFS $POUBJOFS $POUBJOFS $POUBJOFS

    "1*޲͚1SPYZ ूܭϝτϦΫεͷάϧʔϓผʹ FYQPSUFSΛ༻ҙ Ϣʔβ༷μογϡϘʔυ
 ʢ։ൃதʣ ӡ༻ΤϯδχΞ
 σΟϨΫλʔͳͲ "MFSUNBOBHFS
  26. 1SPNFUIFVTͷ࠾༻ w(PPHMFͷ#PSHNPOʹӨڹΛड͚ͨ؂ࢹπʔϧ w#PSHNPOͷ঺հ͸43&ຊʹ΋͋Δͦ͏ wΠϯετʔϧʢجຊతʹ͸ϫϯόΠφϦΛ഑උ͢Ε͹0,ʣ΍ɺ
 ӡ༻ʢجຊతʹΦϯϝϞϦʴϩʔΧϧετϨʔδʣ͕༰қ wॊೈͳσʔλߏ଄ͱߴػೳͳΫΤϦ w"MFSU࿈ܞ͕༰қ w(SBGBOBͳͲՄࢹԽπʔϧ΍֤छFYQPSUFS͕ἧ͍ͬͯΔ

  27. 1SPNFUIFVTͱ(SBGBOBͷ࿈ܞ w(SBGBOB͸΋͸΍આ໌ͷඞཁ΋ͳ͍΄Ͳͷఆ൪μογϡϘʔυ wެࣜʹ΋υΩϡϝϯτ͕͋Δ wIUUQTQSPNFUIFVTJPEPDTWJTVBMJ[BUJPOHSBGBOB w1SPNFUIFVTͷΫΤϦΛͦͷ·Μ·ॻ͚ͨΓͯ͠ศརײ͕ߴ͍

  28. Մ༻ੑͷߟ͑ํ w1SPNFUIFVT͸֤ϊʔυͷFYQPSUFS͔ΒσʔλΛQVMM͍ͯ͘͠ελΠ ϧΛͱ͍ͬͯΔʢDG/BHJPT /31&4BB4ܕ؂ࢹʣ wͦͷͨΊ"DUJWF"DUJWFߏ੒ΛऔΕ͹ಉ͡σʔλʹͳΓɺ৑௕ߏ੒ͱ ͍͏͜ͱʹͳΔɻ wSFGIUUQTHJUIVCDPNQSPNFUIFVTQSPNFUIFVTJTTVFT

  29. 4FSWJDF%JTDPWFSZ wϊʔυ͔ΒQVMM͢ΔͷͰɺ1SPNFUIFVT͸ϊʔυ͕Ͳ͜ʹ͋Δͷ͔ͱ ͍͏৘ใΛ͍࣋ͬͯͳ͍ͱ͍͚ͳ͍ɻ w4FSWJDF%JTDPWFSZͷͨΊͷํ๏ࣗମ͸͍͔ͭ͋͘Δɻ wϑΝΠϧϕʔεɺ%/4ϕʔεɺ"840QFO4UBDLͳͲ%ZOBNJD*OGSB ޲͚ઃఆɺ͋Δ͍͸.BSBUIPO΍$POTVMͱ΋࿈ܞՄೳ

  30. FYQPSUFSͷ֓೦ʹ͍ͭͯ wσʔλΛऔΓ͍ͨϊʔυʹɺFYQPSUFSͱݴ͏ܾ·ͬͨϑΥʔϚοτͰ σʔλΛు͖ग़͢ਓΛৗற͓ͤͯ͘͞ʢ࣮ଶ͸IUUQ"1*ʣɻ w1SPNFUIFVT͸ɺ4FSWJDF%JTDPWFSZΛͯ͠ɺͦͷFYQPSUFSʹΞΫη ε͠ɺఆظతʹσʔλΛಘΔ wFYQPSUFS͸ɺطʹ࣮૷͞Ε͍ͯΔ΋ͷ΋ͨ͘͞Μ͋Δ͠ɺ
 ࣗ෼Ͱ࣮૷͢Δ͜ͱ΋Մೳɻ

  31. FYQPSUFSͷྫ wOPEF@FYQPSUFSl&YQPSUFSGPSNBDIJOFNFUSJDTz w$16ɺϝϞϦɺ-"ɺωοτϫʔΫͳͲϚγϯ୯ҐͰ৭ʑͱूܭ w(SBGBOBͳͲͰ͍͍ײ͡ʹάϥϑԽͯ͠׆༻ wD"EWJTPS(PPHMFࣾ಺޲͚ίϯςφ؂ࢹج൫Λϕʔεʹɺ%PDLFSͳ ͲʹରԠͨ͠΋ͷɻͪͳΈʹTZTUFNEͳͲͷ؂ࢹͰ΋ศར wͦͷଞɺଟ͘ͷ࣮૷͕͋Δɻී௨͸ࣗ࡞͠ͳ͍͍ͯ͘͸ͣ
 IUUQTHJUIVCDPNQSPNFUIFVTEPDTCMPCNBTUFSDPOUFOUEPDTJOTUSVNFOUJOHFYQPSUFSTNE

  32. D"EWJTPSͰ)BDPOJXBͷϝτϦΫεΛऔΔ

  33. D"EWJTPSͰ)BDPOJXBͷϝτϦΫεΛऔΔ

  34. NUBJMIUUQTHJUIVCDPNHPPHMFNUBJM wઌड़ͨ͠/HJOYͳͲͷϩάΛύʔε্ͨ͠ͰɺFYQPSUFSʹ΋ͳͬͯ͘ ΕΔπʔϧ w%4-ʹΑΓϧʔϧΛهड़ wԿ͔طࢹײͷ͋Δྗڧ͞
 DG'MVFOUEͷJO@UBJM  w/HJOYͷྫˠ IUUQTHJUIVCDPNOEJB[HOHJOYQSPNFUIFVTFYQPSUFSCMPCNBTUFSpMFTQSPHTOHJOYNUBJMɹ

  35. ܭଌͷର৅ w$16ϝϞϦར༻཰ͷ؂ࢹɾܭଌ wωοτϫʔΫͷ؂ࢹɾܭଌ w*0΍σΟεΫͷ؂ࢹɾܭଌ wϦΫΤετͷτϨʔγϯά wίϯςφ಺ϛυϧ΢ΣΞͷϩά (PPHMFπʔϧ܈ͷΤίγεςϜ͕ศརɻ ˡD"EWJTFSͰऔΕΔ ˡD"EWJTFSͰ޻෉͢Ε͹औΕΔ ˡ·ͩཁݕ౼

    ˡNUBJMͰָͰ͖Δ ˡώοʢOHY@NSVCZͳͲͰؤுΕͦ͏ʣ
  36. FYQPSUFSࣗ࡞ wͱ͸͍͑ɺࣗ࡞΋Ұͭͷख
  (PͳͲͳΒ1SPNFUIFVTެࣜͷϥΠϒϥϦ͕࢖͑Δ
  طଘ࣮૷ׂ͕ͱ๛෋
  ͦ΋ͦ΋ɺຬͨ͢΂͖ϑΥʔϚοτ͕୯७ wIUUQTQSPNFUIFVTJPEPDTJOTUSVNFOUJOHXSJUJOH@FYQPSUFST
 ΨΠυΛݟͳ͕Βɺ͋ͱطଘ࣮૷ΛோΊͯॻ͘ͱྑͦ͞͏

  37. FYQPSUFSͱ1SPNFUIFVTͷ࿈ܞ w4FSWJDF%JTDPWFSZͷதͰ΋$POTVMͱͷ૬ੑ͸ྑ͍ͱ͍͏ࢲݟ wFYQPSUFSଆͰɺ$POTVMͷTFSWJDFDIFDLΛ௥ՃˠಛఆͷαʔϏεΛ 1SPNFUIFVT͔ΒσΟεΧόϦɺ͕ඇৗʹ༰қʹͳ͍ͬͯΔ wྫ͑͹Ұ୆ʹෳ਺FYQPSUFSͳͲ΋؆୯ͩ͠ɺFYQPSUFSͷ௥Ճ࡟আɺ ࢮ׆؂ࢹ΋$POTVMʹ৐Δ͜ͱͰߟ͑ͳͯ͘ࡁΉ

  38. ͜ΜͳTFSWJDFఆٛΛॻ͚͹ wεοͱ+0*/͢Δ wศར { "service": { "name": "node_exporter", "tags": [

    "development" ], "port": 9100, "checks": [ { "id": "ping", "tcp": "localhost:9100", "interval": "15s", "timeout": "1s" } ] } }
  39. ͜ͷষ·ͱΊ wʮσʔλͷஷΊͲ͜Ζʯʹ1SPNFUIFVTΛਅΜதʹਾ͑Δ w1SPNFUIFVTͷΤίγεςϜ͸ڧ͍ɻ w͍͟ͱͳΕ͹ཉ͍͠σʔλ͸ࣗ෼ͰऔΓʹߦ͘ w͜ΕΒͷͭͳ͗ʹ$POTVM͕ศར

  40. ࠓޙ΍Γ͍ͨ͜ͱ

  41. ࠓޙ΍Γ͍ͨ͜ͱ wूΊΔͱ͜Ζ·Ͱ͸Կͱ͔ͳΓͦ͏ wཷΊࠐΜͩσʔλͷ׆༻ʹ͍ͭͯ w"MFSUNBOBHFSͷ֓೦ʢࠓ೔͸লུʣ wҟৗݕ஌ͷࣗಈԽɺՄࢹԽͳͲ

  42. ҟৗݕ஌ͷࣗಈԽɺՄࢹԽ wίϯςφʹؔ͢ΔϝτϦοΫε͸ɺ$.%#ܦ༝Ͱू໿Ͱ͖Δঢ়ଶ wϓϩδΣΫτ͝ͱͷσʔλΛूΊͯԿ͔Ͱ͖ͳ͍͔ 1SPKFDUT EPNBJO $POUBJOFST $*% DPOUBJOFSJE $POUBJOFST $*%

    DPOUBJOFSJE .FUSJDT\EBUBlBBBz ^999 .FUSJDT\EBUBlCCCz ^::: .FUSJDT\EBUBlDDDz ^;;; .FUSJDT\EBUBlEEEz ^888
  43. ूΊͨσʔλΛͲ͏͢Δ͔ w͜Ε͕ɺࠓޙͷ՝୊ w୯७ʹάϥϑԽͯ͠Έͯ΋ɺͦΕͳΓʹঢ়ଶ͕Θ͔Δؾ΋͢Δ͠ɺٸ ʹෛՙ͕ߴ͍ɺ௨৴Ͱ͖ͳ͘ͳͬͯΔɺͳͲ͸໨ࢹͰ͖ͦ͏Ͱ͸͋Δ w·ͣ͸ӡ༻ͯ͠Έ͍ͨͱ͜Ζ

  44. ՄࢹԽɾࣗಈ൑ఆͷख๏ wϓϩδΣΫτʹඥ෇͍ͨσʔλ͸ɺجຊతʹଟ࣍ݩͷσʔλͰ͋Δͷ Ͱɺ࣍ݩѹॖΛͯ͠ɺʮ͜Ε͸ਖ਼ৗʯʮ͜Ε͸ো֐ঢ়ଶʯͷΑ͏ʹ෼ ྨ͕Ͱ͖ΔΜͰ͸ͳ͍͔ͱߟ͍͑ͯΔ͕ɺࠓޙͷ՝୊ wେྔͷίϯςφϓϩδΣΫτΛɺࣗݾ૊৫ԽϚοϓͳͲͰࣗಈͰ
 ෼ྨͯ͠ɺͦΕΛՄࢹԽͰ͖Δͱ໘നͦ͏ʁ w͜ͷลΓʹɺʮ਺ͷ໰୊ʯͷղܾࡦ͕ݟ͑Δؾ͕͍ͯ͠Δ

  45. 5P#F$POUJOVFE

  46. ·ͩ·ͩख୳Γͷ ཁૉ΋ଟ͍Ͱ͕͢ ΍Γ͕͍͕͋Γ·͢ʂ

  47. 13

  48. ϚωʔδυΫϥ΢υΛҰॹʹ࡞Γ·͠ΐ͏ ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU QIPUPIUUQTUFDIQFQBCPDPNFOHJOFFSUSBJOJOHWPM