Cookpad Tech Kitchen #12
dϩϦϙοϓʂϚωʔδυΫϥυͷཪଆdۙ౻Ӊஐ࿕(.01FQBCP *OD$PPLQBE5FDI,JUDIFOίϯςφͨͪΛܭଌ͢Δ͜ͱ
View Slide
ΤϯδχΞۙ౻Ӊஐ࿕!VE[VSBٕज़෦ٕज़ج൫νʔϜ͖ͳϥʔϝϯΒʔΊΜཾ(.0ϖύϘॴଐͬͯΔ3VCZɺNSVCZɺίϯςφΓ͍ͨγεςϜϓϩάϥϛϯάɺ-JOVY֦ு
ࢹɾܭଌʹ͍ͭͯɺࠓࢼߦࡨޡ͕ଓ͍͍ͯ·͕͢ࠓݕূͷաఔΛڞ༗͠·͢
ϩϦϙοϓʂϚωʔδυΫϥυIUUQTNDMPMJQPQKQΞϧϑΝςετத
IUUQTNDMPMJQPQKQΞϧϑΝςετத˞͝ڵຯͷ͋ΔํձͰΛֻ͓͚Լ͍͞
ࢹͱ
*5ΠϯϑϥͷࢹͱɺຊॻͰ࣍ͷ௨Γఆٛ͠·͢ɻোൃੜͷ༗ແΛҰఆظؒͰΓͳ֬͘ೝ͢Δ͜ͱɻʰ*5Πϯϑϥࢹ<࣮ફ>ೖʱΑΓ
ࢹɾϝτϦΫεͷతwӡ༻ͱͯ͠wαʔϏεμϯʹ͙͢ʹؾ͖͍ͨɻ·ͨɺ෮چΛૉૣ͍ͨ͘͠wαʔϏεͷػೳͱͯ͠wίϯςφͷෛՙঢ়گɺোঢ়ଶʹͳ͍ͬͯΔ͔൱͔ͳͲɻ·ͨͦͷཤྺΛͪΌΜͱՄࢹԽͯ͠ఏڙͰ͖Δͱخ͍͠wˠͦͷͨΊʹɺϝτϦΫεΛܭଌɺࢹɺෆਖ਼ͳঢ়ଶͷ࣌ΞϥʔτͳͲΛ͋͛ΒΕΔΑ͏ʹ͠ͳ͚ΕͳΒͳ͍ɻ
ࢹͷͨΊʹ·ͣܭଌ͠ͳ͍ͱ͍͚ͳ͍
ࠓɺࠔ͍ͬͯΔ͜ͱwίϯςφී௨ͷ7.ͷࢹͷΑ͏ʹ͍͔ͳ͍ͱ͜Ζ͋Δwˠϓϩηεʢ܈ʣͳͷͰ-"ͷΑ͏ͳఆ൪ͷ͕औΕͳ͔ͬͨΓͳͲwίϯςφϗεςΟϯάͰ͋Δ͜ͱɺ·ͨϚΠΫϩαʔϏεతͰ͋Δ͜ͱʹΑΔɺͭͷʮʯͷ͕͋Δ
ʮίϯςφʯͷwϚωʔδυΫϥυɺίϯςφΛར༻ͨ͠ߴूੵϓϥοτϑΥʔϜͰ͋Δɻ͢ͳΘͪɺίϯςφ͕ͱͯଟ͍ɻw͔͠ɺ'BTU$POUBJOFSΛ࠾༻͍ͯ͠ΔͳͲɺίϯςφىಈ͠ଓ͚ΔΘ͚Ͱͳ͍ɻ·ͨɺҰͭ࣌ͷυϝΠϯʹඥͮ͘ίϯςφͷมΘΔɻwͨ͘͞Μ͋ΔͷͰɺूΊΔͱɺ͔Γ͘͢ݟͤΔ͕͋Δ
ʮίϯϙʔωϯτʯͷwԿ͔ର֎ൃද͍ͯ͠Δ௨ΓɺϚωʔδυΫϥυίϯϙʔωϯτ͕ଟ͘ɺϚΠΫϩαʔϏεతʹΈ߹Θ͍ͬͯ͞ΔwʮαΠτ͕ݟ͑ͳ͍ʯͷΑ͏ͳ߹ʹɺݪҼͱߟ͑ΒΕΔՕॴͷಛఆ͕͘͠ɺΤϥʔτϥοΩϯάʹ՝͕͋ΔɻwϘτϧωοΫͷಛఆͳͲಉ༷
গͣͭ͠ɺ֎ງΛຒΊΔͦΜͳઓ͍ͷաఔΛ͠·͢
ͲΜͳ͕औΕΔ͔ʁ
ܭଌͷରw$16ϝϞϦར༻ͷࢹɾܭଌwωοτϫʔΫͷࢹɾܭଌw*0σΟεΫͷࢹɾܭଌwϦΫΤετͷτϨʔγϯάwίϯςφϛυϧΣΞͷϩάʢ͜ΕϩάύʔεͰૉʹूΊΔʣ
ࢀߟwͦʔ͍ͩ͞ΜͷࢿྉΑΓɺجຊతͳγεςϜʹؔΘΔϝτϦοΫIUUQTPVEBJIBUFOBCMPHDPNFOUSZ
DHSPVQͷTUBUͷར༻wDQVDQVBDDUαϒγεςϜwNFNPSZαϒγεςϜw ଞɺCMLJP WͷJP wςΩετσʔλͰɺར༻ঢ়گ͕Θ͔Δˠύʔεͯ͠׆༻͢ΔDHSPVQࣗମʹ͍ͭͯʮDHSPVQVE[VSBʯ·ͨ ʮDHSPVQ5FO'PSXBSEʯͰݕࡧ͢Δͱຊޠͷղઆ͕ɻɻɻroot@compute-1:~# cat /sys/fs/cgroup/cpu/$CONTAINER/cpu.statnr_periods 140373nr_throttled 25throttled_time 1112602844
ωοτϫʔΫʹؔ͢ΔϝτϦΫεwݪଇɺίϯςφࣗͷ/FUXPSL/BNFTQBDFʹॴଐ͢Δw/FUXPSL/BNFTQBDFͲ͏͠ʢࢠ͔ΒʣɺWFUIσόΠεΛܦ༝ͯ͠௨৴͢Δ͜ͱʹͳΔwͦͷWFUIσόΠεʹؔ͢Δز͔ͭͱΔखஈ͕͋ΔwJGUPQͷΑ͏ͳίϚϯυɺQSPDOFUEFWϑΝΠϧwϓϩηε୯ҐͷωοτϫʔΫใQSPD1*%OFU\EFW UDQ VEQ ^
':*ίϯςφωοτϫʔΫͷτϙϩδwCSJEHFઃఆͷྫɻ%PDLFSͷσϑΥϧτઃఆ͜ΕɻFUIEPDLFSWFUIIPTUWFUIHVFTUWFUIIPTUWFUIHVFTUWFUIIPTUWFUIHVFTUFHFH ඞཁʹԠ͡/"5ͳͲ/FUXPSL/BNFTQBDFͻͱͭͷ)PTUCSJEHF͜ΕΒΛࢹ͢Δʂ
*0ΞΧϯςΟϯάwDHSPVQͷCMLJPJPɺσόΠε୯ҐͷܭଌεϩοτϧͳͷͰɺ/'4ͷΑ͏ʹσόΠε൪߸͕ৼΒΕͳ͍ϑΝΠϧγεςϜʹ͍͠wҰํɺϓϩηε୯ҐͷSFBEʢʣɺXSJUFʢʣͷݺͼग़͠ճɺಡΈࠐΈॻ͖ࠐΈʹޭͨ͠όΠτͷྦྷܭɺΧʔωϧ͕ूܭ͍ͯͯ͠ར༻Ͱ͖ΔɻwQSPD1*%JPϑΝΠϧܦ༝Ͱ؆୯ʹ֬ೝՄೳwྫ͑͜ΕΛίϯςφʹॴଐ͢Δϓϩηε͝ͱʹूܭ͢Δ
ϦΫΤεττϥοΩϯάw͋ΔαʔόΛ௨ա͢Δ͝ͱʹಛఆͷϔομʔʹ࣌ؒΛه͍ͯ͘͠wDG9'PSXBSEFE'PSϔομʔw3BDLͷൃΛਅࣅͯɺϚʔΧʔΛωετ͍ͯ͘͠ͷͲ͏͔ʁwίϯϙωϯτ͝ͱͷܦա͕࣌ؒΘ͔ΔΑ͏ʹͳΓɺϘτϧωοΫͳͲ͕Θ͔ΔɻͦΕͦ͜(SBGBOBͳͲͰՄࢹԽͯ͠ྑ͍wΤϥʔτϥοΩϯάʹԠ༻Ͱ͖ΔʁIUUQTQZMPOTXFCGSBNFXPSLSFBEUIFEPDTJPFOWSDDPODFQUTIUNM
DGޚࣾͷࢄτϨʔγϯάج൫wؾʹͳΓ·͢IUUQUFDIMJGFDPPLQBEDPNFOUSZ
σʔλΛཷΊࠐΉ
େ·͔ͳΞʔΩςΫνϟwूΊΔˠஷΊΔˠՄࢹԽ͢Δwσʔλͷอଘઌ1SPNFUIFVTwՄࢹԽ(SBGBOBwσʔλΤʔδΣϯτD"EWJTPSNUBJMɺࣗ࡞FYQPSUFS
1SPNFUIFVT(SBGBOB&YQPSUFS&YQPSUFS&YQPSUFS$POUBJOFS$POUBJOFS$POUBJOFS$POUBJOFS$POUBJOFS"1*͚1SPYZूܭϝτϦΫεͷάϧʔϓผʹFYQPSUFSΛ༻ҙϢʔβ༷μογϡϘʔυ ʢ։ൃதʣӡ༻ΤϯδχΞ σΟϨΫλʔͳͲ"MFSUNBOBHFS
1SPNFUIFVTͷ࠾༻w(PPHMFͷ#PSHNPOʹӨڹΛड͚ͨࢹπʔϧw#PSHNPOͷհ43&ຊʹ͋Δͦ͏wΠϯετʔϧʢجຊతʹϫϯόΠφϦΛඋ͢Ε0,ʣɺ ӡ༻ʢجຊతʹΦϯϝϞϦʴϩʔΧϧετϨʔδʣ͕༰қwॊೈͳσʔλߏͱߴػೳͳΫΤϦw"MFSU࿈ܞ͕༰қw(SBGBOBͳͲՄࢹԽπʔϧ֤छFYQPSUFS͕ἧ͍ͬͯΔ
1SPNFUIFVTͱ(SBGBOBͷ࿈ܞw(SBGBOBઆ໌ͷඞཁͳ͍΄Ͳͷఆ൪μογϡϘʔυwެࣜʹυΩϡϝϯτ͕͋ΔwIUUQTQSPNFUIFVTJPEPDTWJTVBMJ[BUJPOHSBGBOBw1SPNFUIFVTͷΫΤϦΛͦͷ·Μ·ॻ͚ͨΓͯ͠ศརײ͕ߴ͍
Մ༻ੑͷߟ͑ํw1SPNFUIFVT֤ϊʔυͷFYQPSUFS͔ΒσʔλΛQVMM͍ͯ͘͠ελΠϧΛͱ͍ͬͯΔʢDG/BHJPT/31&4BB4ܕࢹʣwͦͷͨΊ"DUJWF"DUJWFߏΛऔΕಉ͡σʔλʹͳΓɺߏͱ͍͏͜ͱʹͳΔɻwSFGIUUQTHJUIVCDPNQSPNFUIFVTQSPNFUIFVTJTTVFT
4FSWJDF%JTDPWFSZwϊʔυ͔ΒQVMM͢ΔͷͰɺ1SPNFUIFVTϊʔυ͕Ͳ͜ʹ͋Δͷ͔ͱ͍͏ใΛ͍࣋ͬͯͳ͍ͱ͍͚ͳ͍ɻw4FSWJDF%JTDPWFSZͷͨΊͷํ๏ࣗମ͍͔ͭ͋͘ΔɻwϑΝΠϧϕʔεɺ%/4ϕʔεɺ"840QFO4UBDLͳͲ%ZOBNJD*OGSB͚ઃఆɺ͋Δ͍.BSBUIPO$POTVMͱ࿈ܞՄೳ
FYQPSUFSͷ֓೦ʹ͍ͭͯwσʔλΛऔΓ͍ͨϊʔυʹɺFYQPSUFSͱݴ͏ܾ·ͬͨϑΥʔϚοτͰσʔλΛు͖ग़͢ਓΛৗற͓ͤͯ͘͞ʢ࣮ଶIUUQ"1*ʣɻw1SPNFUIFVTɺ4FSWJDF%JTDPWFSZΛͯ͠ɺͦͷFYQPSUFSʹΞΫηε͠ɺఆظతʹσʔλΛಘΔwFYQPSUFSɺطʹ࣮͞Ε͍ͯΔͷͨ͘͞Μ͋Δ͠ɺ ࣗͰ࣮͢Δ͜ͱՄೳɻ
FYQPSUFSͷྫwOPEF@FYQPSUFSl&YQPSUFSGPSNBDIJOFNFUSJDTzw$16ɺϝϞϦɺ-"ɺωοτϫʔΫͳͲϚγϯ୯ҐͰ৭ʑͱूܭw(SBGBOBͳͲͰ͍͍ײ͡ʹάϥϑԽͯ͠׆༻wD"EWJTPS(PPHMF͚ࣾίϯςφࢹج൫Λϕʔεʹɺ%PDLFSͳͲʹରԠͨ͠ͷɻͪͳΈʹTZTUFNEͳͲͷࢹͰศརwͦͷଞɺଟ͘ͷ࣮͕͋Δɻී௨ࣗ࡞͠ͳ͍͍ͯͣ͘ IUUQTHJUIVCDPNQSPNFUIFVTEPDTCMPCNBTUFSDPOUFOUEPDTJOTUSVNFOUJOHFYQPSUFSTNE
D"EWJTPSͰ)BDPOJXBͷϝτϦΫεΛऔΔ
NUBJMIUUQTHJUIVCDPNHPPHMFNUBJMwઌड़ͨ͠/HJOYͳͲͷϩάΛύʔε্ͨ͠ͰɺFYQPSUFSʹͳͬͯ͘ΕΔπʔϧw%4-ʹΑΓϧʔϧΛهड़wԿ͔طࢹײͷ͋Δྗڧ͞ DG'MVFOUEͷJO@UBJMw/HJOYͷྫˠIUUQTHJUIVCDPNOEJB[HOHJOYQSPNFUIFVTFYQPSUFSCMPCNBTUFSpMFTQSPHTOHJOYNUBJMɹ
ܭଌͷରw$16ϝϞϦར༻ͷࢹɾܭଌwωοτϫʔΫͷࢹɾܭଌw*0σΟεΫͷࢹɾܭଌwϦΫΤετͷτϨʔγϯάwίϯςφϛυϧΣΞͷϩά(PPHMFπʔϧ܈ͷΤίγεςϜ͕ศརɻˡD"EWJTFSͰऔΕΔˡD"EWJTFSͰ͢ΕऔΕΔˡ·ͩཁݕ౼ˡNUBJMͰָͰ͖ΔˡώοʢOHY@NSVCZͳͲͰؤுΕͦ͏ʣ
FYQPSUFSࣗ࡞wͱ͍͑ɺࣗ࡞Ұͭͷख (PͳͲͳΒ1SPNFUIFVTެࣜͷϥΠϒϥϦ͕͑Δ طଘׂ࣮͕ͱ๛ ͦͦɺຬ͖ͨ͢ϑΥʔϚοτ͕୯७wIUUQTQSPNFUIFVTJPEPDTJOTUSVNFOUJOHXSJUJOH@FYQPSUFST ΨΠυΛݟͳ͕Βɺ͋ͱطଘ࣮ΛோΊͯॻ͘ͱྑͦ͞͏
FYQPSUFSͱ1SPNFUIFVTͷ࿈ܞw4FSWJDF%JTDPWFSZͷதͰ$POTVMͱͷ૬ੑྑ͍ͱ͍͏ࢲݟwFYQPSUFSଆͰɺ$POTVMͷTFSWJDFDIFDLΛՃˠಛఆͷαʔϏεΛ1SPNFUIFVT͔ΒσΟεΧόϦɺ͕ඇৗʹ༰қʹͳ͍ͬͯΔwྫ͑ҰʹෳFYQPSUFSͳͲ؆୯ͩ͠ɺFYQPSUFSͷՃআɺࢮ׆ࢹ$POTVMʹΔ͜ͱͰߟ͑ͳͯ͘ࡁΉ
͜ΜͳTFSWJDFఆٛΛॻ͚wεοͱ+0*/͢Δwศར{"service": {"name": "node_exporter","tags": ["development"],"port": 9100,"checks": [{"id": "ping","tcp": "localhost:9100","interval": "15s","timeout": "1s"}]}}
͜ͷষ·ͱΊwʮσʔλͷஷΊͲ͜Ζʯʹ1SPNFUIFVTΛਅΜதʹਾ͑Δw1SPNFUIFVTͷΤίγεςϜڧ͍ɻw͍͟ͱͳΕཉ͍͠σʔλࣗͰऔΓʹߦ͘w͜ΕΒͷͭͳ͗ʹ$POTVM͕ศར
ࠓޙΓ͍ͨ͜ͱ
ࠓޙΓ͍ͨ͜ͱwूΊΔͱ͜Ζ·ͰԿͱ͔ͳΓͦ͏wཷΊࠐΜͩσʔλͷ׆༻ʹ͍ͭͯw"MFSUNBOBHFSͷ֓೦ʢࠓলུʣwҟৗݕͷࣗಈԽɺՄࢹԽͳͲ
ҟৗݕͷࣗಈԽɺՄࢹԽwίϯςφʹؔ͢ΔϝτϦοΫεɺ$.%#ܦ༝ͰूͰ͖Δঢ়ଶwϓϩδΣΫτ͝ͱͷσʔλΛूΊͯԿ͔Ͱ͖ͳ͍͔1SPKFDUTEPNBJO$POUBJOFST$*% DPOUBJOFSJE$POUBJOFST$*% DPOUBJOFSJE.FUSJDT\EBUBlBBBz ^999.FUSJDT\EBUBlCCCz ^:::.FUSJDT\EBUBlDDDz ^;;;.FUSJDT\EBUBlEEEz ^888
ूΊͨσʔλΛͲ͏͢Δ͔w͜Ε͕ɺࠓޙͷ՝w୯७ʹάϥϑԽͯ͠ΈͯɺͦΕͳΓʹঢ়ଶ͕Θ͔Δؾ͢Δ͠ɺٸʹෛՙ͕ߴ͍ɺ௨৴Ͱ͖ͳ͘ͳͬͯΔɺͳͲࢹͰ͖ͦ͏Ͱ͋Δw·ͣӡ༻ͯ͠Έ͍ͨͱ͜Ζ
ՄࢹԽɾࣗಈఆͷख๏wϓϩδΣΫτʹඥ͍ͨσʔλɺجຊతʹଟ࣍ݩͷσʔλͰ͋ΔͷͰɺ࣍ݩѹॖΛͯ͠ɺʮ͜Εਖ਼ৗʯʮ͜Εোঢ়ଶʯͷΑ͏ʹྨ͕Ͱ͖ΔΜͰͳ͍͔ͱߟ͍͑ͯΔ͕ɺࠓޙͷ՝wେྔͷίϯςφϓϩδΣΫτΛɺࣗݾ৫ԽϚοϓͳͲͰࣗಈͰ ྨͯ͠ɺͦΕΛՄࢹԽͰ͖Δͱ໘നͦ͏ʁw͜ͷลΓʹɺʮͷʯͷղܾࡦ͕ݟ͑Δؾ͕͍ͯ͠Δ
5P#F$POUJOVFE
·ͩ·ͩख୳Γͷཁૉଟ͍Ͱ͕͢Γ͕͍͕͋Γ·͢ʂ
13
ϚωʔδυΫϥυΛҰॹʹ࡞Γ·͠ΐ͏࠷৽ͷ࠾༻ใΛνΣοΫˠ !QC@SFDSVJUQIPUPIUUQTUFDIQFQBCPDPNFOHJOFFSUSBJOJOHWPM