Monitoring at AbemaTV

"CFNB57ʹ͓͚ΔϞχλϦϯά #BDLFOE&OHJOFFS -&7"//()*"

"CPVUNF w -&7"//()*" ΪΞʣ w ϕτφϜग़਎ w (JUIVC!OHIJBMW w 5XJUUFS!OHIJBMW
w ೥৽ଔͱͯ͠$ZCFS"HFOUʹೖࣾ w "NFCB 'SFTIੜ์ૹʂͰJ04ΤϯδχΞ w ͔Β"CFNB57ͷόοΫΤϯυΤϯδχΞ

"HFOEB w "CFNB57ͷΞʔΩςΫνϟ w ϞχλϦϯάγεςϜͷਐԽ w τϥϑΟοΫͷϦΞϧλΠϜՄࢹԽ w ࠓޙͷ͜ͱ w
⁞ w   w ⁠ w ⁡

⁞"CFNB57ͷΞʔΩςΫνϟ

($1 ,VCFSOFUFT H31$ (PPHMF$MPVE1MBUGPSN ,VCFSOFUFT .JDSPTFSWJDF .JDSPTFSWJDF .JDSPTFSWJDF H31$ H31$

αʔϏε͔ΒݟΔ

 ϞχλϦϯάγεςϜͷਐԽ

4UBDLESJWFSͷΈͷ࣌ظ w ຊ։ہ͔Β൒೥͙Β͍ w ($&Πϯελϯε w ϚγϯϝτϦΫε $16 .FNPSZ /FUXPSLʜ
w 7.্ʹಈ͍͍ͯΔ.POHP%# 3FEJTͷϝτϦΫε w ($1αʔϏε w 1VC4VC 4UPSBHF -PBE#BMBODJOH ʜ 4UBDLESJWFS"HFOU .POHP%# 7. NPOHPECWN

؂ࢹ͍ͨ͠ϨΠϠ )PTU 0SDIFTUSBUJPO $POUBJOFS "QQMJDBUJPO

"CFNB57༻ͷϞχλϦϯάγεςϜ͕ඞཁ

1SPNFUIFVT (SBGBOB

Կނ1SPNFUIFVT w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w
"MFSUNBOBHFS͕ศར

"MFSUNBOBHFS͕ศར ˔ 1SPNFUIFVT BQJTFSWFS 1PE 1PE 1PE TFSWJDFEJTDPWFSZ TDSBQFNFUSJDT .BTUFSOPEF 8PSLFSOPEF

Կނ1SPNFUIFVT 1SPNFUIFVT BQJTFSWFS 1PE 1PE 1PE TFSWJDFEJTDPWFSZ TDSBQFNFUSJDT .BTUFSOPEF 8PSLFSOPEF
w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔

Կނ1SPNFUIFVT 1SPNFUIFVT BQJTFSWFS 1PE 1PE TFSWJDFEJTDPWFSZ TDSBQFNFUSJDT .BTUFSOPEF 8PSLFSOPEF 1PE

Կނ1SPNFUIFVT 1SPNFUIFVT BQJTFSWFS 1PE 1PE TFSWJDFEJTDPWFSZ TDSBQFNFUSJDT .BTUFSOPEF 8PSLFSOPEF 1PE
1PE w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔

"MFSUNBOBHFS͕ศར ˔ ύϑΥʔϚϯεͷ໘ JOTUBODFLTBNQMFTTFDPOEɹ 4PVOE$MPVEͷ࣮੷ UJNFTFSJFTQFSUBSHFU TTDBQFJOUFSWBM JOTUBODF͕ UBSHFUT·ͰରԠͰ͖Δ

Կނ1SPNFUIFVT σʔλྔͷ໘ NJMMJPOBDUJWFUJNFTFSJFT TTDSBQFJOUFSWBM NPOUIPGSFUFOUJPO ѹॖͳ͠ CZUFUJNFTUBNQ CZUFWBMVF‎5# 1SPNFUIFVTͷWBSCJUΤϯίʔυ CZUFTBNQMF‎5#

"MFSUNBOBHFS͕ศར ˔

"MFSUNBOBHFS͕ศར ˔ w QSPNFUIFVTDMJFOU@HPMBOH w $PVOUFS w (BVHF w 4VNNBSZ w )JTUPHSBN w طଘͷ&YQPSUFS΋ଟ͍ w OPEFFYQPSUFS w NPOHPEC SFEJT NZTRM ʜ

"MFSUNBOBHFS͕ศར ˔ w 1SPNFUIFVT͕ൃੜͨ͠Ξϥʔτͷ؅ཧ w ॏෳ͍ͯ͠ΔΞϥʔτͷղআ w ΞϥʔτͷάϧʔϓԽ w ௨஌ͷҰ࣌తࢭͳఀࢭ w ৭ʑͳ௨৴ઌͷରԠ w 4MBDL &NBJM )5518FCIPPL

IUUQTEFWFMPQFSTDZCFSBHFOUDPKQCMPHBSDIJWFT #MPHl"CFNB57ʹ1SPNFUIFVTͱ͍͏ϞχλϦϯάγεςϜΛಋೖͨ͠࿩z

1SPNFUIFVTͷಋೖͷॳ࣌ظ w Ϋϥελ಺ʹͭͷ1SPNFUIFVTαʔό w ΞϓϦέʔγϣϯͷϝτϦΫεͷΈ w 1SPNFUIFVTͷ$POpHΛ  $POpH.BQʹॻ͘ w λʔήοταʔϏεͷTFSWJDFZBNMʹ
BOOPUBUJPO௥Ճ (SBGBOB "MFSUNBOBHFS 1SPNFUIFVT (BUFXBZ &YQPSUFS .FEJBQSPYZ &YQPSUFS 6TFS &YQPSUFS $PNNFOU &YQPSUFS 8BUDINBO &YQPSUFS ʜ $POpH.BQ FUDE apiVersion: v1 kind: Service metadata: name: abema-demo annotations: prometheus.io/scrape: "true" prometheus.io/port: "12345" prometheus.io/path: “/metrics"

ϝτϦΫε w H31$ w SFRVFTU FSSPS EVSBUJPO w IUUQ w
SFRVFTU FSSPS EVSBUJPO w HPMBOH w HPSPVUJOF IFBQ TUBDL HBSCBHFDPMMFDUJPO w QSPDFTT w DQVUJNF NFNPSZTJ[F

4MPXRVFSZ໰୊ w (SBGBOBଆͰෳࡶͳ2VFSZΛॻ͘ͱɺ  ଴͕ͪ࣌ؒ௕͍ w 1SPNFUIFVTͰ3FDPSEJOH3VMFTΛ௥Ճ w ࣄલܭࢉΛߦ͍ɺ  ৽͍͠UJNFTFSJFTͱͯ͠อଘ 100
* sum by (service) ( rate(abema_http_requests_total{status=~"^5..$"}[2m]) ) / sum by (service) ( rate(abema_http_requests_total[2m]) ) (SBGBOB service:abema_http_request_5xx_percent:rate2m = 100 * sum by (service) ( rate(abema_http_requests_total{status=~"^5..$"}[2m]) ) / sum by (service) ( rate(abema_http_requests_total[2m]) ) 1SPNFUIFVT service:abema_http_request_5xx_percent:rate2m (SBGBOB

਺ϲ݄ޙ

໰୊ɾ՝୊ w ࣌ܥྻσʔλྔͷ૿Ճ w )JHI"WBJMBCJMJUZ1SPNFUIFVTͷඞཁ w ӡ༻ͷେม

࣌ܥྻσʔλྔͷ૿Ճ w ϞχλϦϯάλʔήοτͷ૿Ճ w αʔόʔ୆਺ w $POUBJOFS਺ w .JDSPTFSWJDFT਺ w
৽͍͠ϝτϦΫεͷ૿Ճ

৽͍͠ϝτϦΫεͷྫ

/PEFϝτϦΫε w /PEFFYQPSUFS w $16 .FNPSZ %JTL /FUXPSL ʜ w
,TOPEF w %BFNPO4FUͱ֤ͯ͠OPEFʹ  ҰͭͷQPEΛಈ͔͢ w ($&OPEF w 1BDLFSͰNBDIJOFJNBHFͷ  Ϗϧυ࣌ʹೖΕΔ OPEFEBTICPBSE

LTͷϝτϦΫε w BQJTFSWFS w LVCFMFU w LTTUBUFNFUSJDT w LVCFEOT QPEEBTICPBSE

$$6ϝτϦΫε w ಉ࣌઀ଓ਺ͷϝτϦΫε w ֤$IBOOFM 04 4USFBNJOH1SPUPDPM   3FTPMVUJPO 1MBZFS
ʜ w ΞΫηεϩά͔ΒϦΞϧλΠϜܭࢉ w )ZQFS-PH-PHΞϧΰϦζϜ w )PSJ[POUBM1PE"VUPTDBMFSͰ  DDVDPMMFDUPSΛࣗಈεέʔϦϯά .FEJBQSPYZ 1VC4VC DDVDPMMFDUPS DDVFYQPSUFS DDVSFEJT 1SPNFUIFVT DDVDPMMFDUPS .FEJBQSPYZ MPH #JH2VFSZ

4UBDLESJWFSͷϝτϦΫε w (PPHMFͷNPOJUPSJOHBQJΛ༻͍ͯಠࣗͷFYQPSUFSΛ࡞੒ w -PBE#BMBODJOH w 4UPSBHF w 1VC4VC w
#JH2VFSZ w %BUBqPX TUBDLESJWFSFYQPSUFS HPPHMFNPOJUPSJOHBQJ 1SPNFUIFVT

Ұͭͷ1SPNFUIFVTͰ͸ݫ͍͠ ࣌ܥྻσʔλྔͷ૿ՃͷͨΊ

1SPNFUIFVTTIBSEJOH w ϝτϦΫελΠϓ w ϝτϦΫεछྨͰ1SPNFUIFVTΛ෼͚Δ w λʔήοτ w औಘର৅Ͱ1SPNFUIFVTΛ෼͚Δ

1SPNFUIFVTTIBSEJOH w ϝτϦΫελΠϓ w ϝτϦΫεछྨͰ1SPNFUIFVTΛ෼͚Δ w λʔήοτ w औಘର৅Ͱ1SPNFUIFVTΛ෼͚Δ ˔
QSPNFUIFVT TFSWJDF" TFSWJDF# NFUSJDT NFUSJDT OPEF FYQPSUFS NFUSJDT

OPEF FYQPSUFS OPEFNFUSJDT TFSWJDF" HPNFUSJDT IUUQNFUSJDT TFSWJDF# HPNFUSJDT HSQDNFUSJDT QSPNFUIFVT

OPEF FYQPSUFS OPEFNFUSJDT TFSWJDF" HPNFUSJDT IUUQNFUSJDT TFSWJDF# HPNFUSJDT HSQDNFUSJDT QSPNFUIFVTHP QSPNFUIFVTHSQD QSPNFUIFVTIUUQ QSPNFUIFVTOPEF

QSPNFUIFVT TFSWJDF" NFUSJDT TFSWJDF# NFUSJDT TFSWJDF$ NFUSJDT TFSWJDF% NFUSJDT

1SPNFUIFVTTIBSEJOH TFSWJDF" NFUSJDT TFSWJDF# NFUSJDT TFSWJDF$ NFUSJDT TFSWJDF% NFUSJDT QSPNFUIFVT
QSPNFUIFVT w ϝτϦΫελΠϓ w ϝτϦΫεछྨͰ1SPNFUIFVTΛ෼͚Δ w λʔήοτ w औಘର৅Ͱ1SPNFUIFVTΛ෼͚Δ ˔

)"1SPNFUIFVT w ܦҢ w 1SPNFUIFVTͷϝτϦΫε͔ΒNJDSPTFSWJDFTΛࣗಈεέʔϦϯά w ಉ͡$POpHɺಉ͡λʔήοτϦετͰෳ਺1SPNFUIFVTΛߏங w NFNPSZ΍EJTLͷσʔλ͸΄΅ಉ͡ w
σʔλͷࠩ෼ͷٵऩ w Ξϥʔτ w ීஈͷΞϥʔτ͸Ұఆͷ࣌ؒͰ൑ఆ w (SBGBOB w εςΟοΩʔηογϣϯ TFSWJDF" NFUSJDT TFSWJDF# NFUSJDT QSPNFUIFVT QSPNFUIFVT QSPNFUIFVT

ӡ༻ w $POpHϑΝΠϧΛॻ͘ͷ͕େม w TIBSEJOHͰྔ͕ଟ͘ͳΓɺઃఆ͕ෳࡶ w SFMBCFMJOH͕෼͔Γʹ͍͘ w ৽ϝτϦΫεͷ௥Ճ΍EBTICPBSETͷमਖ਼͸  νʔϜϝϯόʔͦΕͧΕͰ΍Γ͍ͨ
w DPOpH΍EBTICPBSEΛߋ৽͍ͨ͠৔߹ɺ  Ϧελʔτ͕ඞཁ OPEF FYQPSUFS OPEFNFUSJDT TFSWJDF" HPNFUSJDT IUUQNFUSJDT TFSWJDF# HPNFUSJDT HSQDNFUSJDT QSPNFUIFVTHP QSPNFUIFVTHSQD QSPNFUIFVTIUUQ QSPNFUIFVTOPEF

1SPNFUIFVT0QFSBUPS

1SPNFUIFVT0QFSBUPS w $POpHϑΝΠϧΛࣗಈੜ੒ w DPOpHͷ୅ΘΓʹɺ  TFSWJDFNPOJUPSΛهड़  w ϗοτϦϩʔυ w 4JEFDBSίϯςφʔͰDPOpHɺ
EBTICPBSEΛ؂ࢹɺϦϩʔυ TFSWJDF" MBCFMT UZQFIUUQ TFSWJDF# MBCFMT UZQFIUUQ QSPNFUIFVTIUUQ TFSWJDFNPOJUPS QSPNUIFVTIUUQ TFMFDUPS NBUDI-BCFMT UZQFIUUQ $POpH 8BUDI (FOFSBUF QSPNFUIFVTPQFSBUPS

1SPNFUIFVT0QFSBUPS w $POpHϑΝΠϧΛࣗಈੜ੒ w DPOpHͷ୅ΘΓʹɺ  TFSWJDFNPOJUPSΛهड़  w ϗοτϦϩʔυ w 4JEFDBSίϯςφʔͰDPOpHɺ
EBTICPBSEΛ؂ࢹɺϦϩʔυ SFMPBEFS QSPNFUIFVTIUUQ 6QEBUF 3FMPBE 8BUDI 1SPNFUIFVT1PE 3FMPBE $POpH QSPNFUIFVTPQFSBUPS

"MFSUNBOHFS w /PEF@)JHI$166TBHF w /PEF@)JHI.FNPSZ6TBHF w 1PE@)JHI.FNPSZ6TBHF w )UUQ3FRVFTU@)JHI&SSPS3BUF w
)UUQ3FRVFTU@)JHI-BUFODZ w (31$@)JHI&SSPS3BUF w 3FEJT@)JHI&SSP3BUF w ʜ "-&35(31$@)JHI&SSPS3BUF *'NFUIPEBCFNB@HSQD@SFRVFTU@JOUFSOBM@FSSPS@QFSDFOUSBUFN '03N -"#&-4\ TFWFSJUZXBSOJOH ^ "//05"5*0/4\ EFTDSJQUJPO\\MBCFMTTFSWJDF^^@\\MBCFMTNFUIPE^^@IBTBOFSSPSSBUF  PG\\QSJOUGaGaWBMVF^^GPSN ^ HSQDSVMFT TMBDL΁௨஌

໰୊ w LTͷඞཁͳZBNMϑΝΠϧͷॏෳ w ؀ڥ EFW QSE w QSPNFUIFVTTIBSE
w BQQMZॱ൪ w QFSTJTUFOUEJTLDPOpHNBQTUBUFGVMTFUTFSWJDF w EFQMPZNFOUҎ֎ͷSPMMCBDL %3: %0/`53&1&"5:0634&-'

)FMN w ,VCFSOFUFTQBDLBHFNBOBHFS w )FMN$IBSUΛ࡞੒ w ඞཁͳϑΝΠϧͷςϯϓϨʔτͷఆٛ w )FMNͰΠϯετϧ w
WBMVFTZBNMϑΝΠϧͰม਺Λઃఆ helm install --name prometheus-grpc -f dev-values.yaml ./charts/prometheus

⁠τϥϑΟοΫͷϦΞϧλΠϜՄࢹԽ

໰୊ ΫϥελશମͷτϥϑΟΫεঢ়ଶͷ೺Ѳ͕͠ʹ͍͘

1SPNFUIFVTσʔλ͔ΒτϥϑΟοΫՄࢹԽ͍ͨ͠ 1SPNFUIFVTʹͨ͘͞Μͷσʔλ͕͋ΔͷͰ

"CFNB57Ͱ͸ w 3FBMUJNF w 3FQMBZ w /PUJDFT w 'JMUFSJOH w
(SPVQJOH

1SPNWJ[ IUUQTHJUIVCDPN OHIJBMWQSPNWJ[ IUUQTHJUIVCDPN NKIEEFWMJPOQSPNWJ[GSPOU

1SPNWJ[ w 1SPNWJ[ w 1SPNFUIFVT͔ΒͷσʔλΛऔಘɺ  άϥϑσʔλΛੜ੒ w (SBQI"1*Λఏڙ w 1SPNWJ[GSPOU
w /FUqJYͷWJ[DFSBMΛج͍ͮͯɺ  τϥϑΟοΫͷϨϯμϦϯά promviz Retrieval API Cache Storage HDD/SSD promviz-front Prometheus Prometheus Prometheus

⁡ࠓޙͷ͜ͱ

ࠓޙͷ͜ͱ w 1SPNFUIFVT w (SBGBOBͱ1SPNFUIFVTؒͷϓϩΩγαʔό w Ͳͷ1SPNFUIFVTʹඞཁͳσʔλ͕͋Δ͔Ͳ͏͔  (SBGBOBଆͰ೺Ѳͷඞཁ͕ͳ͘ͳΔ

5IBOL:PV

8F`SFIJSJOH IUUQTSFDSVJUBCFNBUWDPKQ IUUQTXXXXBOUFEMZDPNDPNQBOJFTBCFNB

Monitoring at AbemaTV

Monitoring at AbemaTV

More Decks by nghialv

Other Decks in Programming

Featured

Transcript