Monitoring at AbemaTV - Speaker Deck

Monitoring at AbemaTV

by nghialv

Slide 1

Slide 1 text

"CFNB57ʹ͓͚ΔϞχλϦϯά #BDLFOE&OHJOFFS -&7"//()*"

Slide 2

Slide 2 text

"CPVUNF w -&7"//()*" ΪΞʣ w ϕτφϜग़਎ w (JUIVC!OHIJBMW w 5XJUUFS!OHIJBMW w ೥৽ଔͱͯ͠$ZCFS"HFOUʹೖࣾ w "NFCB 'SFTIੜ์ૹʂͰJ04ΤϯδχΞ w ͔Β"CFNB57ͷόοΫΤϯυΤϯδχΞ

Slide 3

Slide 3 text

"HFOEB w "CFNB57ͷΞʔΩςΫνϟ w ϞχλϦϯάγεςϜͷਐԽ w τϥϑΟοΫͷϦΞϧλΠϜՄࢹԽ w ࠓޙͷ͜ͱ w ⁞ w   w ⁠ w ⁡

Slide 4

Slide 4 text

⁞"CFNB57ͷΞʔΩςΫνϟ

Slide 5

Slide 5 text

($1,VCFSOFUFTH31$ (PPHMF$MPVE1MBUGPSN ,VCFSOFUFT .JDSPTFSWJDF .JDSPTFSWJDF .JDSPTFSWJDF H31$ H31$

Slide 6

Slide 6 text

αʔϏε͔ΒݟΔ

Slide 7

Slide 7 text

 ϞχλϦϯάγεςϜͷਐԽ

Slide 8

Slide 8 text

4UBDLESJWFSͷΈͷ࣌ظ w ຊ։ہ͔Β൒೥͙Β͍ w ($&Πϯελϯε w ϚγϯϝτϦΫε $16 .FNPSZ /FUXPSLʜ w 7.্ʹಈ͍͍ͯΔ.POHP%# 3FEJTͷϝτϦΫε w ($1αʔϏε w 1VC4VC 4UPSBHF -PBE#BMBODJOH ʜ 4UBDLESJWFS"HFOU .POHP%# 7. NPOHPECWN

Slide 9

Slide 9 text

؂ࢹ͍ͨ͠ϨΠϠ )PTU 0SDIFTUSBUJPO $POUBJOFS "QQMJDBUJPO

Slide 10

Slide 10 text

"CFNB57༻ͷϞχλϦϯάγεςϜ͕ඞཁ

Slide 11

Slide 11 text

1SPNFUIFVT(SBGBOB

Slide 12

Slide 12 text

Կނ1SPNFUIFVT w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར

Slide 13

Slide 13 text

Կނ1SPNFUIFVT w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔ 1SPNFUIFVT BQJTFSWFS 1PE 1PE 1PE TFSWJDFEJTDPWFSZ TDSBQFNFUSJDT .BTUFSOPEF 8PSLFSOPEF

Slide 14

Slide 14 text

Կނ1SPNFUIFVT 1SPNFUIFVT BQJTFSWFS 1PE 1PE 1PE TFSWJDFEJTDPWFSZ TDSBQFNFUSJDT .BTUFSOPEF 8PSLFSOPEF w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔

Slide 15

Slide 15 text

Կނ1SPNFUIFVT 1SPNFUIFVT BQJTFSWFS 1PE 1PE TFSWJDFEJTDPWFSZ TDSBQFNFUSJDT .BTUFSOPEF 8PSLFSOPEF 1PE w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔

Slide 16

Slide 16 text

Կނ1SPNFUIFVT 1SPNFUIFVT BQJTFSWFS 1PE 1PE TFSWJDFEJTDPWFSZ TDSBQFNFUSJDT .BTUFSOPEF 8PSLFSOPEF 1PE 1PE w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Կނ1SPNFUIFVT w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔ ύϑΥʔϚϯεͷ໘ JOTUBODFLTBNQMFTTFDPOEɹ 4PVOE$MPVEͷ࣮੷ UJNFTFSJFTQFSUBSHFU TTDBQFJOUFSWBM JOTUBODF͕ UBSHFUT·ͰରԠͰ͖Δ

Slide 19

Slide 19 text

Կނ1SPNFUIFVT σʔλྔͷ໘ NJMMJPOBDUJWFUJNFTFSJFT TTDSBQFJOUFSWBM NPOUIPGSFUFOUJPO ѹॖͳ͠ CZUFUJNFTUBNQCZUFWBMVF‎5# 1SPNFUIFVTͷWBSCJUΤϯίʔυ CZUFTBNQMF‎5# w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔

Slide 20

Slide 20 text

Կނ1SPNFUIFVT w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔

Slide 21

Slide 21 text

Կނ1SPNFUIFVT w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔ w QSPNFUIFVTDMJFOU@HPMBOH w $PVOUFS w (BVHF w 4VNNBSZ w )JTUPHSBN w طଘͷ&YQPSUFS΋ଟ͍ w OPEFFYQPSUFS w NPOHPEC SFEJT NZTRM ʜ

Slide 22

Slide 22 text

Կނ1SPNFUIFVT w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔ w 1SPNFUIFVT͕ൃੜͨ͠Ξϥʔτͷ؅ཧ w ॏෳ͍ͯ͠ΔΞϥʔτͷղআ w ΞϥʔτͷάϧʔϓԽ w ௨஌ͷҰ࣌తࢭͳఀࢭ w ৭ʑͳ௨৴ઌͷରԠ w 4MBDL &NBJM )5518FCIPPL

Slide 23

Slide 23 text

IUUQTEFWFMPQFSTDZCFSBHFOUDPKQCMPHBSDIJWFT #MPHl"CFNB57ʹ1SPNFUIFVTͱ͍͏ϞχλϦϯάγεςϜΛಋೖͨ͠࿩z

Slide 24

Slide 24 text

1SPNFUIFVTͷಋೖͷॳ࣌ظ w Ϋϥελ಺ʹͭͷ1SPNFUIFVTαʔό w ΞϓϦέʔγϣϯͷϝτϦΫεͷΈ w 1SPNFUIFVTͷ$POpHΛ  $POpH.BQʹॻ͘ w λʔήοταʔϏεͷTFSWJDFZBNMʹ BOOPUBUJPO௥Ճ (SBGBOB "MFSUNBOBHFS 1SPNFUIFVT (BUFXBZ &YQPSUFS .FEJBQSPYZ &YQPSUFS 6TFS &YQPSUFS $PNNFOU &YQPSUFS 8BUDINBO &YQPSUFS ʜ $POpH.BQ FUDE apiVersion: v1 kind: Service metadata: name: abema-demo annotations: prometheus.io/scrape: "true" prometheus.io/port: "12345" prometheus.io/path: “/metrics"

Slide 25

Slide 25 text

ϝτϦΫε w H31$ w SFRVFTU FSSPS EVSBUJPO w IUUQ w SFRVFTU FSSPS EVSBUJPO w HPMBOH w HPSPVUJOF IFBQ TUBDL HBSCBHFDPMMFDUJPO w QSPDFTT w DQVUJNF NFNPSZTJ[F

Slide 26

Slide 26 text

4MPXRVFSZ໰୊ w (SBGBOBଆͰෳࡶͳ2VFSZΛॻ͘ͱɺ  ଴͕ͪ࣌ؒ௕͍ w 1SPNFUIFVTͰ3FDPSEJOH3VMFTΛ௥Ճ w ࣄલܭࢉΛߦ͍ɺ  ৽͍͠UJNFTFSJFTͱͯ͠อଘ 100 * sum by (service) ( rate(abema_http_requests_total{status=~"^5..$"}[2m]) ) / sum by (service) ( rate(abema_http_requests_total[2m]) ) (SBGBOB service:abema_http_request_5xx_percent:rate2m = 100 * sum by (service) ( rate(abema_http_requests_total{status=~"^5..$"}[2m]) ) / sum by (service) ( rate(abema_http_requests_total[2m]) ) 1SPNFUIFVT service:abema_http_request_5xx_percent:rate2m (SBGBOB

Slide 27

Slide 27 text

)BQQZ

Slide 28

Slide 28 text

਺ϲ݄ޙ

Slide 29

Slide 29 text

໰୊ɾ՝୊ w ࣌ܥྻσʔλྔͷ૿Ճ w )JHI"WBJMBCJMJUZ1SPNFUIFVTͷඞཁ w ӡ༻ͷେม

Slide 30

Slide 30 text

࣌ܥྻσʔλྔͷ૿Ճ w ϞχλϦϯάλʔήοτͷ૿Ճ w αʔόʔ୆਺ w $POUBJOFS਺ w .JDSPTFSWJDFT਺ w ৽͍͠ϝτϦΫεͷ૿Ճ

Slide 31

Slide 31 text

৽͍͠ϝτϦΫεͷྫ

Slide 32

Slide 32 text

/PEFϝτϦΫε w /PEFFYQPSUFS w $16 .FNPSZ %JTL /FUXPSL ʜ w ,TOPEF w %BFNPO4FUͱ֤ͯ͠OPEFʹ  ҰͭͷQPEΛಈ͔͢ w ($&OPEF w 1BDLFSͰNBDIJOFJNBHFͷ  Ϗϧυ࣌ʹೖΕΔ OPEFEBTICPBSE

Slide 33

Slide 33 text

LTͷϝτϦΫε w BQJTFSWFS w LVCFMFU w LTTUBUFNFUSJDT w LVCFEOT QPEEBTICPBSE

Slide 34

Slide 34 text

$$6ϝτϦΫε w ಉ࣌઀ଓ਺ͷϝτϦΫε w ֤$IBOOFM 04 4USFBNJOH1SPUPDPM   3FTPMVUJPO 1MBZFS ʜ w ΞΫηεϩά͔ΒϦΞϧλΠϜܭࢉ w )ZQFS-PH-PHΞϧΰϦζϜ w )PSJ[POUBM1PE"VUPTDBMFSͰ  DDVDPMMFDUPSΛࣗಈεέʔϦϯά .FEJBQSPYZ 1VC4VC DDVDPMMFDUPS DDVFYQPSUFS DDVSFEJT 1SPNFUIFVT DDVDPMMFDUPS .FEJBQSPYZ MPH #JH2VFSZ

Slide 35

Slide 35 text

4UBDLESJWFSͷϝτϦΫε w (PPHMFͷNPOJUPSJOHBQJΛ༻͍ͯಠࣗͷFYQPSUFSΛ࡞੒ w -PBE#BMBODJOH w 4UPSBHF w 1VC4VC w #JH2VFSZ w %BUBqPX TUBDLESJWFSFYQPSUFS HPPHMFNPOJUPSJOHBQJ 1SPNFUIFVT

Slide 36

Slide 36 text

Ұͭͷ1SPNFUIFVTͰ͸ݫ͍͠ ࣌ܥྻσʔλྔͷ૿ՃͷͨΊ

Slide 37

Slide 37 text

1SPNFUIFVTTIBSEJOH w ϝτϦΫελΠϓ w ϝτϦΫεछྨͰ1SPNFUIFVTΛ෼͚Δ w λʔήοτ w औಘର৅Ͱ1SPNFUIFVTΛ෼͚Δ

Slide 38

Slide 38 text

1SPNFUIFVTTIBSEJOH w ϝτϦΫελΠϓ w ϝτϦΫεछྨͰ1SPNFUIFVTΛ෼͚Δ w λʔήοτ w औಘର৅Ͱ1SPNFUIFVTΛ෼͚Δ ˔ QSPNFUIFVT TFSWJDF" TFSWJDF# NFUSJDT NFUSJDT OPEF FYQPSUFS NFUSJDT

Slide 39

Slide 39 text

1SPNFUIFVTTIBSEJOH w ϝτϦΫελΠϓ w ϝτϦΫεछྨͰ1SPNFUIFVTΛ෼͚Δ w λʔήοτ w औಘର৅Ͱ1SPNFUIFVTΛ෼͚Δ ˔ OPEF FYQPSUFS OPEFNFUSJDT TFSWJDF" HPNFUSJDT IUUQNFUSJDT TFSWJDF# HPNFUSJDT HSQDNFUSJDT QSPNFUIFVT

Slide 40

Slide 40 text

Slide 41

Slide 41 text

1SPNFUIFVTTIBSEJOH w ϝτϦΫελΠϓ w ϝτϦΫεछྨͰ1SPNFUIFVTΛ෼͚Δ w λʔήοτ w औಘର৅Ͱ1SPNFUIFVTΛ෼͚Δ ˔ QSPNFUIFVT TFSWJDF" NFUSJDT TFSWJDF# NFUSJDT TFSWJDF$ NFUSJDT TFSWJDF% NFUSJDT

Slide 42

Slide 42 text

1SPNFUIFVTTIBSEJOH TFSWJDF" NFUSJDT TFSWJDF# NFUSJDT TFSWJDF$ NFUSJDT TFSWJDF% NFUSJDT QSPNFUIFVT QSPNFUIFVT w ϝτϦΫελΠϓ w ϝτϦΫεछྨͰ1SPNFUIFVTΛ෼͚Δ w λʔήοτ w औಘର৅Ͱ1SPNFUIFVTΛ෼͚Δ ˔

Slide 43

Slide 43 text

)"1SPNFUIFVT w ܦҢ w 1SPNFUIFVTͷϝτϦΫε͔ΒNJDSPTFSWJDFTΛࣗಈεέʔϦϯά w ಉ͡$POpHɺಉ͡λʔήοτϦετͰෳ਺1SPNFUIFVTΛߏங w NFNPSZ΍EJTLͷσʔλ͸΄΅ಉ͡ w σʔλͷࠩ෼ͷٵऩ w Ξϥʔτ w ීஈͷΞϥʔτ͸Ұఆͷ࣌ؒͰ൑ఆ w (SBGBOB w εςΟοΩʔηογϣϯ TFSWJDF" NFUSJDT TFSWJDF# NFUSJDT QSPNFUIFVT QSPNFUIFVT QSPNFUIFVT

Slide 44

Slide 44 text

ӡ༻ w $POpHϑΝΠϧΛॻ͘ͷ͕େม w TIBSEJOHͰྔ͕ଟ͘ͳΓɺઃఆ͕ෳࡶ w SFMBCFMJOH͕෼͔Γʹ͍͘ w ৽ϝτϦΫεͷ௥Ճ΍EBTICPBSETͷमਖ਼͸  νʔϜϝϯόʔͦΕͧΕͰ΍Γ͍ͨ w DPOpH΍EBTICPBSEΛߋ৽͍ͨ͠৔߹ɺ  Ϧελʔτ͕ඞཁ OPEF FYQPSUFS OPEFNFUSJDT TFSWJDF" HPNFUSJDT IUUQNFUSJDT TFSWJDF# HPNFUSJDT HSQDNFUSJDT QSPNFUIFVTHP QSPNFUIFVTHSQD QSPNFUIFVTIUUQ QSPNFUIFVTOPEF

Slide 45

Slide 45 text

1SPNFUIFVT0QFSBUPS

Slide 46

Slide 46 text

1SPNFUIFVT0QFSBUPS w $POpHϑΝΠϧΛࣗಈੜ੒ w DPOpHͷ୅ΘΓʹɺ  TFSWJDFNPOJUPSΛهड़  w ϗοτϦϩʔυ w 4JEFDBSίϯςφʔͰDPOpHɺ EBTICPBSEΛ؂ࢹɺϦϩʔυ TFSWJDF" MBCFMT UZQFIUUQ TFSWJDF# MBCFMT UZQFIUUQ QSPNFUIFVTIUUQ TFSWJDFNPOJUPS QSPNUIFVTIUUQ TFMFDUPS NBUDI-BCFMT UZQFIUUQ $POpH 8BUDI (FOFSBUF QSPNFUIFVTPQFSBUPS

Slide 47

Slide 47 text

1SPNFUIFVT0QFSBUPS w $POpHϑΝΠϧΛࣗಈੜ੒ w DPOpHͷ୅ΘΓʹɺ  TFSWJDFNPOJUPSΛهड़  w ϗοτϦϩʔυ w 4JEFDBSίϯςφʔͰDPOpHɺ EBTICPBSEΛ؂ࢹɺϦϩʔυ SFMPBEFS QSPNFUIFVTIUUQ 6QEBUF 3FMPBE 8BUDI 1SPNFUIFVT1PE 3FMPBE $POpH QSPNFUIFVTPQFSBUPS

Slide 48

Slide 48 text

"MFSUNBOHFS w /PEF@)JHI$166TBHF w /PEF@)JHI.FNPSZ6TBHF w 1PE@)JHI.FNPSZ6TBHF w )UUQ3FRVFTU@)JHI&SSPS3BUF w )UUQ3FRVFTU@)JHI-BUFODZ w (31$@)JHI&SSPS3BUF w 3FEJT@)JHI&SSP3BUF w ʜ "-&35(31$@)JHI&SSPS3BUF *'NFUIPEBCFNB@HSQD@SFRVFTU@JOUFSOBM@FSSPS@QFSDFOUSBUFN '03N -"#&-4\ TFWFSJUZXBSOJOH ^ "//05"5*0/4\ EFTDSJQUJPO\\MBCFMTTFSWJDF^^@\\MBCFMTNFUIPE^^@IBTBOFSSPSSBUF  PG\\QSJOUGaGaWBMVF^^GPSN ^ HSQDSVMFT TMBDL΁௨஌

Slide 49

Slide 49 text

)BQQZ

Slide 50

Slide 50 text

໰୊ w LTͷඞཁͳZBNMϑΝΠϧͷॏෳ w ؀ڥ EFW QSE w QSPNFUIFVTTIBSE w BQQMZॱ൪ w QFSTJTUFOUEJTLDPOpHNBQTUBUFGVMTFUTFSWJDF w EFQMPZNFOUҎ֎ͷSPMMCBDL %3: %0/`53&1&"5:0634&-'

Slide 51

Slide 51 text

)FMN w ,VCFSOFUFTQBDLBHFNBOBHFS w )FMN$IBSUΛ࡞੒ w ඞཁͳϑΝΠϧͷςϯϓϨʔτͷఆٛ w )FMNͰΠϯετϧ w WBMVFTZBNMϑΝΠϧͰม਺Λઃఆ helm install --name prometheus-grpc -f dev-values.yaml ./charts/prometheus

Slide 52

Slide 52 text

)BQQZ

Slide 53

Slide 53 text

⁠τϥϑΟοΫͷϦΞϧλΠϜՄࢹԽ

Slide 54

Slide 54 text

໰୊ ΫϥελશମͷτϥϑΟΫεঢ়ଶͷ೺Ѳ͕͠ʹ͍͘

Slide 55

Slide 55 text

1SPNFUIFVTσʔλ͔ΒτϥϑΟοΫՄࢹԽ͍ͨ͠ 1SPNFUIFVTʹͨ͘͞Μͷσʔλ͕͋ΔͷͰ

Slide 56

Slide 56 text

"CFNB57Ͱ͸ w 3FBMUJNF w 3FQMBZ w /PUJDFT w 'JMUFSJOH w (SPVQJOH

Slide 57

Slide 57 text

1SPNWJ[ IUUQTHJUIVCDPN OHIJBMWQSPNWJ[ IUUQTHJUIVCDPN NKIEEFWMJPOQSPNWJ[GSPOU

Slide 58

Slide 58 text

1SPNWJ[ w 1SPNWJ[ w 1SPNFUIFVT͔ΒͷσʔλΛऔಘɺ  άϥϑσʔλΛੜ੒ w (SBQI"1*Λఏڙ w 1SPNWJ[GSPOU w /FUqJYͷWJ[DFSBMΛج͍ͮͯɺ  τϥϑΟοΫͷϨϯμϦϯά promviz Retrieval API Cache Storage HDD/SSD promviz-front Prometheus Prometheus Prometheus

Slide 59

Slide 59 text

⁡ࠓޙͷ͜ͱ

Slide 60

Slide 60 text

ࠓޙͷ͜ͱ w 1SPNFUIFVT w (SBGBOBͱ1SPNFUIFVTؒͷϓϩΩγαʔό w Ͳͷ1SPNFUIFVTʹඞཁͳσʔλ͕͋Δ͔Ͳ͏͔  (SBGBOBଆͰ೺Ѳͷඞཁ͕ͳ͘ͳΔ

Slide 61

Slide 61 text

5IBOL:PV

Slide 62

Slide 62 text

8F`SFIJSJOH IUUQTSFDSVJUBCFNBUWDPKQ IUUQTXXXXBOUFEMZDPNDPNQBOJFTBCFNB