Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Monitoring at AbemaTV

439d941a6e41920af9bf4d5bd5f97e3a?s=47 nghialv
October 21, 2017

Monitoring at AbemaTV

439d941a6e41920af9bf4d5bd5f97e3a?s=128

nghialv

October 21, 2017
Tweet

Transcript

  1. "CFNB57ʹ͓͚ΔϞχλϦϯά #BDLFOE&OHJOFFS -&7"//()*"

  2. "CPVUNF w -&7"//()*" ΪΞʣ w ϕτφϜग़਎ w (JUIVC!OHIJBMW w 5XJUUFS!OHIJBMW

    w ೥৽ଔͱͯ͠$ZCFS"HFOUʹೖࣾ w "NFCB 'SFTIੜ์ૹʂͰJ04ΤϯδχΞ w ͔Β"CFNB57ͷόοΫΤϯυΤϯδχΞ
  3. "HFOEB w "CFNB57ͷΞʔΩςΫνϟ w ϞχλϦϯάγεςϜͷਐԽ w τϥϑΟοΫͷϦΞϧλΠϜՄࢹԽ w ࠓޙͷ͜ͱ w

    ⁞ w   w ⁠ w ⁡
  4. ⁞"CFNB57ͷΞʔΩςΫνϟ

  5. ($1 ,VCFSOFUFT H31$ (PPHMF$MPVE1MBUGPSN ,VCFSOFUFT .JDSPTFSWJDF .JDSPTFSWJDF .JDSPTFSWJDF H31$ H31$

  6. αʔϏε͔ΒݟΔ

  7.  ϞχλϦϯάγεςϜͷਐԽ

  8. 4UBDLESJWFSͷΈͷ࣌ظ w ຊ։ہ͔Β൒೥͙Β͍ w ($&Πϯελϯε w ϚγϯϝτϦΫε $16 .FNPSZ /FUXPSLʜ

     w 7.্ʹಈ͍͍ͯΔ.POHP%# 3FEJTͷϝτϦΫε w ($1αʔϏε w 1VC4VC 4UPSBHF -PBE#BMBODJOH ʜ 4UBDLESJWFS"HFOU .POHP%# 7. NPOHPECWN
  9. ؂ࢹ͍ͨ͠ϨΠϠ )PTU 0SDIFTUSBUJPO $POUBJOFS "QQMJDBUJPO

  10. "CFNB57༻ͷϞχλϦϯάγεςϜ͕ඞཁ

  11. 1SPNFUIFVT (SBGBOB

  12. Կނ1SPNFUIFVT w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w

    "MFSUNBOBHFS͕ศར
  13. Կނ1SPNFUIFVT w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w

    "MFSUNBOBHFS͕ศར ˔ 1SPNFUIFVT BQJTFSWFS 1PE 1PE 1PE TFSWJDFEJTDPWFSZ TDSBQFNFUSJDT .BTUFSOPEF 8PSLFSOPEF
  14. Կނ1SPNFUIFVT 1SPNFUIFVT BQJTFSWFS 1PE 1PE 1PE TFSWJDFEJTDPWFSZ TDSBQFNFUSJDT .BTUFSOPEF 8PSLFSOPEF

    w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔
  15. Կނ1SPNFUIFVT 1SPNFUIFVT BQJTFSWFS 1PE 1PE TFSWJDFEJTDPWFSZ TDSBQFNFUSJDT .BTUFSOPEF 8PSLFSOPEF 1PE

    w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔
  16. Կނ1SPNFUIFVT 1SPNFUIFVT BQJTFSWFS 1PE 1PE TFSWJDFEJTDPWFSZ TDSBQFNFUSJDT .BTUFSOPEF 8PSLFSOPEF 1PE

    1PE w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔
  17. Կނ1SPNFUIFVT 1SPNFUIFVT BQJTFSWFS 1PE 1PE TFSWJDFEJTDPWFSZ TDSBQFNFUSJDT .BTUFSOPEF 8PSLFSOPEF 1PE

    1PE w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔
  18. Կނ1SPNFUIFVT w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w

    "MFSUNBOBHFS͕ศར ˔ ύϑΥʔϚϯεͷ໘ JOTUBODFLTBNQMFTTFDPOEɹ 4PVOE$MPVEͷ࣮੷ UJNFTFSJFTQFSUBSHFU TTDBQFJOUFSWBM JOTUBODF͕ UBSHFUT·ͰରԠͰ͖Δ
  19. Կނ1SPNFUIFVT σʔλྔͷ໘ NJMMJPOBDUJWFUJNFTFSJFT TTDSBQFJOUFSWBM NPOUIPGSFUFOUJPO ѹॖͳ͠ CZUFUJNFTUBNQ CZUFWBMVF‎5# 1SPNFUIFVTͷWBSCJUΤϯίʔυ CZUFTBNQMF‎5#

    w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔
  20. Կނ1SPNFUIFVT w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w

    "MFSUNBOBHFS͕ศར ˔
  21. Կނ1SPNFUIFVT w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w

    "MFSUNBOBHFS͕ศར ˔ w QSPNFUIFVTDMJFOU@HPMBOH w $PVOUFS w (BVHF w 4VNNBSZ w )JTUPHSBN w طଘͷ&YQPSUFS΋ଟ͍ w OPEFFYQPSUFS w NPOHPEC SFEJT NZTRM ʜ
  22. Կނ1SPNFUIFVT w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w

    "MFSUNBOBHFS͕ศར ˔ w 1SPNFUIFVT͕ൃੜͨ͠Ξϥʔτͷ؅ཧ w ॏෳ͍ͯ͠ΔΞϥʔτͷղআ w ΞϥʔτͷάϧʔϓԽ w ௨஌ͷҰ࣌తࢭͳఀࢭ w ৭ʑͳ௨৴ઌͷରԠ w 4MBDL &NBJM )5518FCIPPL
  23. IUUQTEFWFMPQFSTDZCFSBHFOUDPKQCMPHBSDIJWFT #MPHl"CFNB57ʹ1SPNFUIFVTͱ͍͏ϞχλϦϯάγεςϜΛಋೖͨ͠࿩z

  24. 1SPNFUIFVTͷಋೖͷॳ࣌ظ w Ϋϥελ಺ʹͭͷ1SPNFUIFVTαʔό w ΞϓϦέʔγϣϯͷϝτϦΫεͷΈ w 1SPNFUIFVTͷ$POpHΛ
 $POpH.BQʹॻ͘ w λʔήοταʔϏεͷTFSWJDFZBNMʹ

    BOOPUBUJPO௥Ճ (SBGBOB "MFSUNBOBHFS 1SPNFUIFVT (BUFXBZ &YQPSUFS .FEJBQSPYZ &YQPSUFS 6TFS &YQPSUFS $PNNFOU &YQPSUFS 8BUDINBO &YQPSUFS ʜ $POpH.BQ FUDE apiVersion: v1 kind: Service metadata: name: abema-demo annotations: prometheus.io/scrape: "true" prometheus.io/port: "12345" prometheus.io/path: “/metrics"
  25. ϝτϦΫε w H31$ w SFRVFTU FSSPS EVSBUJPO w IUUQ w

    SFRVFTU FSSPS EVSBUJPO w HPMBOH w HPSPVUJOF IFBQ TUBDL HBSCBHFDPMMFDUJPO w QSPDFTT w DQVUJNF NFNPSZTJ[F
  26. 4MPXRVFSZ໰୊ w (SBGBOBଆͰෳࡶͳ2VFSZΛॻ͘ͱɺ
 ଴͕ͪ࣌ؒ௕͍ w 1SPNFUIFVTͰ3FDPSEJOH3VMFTΛ௥Ճ w ࣄલܭࢉΛߦ͍ɺ
 ৽͍͠UJNFTFSJFTͱͯ͠อଘ 100

    * sum by (service) ( rate(abema_http_requests_total{status=~"^5..$"}[2m]) ) / sum by (service) ( rate(abema_http_requests_total[2m]) ) (SBGBOB service:abema_http_request_5xx_percent:rate2m = 100 * sum by (service) ( rate(abema_http_requests_total{status=~"^5..$"}[2m]) ) / sum by (service) ( rate(abema_http_requests_total[2m]) ) 1SPNFUIFVT service:abema_http_request_5xx_percent:rate2m (SBGBOB
  27. )BQQZ

  28. ਺ϲ݄ޙ

  29. ໰୊ɾ՝୊ w ࣌ܥྻσʔλྔͷ૿Ճ w )JHI"WBJMBCJMJUZ1SPNFUIFVTͷඞཁ w ӡ༻ͷେม

  30. ࣌ܥྻσʔλྔͷ૿Ճ w ϞχλϦϯάλʔήοτͷ૿Ճ w αʔόʔ୆਺ w $POUBJOFS਺ w .JDSPTFSWJDFT਺ w

    ৽͍͠ϝτϦΫεͷ૿Ճ
  31. ৽͍͠ϝτϦΫεͷྫ

  32. /PEFϝτϦΫε w /PEFFYQPSUFS w $16 .FNPSZ %JTL /FUXPSL ʜ w

    ,TOPEF w %BFNPO4FUͱ֤ͯ͠OPEFʹ
 ҰͭͷQPEΛಈ͔͢ w ($&OPEF w 1BDLFSͰNBDIJOFJNBHFͷ
 Ϗϧυ࣌ʹೖΕΔ OPEFEBTICPBSE
  33. LTͷϝτϦΫε w BQJTFSWFS w LVCFMFU w LTTUBUFNFUSJDT w LVCFEOT QPEEBTICPBSE

  34. $$6ϝτϦΫε w ಉ࣌઀ଓ਺ͷϝτϦΫε w ֤$IBOOFM 04 4USFBNJOH1SPUPDPM 
 3FTPMVUJPO 1MBZFS

    ʜ w ΞΫηεϩά͔ΒϦΞϧλΠϜܭࢉ w )ZQFS-PH-PHΞϧΰϦζϜ w )PSJ[POUBM1PE"VUPTDBMFSͰ
 DDVDPMMFDUPSΛࣗಈεέʔϦϯά .FEJBQSPYZ 1VC4VC DDVDPMMFDUPS DDVFYQPSUFS DDVSFEJT 1SPNFUIFVT DDVDPMMFDUPS .FEJBQSPYZ MPH #JH2VFSZ
  35. 4UBDLESJWFSͷϝτϦΫε w (PPHMFͷNPOJUPSJOHBQJΛ༻͍ͯಠࣗͷFYQPSUFSΛ࡞੒ w -PBE#BMBODJOH w 4UPSBHF w 1VC4VC w

    #JH2VFSZ w %BUBqPX TUBDLESJWFSFYQPSUFS HPPHMFNPOJUPSJOHBQJ 1SPNFUIFVT
  36. Ұͭͷ1SPNFUIFVTͰ͸ݫ͍͠ ࣌ܥྻσʔλྔͷ૿ՃͷͨΊ

  37. 1SPNFUIFVTTIBSEJOH w ϝτϦΫελΠϓ w ϝτϦΫεछྨͰ1SPNFUIFVTΛ෼͚Δ w λʔήοτ w औಘର৅Ͱ1SPNFUIFVTΛ෼͚Δ

  38. 1SPNFUIFVTTIBSEJOH w ϝτϦΫελΠϓ w ϝτϦΫεछྨͰ1SPNFUIFVTΛ෼͚Δ w λʔήοτ w औಘର৅Ͱ1SPNFUIFVTΛ෼͚Δ ˔

    QSPNFUIFVT TFSWJDF" TFSWJDF# NFUSJDT NFUSJDT OPEF FYQPSUFS NFUSJDT
  39. 1SPNFUIFVTTIBSEJOH w ϝτϦΫελΠϓ w ϝτϦΫεछྨͰ1SPNFUIFVTΛ෼͚Δ w λʔήοτ w औಘର৅Ͱ1SPNFUIFVTΛ෼͚Δ ˔

    OPEF FYQPSUFS OPEFNFUSJDT TFSWJDF" HPNFUSJDT IUUQNFUSJDT TFSWJDF# HPNFUSJDT HSQDNFUSJDT QSPNFUIFVT
  40. 1SPNFUIFVTTIBSEJOH w ϝτϦΫελΠϓ w ϝτϦΫεछྨͰ1SPNFUIFVTΛ෼͚Δ w λʔήοτ w औಘର৅Ͱ1SPNFUIFVTΛ෼͚Δ ˔

    OPEF FYQPSUFS OPEFNFUSJDT TFSWJDF" HPNFUSJDT IUUQNFUSJDT TFSWJDF# HPNFUSJDT HSQDNFUSJDT QSPNFUIFVTHP QSPNFUIFVTHSQD QSPNFUIFVTIUUQ QSPNFUIFVTOPEF
  41. 1SPNFUIFVTTIBSEJOH w ϝτϦΫελΠϓ w ϝτϦΫεछྨͰ1SPNFUIFVTΛ෼͚Δ w λʔήοτ w औಘର৅Ͱ1SPNFUIFVTΛ෼͚Δ ˔

    QSPNFUIFVT TFSWJDF" NFUSJDT TFSWJDF# NFUSJDT TFSWJDF$ NFUSJDT TFSWJDF% NFUSJDT
  42. 1SPNFUIFVTTIBSEJOH TFSWJDF" NFUSJDT TFSWJDF# NFUSJDT TFSWJDF$ NFUSJDT TFSWJDF% NFUSJDT QSPNFUIFVT

    QSPNFUIFVT w ϝτϦΫελΠϓ w ϝτϦΫεछྨͰ1SPNFUIFVTΛ෼͚Δ w λʔήοτ w औಘର৅Ͱ1SPNFUIFVTΛ෼͚Δ ˔
  43. )"1SPNFUIFVT w ܦҢ w 1SPNFUIFVTͷϝτϦΫε͔ΒNJDSPTFSWJDFTΛࣗಈεέʔϦϯά w ಉ͡$POpHɺಉ͡λʔήοτϦετͰෳ਺1SPNFUIFVTΛߏங w NFNPSZ΍EJTLͷσʔλ͸΄΅ಉ͡ w

    σʔλͷࠩ෼ͷٵऩ w Ξϥʔτ w ීஈͷΞϥʔτ͸Ұఆͷ࣌ؒͰ൑ఆ w (SBGBOB w εςΟοΩʔηογϣϯ TFSWJDF" NFUSJDT TFSWJDF# NFUSJDT QSPNFUIFVT QSPNFUIFVT QSPNFUIFVT
  44. ӡ༻ w $POpHϑΝΠϧΛॻ͘ͷ͕େม w TIBSEJOHͰྔ͕ଟ͘ͳΓɺઃఆ͕ෳࡶ w SFMBCFMJOH͕෼͔Γʹ͍͘ w ৽ϝτϦΫεͷ௥Ճ΍EBTICPBSETͷमਖ਼͸
 νʔϜϝϯόʔͦΕͧΕͰ΍Γ͍ͨ

    w DPOpH΍EBTICPBSEΛߋ৽͍ͨ͠৔߹ɺ
 Ϧελʔτ͕ඞཁ OPEF FYQPSUFS OPEFNFUSJDT TFSWJDF" HPNFUSJDT IUUQNFUSJDT TFSWJDF# HPNFUSJDT HSQDNFUSJDT QSPNFUIFVTHP QSPNFUIFVTHSQD QSPNFUIFVTIUUQ QSPNFUIFVTOPEF
  45. 1SPNFUIFVT0QFSBUPS

  46. 1SPNFUIFVT0QFSBUPS w $POpHϑΝΠϧΛࣗಈੜ੒ w DPOpHͷ୅ΘΓʹɺ
 TFSWJDFNPOJUPSΛهड़
 w ϗοτϦϩʔυ w 4JEFDBSίϯςφʔͰDPOpHɺ

    EBTICPBSEΛ؂ࢹɺϦϩʔυ TFSWJDF" MBCFMT UZQFIUUQ TFSWJDF# MBCFMT UZQFIUUQ QSPNFUIFVTIUUQ TFSWJDFNPOJUPS QSPNUIFVTIUUQ TFMFDUPS NBUDI-BCFMT UZQFIUUQ $POpH 8BUDI (FOFSBUF QSPNFUIFVTPQFSBUPS
  47. 1SPNFUIFVT0QFSBUPS w $POpHϑΝΠϧΛࣗಈੜ੒ w DPOpHͷ୅ΘΓʹɺ
 TFSWJDFNPOJUPSΛهड़
 w ϗοτϦϩʔυ w 4JEFDBSίϯςφʔͰDPOpHɺ

    EBTICPBSEΛ؂ࢹɺϦϩʔυ SFMPBEFS QSPNFUIFVTIUUQ 6QEBUF 3FMPBE 8BUDI 1SPNFUIFVT1PE 3FMPBE $POpH QSPNFUIFVTPQFSBUPS
  48. "MFSUNBOHFS w /PEF@)JHI$166TBHF w /PEF@)JHI.FNPSZ6TBHF w 1PE@)JHI.FNPSZ6TBHF w )UUQ3FRVFTU@)JHI&SSPS3BUF w

    )UUQ3FRVFTU@)JHI-BUFODZ w (31$@)JHI&SSPS3BUF w 3FEJT@)JHI&SSP3BUF w ʜ "-&35(31$@)JHI&SSPS3BUF *'NFUIPEBCFNB@HSQD@SFRVFTU@JOUFSOBM@FSSPS@QFSDFOUSBUFN '03N -"#&-4\ TFWFSJUZXBSOJOH ^ "//05"5*0/4\ EFTDSJQUJPO\\MBCFMTTFSWJDF^^@\\MBCFMTNFUIPE^^@IBTBOFSSPSSBUF
 PG\\QSJOUGaGaWBMVF^^GPSN  ^ HSQDSVMFT TMBDL΁௨஌
  49. )BQQZ

  50. ໰୊ w LTͷඞཁͳZBNMϑΝΠϧͷॏෳ w ؀ڥ EFW QSE  w QSPNFUIFVTTIBSE

    w BQQMZॱ൪ w QFSTJTUFOUEJTLDPOpHNBQTUBUFGVMTFUTFSWJDF w EFQMPZNFOUҎ֎ͷSPMMCBDL %3: %0/`53&1&"5:0634&-'
  51. )FMN w ,VCFSOFUFTQBDLBHFNBOBHFS w )FMN$IBSUΛ࡞੒ w ඞཁͳϑΝΠϧͷςϯϓϨʔτͷఆٛ w )FMNͰΠϯετϧ w

    WBMVFTZBNMϑΝΠϧͰม਺Λઃఆ helm install --name prometheus-grpc -f dev-values.yaml ./charts/prometheus
  52. )BQQZ

  53. ⁠τϥϑΟοΫͷϦΞϧλΠϜՄࢹԽ

  54. ໰୊ ΫϥελશମͷτϥϑΟΫεঢ়ଶͷ೺Ѳ͕͠ʹ͍͘

  55. 1SPNFUIFVTσʔλ͔ΒτϥϑΟοΫՄࢹԽ͍ͨ͠ 1SPNFUIFVTʹͨ͘͞Μͷσʔλ͕͋ΔͷͰ

  56. "CFNB57Ͱ͸ w 3FBMUJNF w 3FQMBZ w /PUJDFT w 'JMUFSJOH w

    (SPVQJOH
  57. 1SPNWJ[ IUUQTHJUIVCDPN OHIJBMWQSPNWJ[ IUUQTHJUIVCDPN NKIEEFWMJPOQSPNWJ[GSPOU

  58. 1SPNWJ[ w 1SPNWJ[ w 1SPNFUIFVT͔ΒͷσʔλΛऔಘɺ
 άϥϑσʔλΛੜ੒ w (SBQI"1*Λఏڙ w 1SPNWJ[GSPOU

    w /FUqJYͷWJ[DFSBMΛج͍ͮͯɺ
 τϥϑΟοΫͷϨϯμϦϯά promviz Retrieval API Cache Storage HDD/SSD promviz-front Prometheus Prometheus Prometheus
  59. ⁡ࠓޙͷ͜ͱ

  60. ࠓޙͷ͜ͱ w 1SPNFUIFVT w (SBGBOBͱ1SPNFUIFVTؒͷϓϩΩγαʔό w Ͳͷ1SPNFUIFVTʹඞཁͳσʔλ͕͋Δ͔Ͳ͏͔
 (SBGBOBଆͰ೺Ѳͷඞཁ͕ͳ͘ͳΔ

  61. 5IBOL:PV

  62. 8F`SFIJSJOH IUUQTSFDSVJUBCFNBUWDPKQ IUUQTXXXXBOUFEMZDPNDPNQBOJFTBCFNB