クラウドネイティブな監視をMackerelで / Mackerel Day#2

Transcript

1. Ϋϥ΢υωΠςΟϒͳ؂ࢹΛ Mackerel Ͱ 2019.12.23 Mackerel Day #2 @fujiwara
2. @fujiwara .BDLFSFM
   Ξϯόαμʔ ʙ HJUIVCDPNLBZBDFDTQSFTTP
   "NB[PO
   &$4
   σϓϩΠπʔϧ HJUIVCDPNGVKJXBSBMBNCSPMM
   "84
   -BNCEB
   σϓϩΠπʔ ϧ
3. Game & Community
5. Ϋϥ΢υωΠςΟϒʁ $/$'
   $MPVE
   /BUJWF
   %FpOJUJPO
   W Ϋϥ΢υωΠςΟϒٕज़͸ɺύϒϦοΫΫϥ΢υɺϓϥΠϕʔτΫϥ΢υɺϋΠϒ ϦουΫϥ΢υͳͲͷۙ୅తͰμΠφϛοΫͳ؀ڥʹ͓͍ͯɺεέʔϥϒϧͳΞϓ ϦέʔγϣϯΛߏங͓Αͼ࣮ߦ͢ΔͨΊͷೳྗΛ૊৫ʹ΋ͨΒ͠·͢ɻ ͜ͷΞϓϩʔνͷ୅දྫʹɺίϯςφɺαʔϏεϝογϡɺϚΠΫϩαʔϏεɺΠ ϛϡʔλϒϧΠϯϑϥετϥΫνϟɺ͓ΑͼએݴܕAPI͕͋Γ·͢ɻ 
   IUUQTHJUIVCDPNDODGUPDCMPCNBTUFS%&'*/*5*0/NE
6. ίϯςφɺαʔϏεϝογϡɺϚΠΫϩαʔϏε… ͦΕΒͷٕज़Λ࢖͍ͬͯΔ
   㱠
   Ϋϥ΢υωΠςΟϒ "Design for Failure" ͱͦΕΛ࣮ݱ͢Δ࢓૊Έ΍ίϯϙʔωϯτɺͦΕΛ࢖͍ ͜ͳ͢͜ͱ͕ͦ͜Ϋϥ΢υωΠςΟϒ ো֐ͷൃੜʹରͯࣗ͠ಈ෮چͰ͖ΔΑ͏ʹσβΠϯ͢Δɻো֐ͷൃੜʹΑΔϢʔ βʔӨڹ͕ͦ΋ͦ΋ͳ͍Α͏ʹΞʔΩςΫνϟΛσβΠϯ͢Δ2 
   IUUQTTQFBLFSEFDLDPNUPSJDMTEFTJHOGPSGBJMVSFJTUIFUSVFDMPVEOBUJWF
7. "Design for Failure" ো֐Λ૝ఆͯ͠γεςϜΛσβΠϯ͢Δ w Πϯελϯε͸ࢮ͵ w ϚωʔδυαʔϏε΋ࢮ͵
   ͚Ͳେ఍GBJMPWFS͢Δ w σʔληϯλʔ͝ͱࢮ͵
   كʹ͋Δ Ͳ͜·ͰΛ૝ఆ͠ɺͲ͔͜Β͸ఘΊΔ͔ ؂ࢹ΋ͦΕΒΛંΓࠐΜͰઃܭ͢Δˠ
   Ϋϥ΢υωΠςΟϒͳ؂ࢹ
8. ؂ࢹର৅͸ಈతʹ૿͑ͨΓݮͬͨΓ͢Δ ίϯςφ͸σϓϩΠ͝ͱʹੜ·ΕมΘΔ ίϯςφͰͳͯ͘΋ w &$ͷΦʔτεέʔϦϯάͰ4QPUΠϯελϯεΛ࢖͏ w ΘΓͱ͙͢མͪΔɺଞͷ্͕͕Δ ϚωʔδυαʔϏε΋ෛՙʹԠͯ͡૿ݮͰ͖ΔΑ͏ʹͳ͖ͬͯͨ "VSPSB
   "VUP
   4DBMJOHͳͲ
9. Mackerel ͰΫϥ΢υωΠςΟϒͷୈҰา ʮDPOOFDUJWJUZ
   ݕ஌Λ΍ΊΔʯ
10. connec%vity ݕ஌Λ΍ΊΔ ͜ͷΞϥʔτ͸
    !
    $SJUJDBM
    ͔͠ͳ͍
    w ։ൃ؀ڥ͕ਂ໷ʹམͪͯඈͼى͖Δඞཁ͸ͳ͍ w ຊ൪؀ڥͰ୆མͪͯ΋αʔϏεʹӨڹ͢Δϗετ͸ͳ͍ Α͏ʹ࡞Δ ϗετ͕ࢮ͵͜ͱΛલఏʹσβΠϯ͢ΔʹΫϥ΢υωΠςΟϒ $SJUJDBM
    ͳΞϥʔτ͸ʮαʔϏεͷܧଓੑʹӨڹ͢Δ΋ͷʯͷΈ
11. Mackerel ͷΫϥ΢υωΠςΟϒ͞ !
    ϗετ͕૿͑ͯ΋ݮͬͯ΋ࣗಈͰ௥ैͰ͖Δ NBDLFSFMDPOUBJOFSBHFOU
    "84"[VSFΠϯςάϨʔγϣϯ !
    ୀ໾͢ΔͱϗετϝτϦοΫ͕ݟ͑ͳ͘ͳΔ $16ͳͲҰ෦ͷϝτϦοΫͷΈ࢒ΔɺΧελϜϝτϦοΫ͸
    ߹ܭ΍ฏۉͳͲ͸௕ظͰ௥͍͍͕ͨɺফ͑ͯ͠·͏ !
    ϗετ୯Ґ՝ۚ ϚΠΫϩϗετ ԁ݄ Ͱ΋424΍-BNCEBͷΑ͏ʹ ͍҆ɺϝτϦοΫ͕গͳ͍ର৅ͷ؂ࢹʹ͸ͪΐͬͱߴ͍ʜ ૯ϝτϦοΫ਺՝ۚϓϥϯ͕΄͍͠
12. ϚωʔδυαʔϏεͷਐԽʹظ଴ͭͭ͠ Ϋϥ΢υωΠςΟϒͳ؂ࢹΛਐΊΔͨΊʹ ʮܺؒՈ۩044ʯ ϚωʔδυαʔϏεͷػೳ΍αʔϏεؒ࿈ܞ͕ࣗ෼ͨͪͷӡ༻ʹ͓͍ͯෆे෼ͳ৔ ߹ʹɺͦͷ伱ؒΛຒΊͯΑΓΑ͍ӡ༻Λ࣮ݱ͢ΔͨΊʹ։ൃ͞Εͨιϑτ΢ΣΞɻ ಛʹOSSͷ΋ͷΛࢦ͢ɻ3 
    IUUQTTQFBLFSEFDLDPNGVKJXBSBYJKJBOKJBKVPTTGBMTFTVTVNF IUUQTTQFBLFSEFDLDPNGVKJXBSBBXTEFWEBZUPLZP
13. ࠓ೔঺հ͢Δ伱ؒՈ۩ OSS w NBQSPCF w NBDLFSFMQMVHJOQSPNFUIFVTRVFSZ 
    IUUQTHJUIVCDPNGVKJXBSBNBDLFSFMQMVHJOQSPNFUIFVTRVFSZ 
    IUUQTHJUIVCDPNGVKJXBSBNBQSPCF
14. ʲ՝୊ʳAWSΠϯςάϨʔγϣϯͰొ࿥͞ΕͨϗετͰ mackerel-plugin ͰͷϝτϦοΫ΋औΓ͍ͨ
15. ྫɿAmazon RDS(MySQL)ʹରͯ͠ mackerel-plugin-mysql Λ࣮ߦ
16. ʲղ๏ʳͲ͔͜ͷϗετͷ mackerel-agent Ͱ plugin ࣮ߦʁ [plugin.metrics.rds01] command = "mackerel-plugin-mysql -host='rds01.***.ap-northeast-1.rds.amazonaws.com' (ུ)" custom_identifier = "rds01.***.ap-northeast-1.rds.amazonaws.com" [plugin.metrics.rds02] command = "mackerel-plugin-mysql -host='rds02.***.ap-northeast-1.rds.amazonaws.com' (ུ)" custom_identifier = "rds02.***.ap-northeast-1.rds.amazonaws.com" !
    ͜ͷϗετ͕མͪͨΒϝτϦοΫऩू͕ࢭ·Δ "
    ͋ͱ͔Β૿͑ͨϗετΛ؂ࢹ͢Δͷʹઃఆมߋ͕໘౗ #
    ࠷ۙ͸
    NBDLFSFMBHFOU
    Λಈ͔͢ϗετ
    ͕ͳ͍͜ͱ΋ʜ ؂ࢹର৅ͷ૿ݮʹࣗಈ௥ै͍ͨ͠ʂ
17. maprobe w .BDLFSFMʹొ࿥͞Εͨϗετʹରͯ͠ w ֎ܗ؂ࢹ
    QJOH
    
    UDQ
    
    IUUQ w NBDLFSFMQMVHJOΛ࣮ߦ ϗετϝτϦοΫͱͯ͠౤ߘ w ొ࿥ࡁΈͷϗετϝτϦοΫΛू໿͠ αʔϏεϝτϦοΫͱͯ͠౤ߘ Λߦ͏ͨΊͷΤʔδΣϯτ
18. ʲղ๏ʳ maprobe Ͱ plugin ࣮ߦ probes: - service: production role: RDS command: command: - 'mackerel-plugin-mysql' - '-host={{.Host.CustomIdentifier}}' - '-username=root' - '-password={{env "RDS_PASSWORD"}}' αʔϏεQSPEVDUJPO ϩʔϧ3%4
    ͷϗετશͯʹରͯ͠
    NBDLFSFMQMVHJONZTRM
    Λ࣮ߦ ݁ՌΛݸʑͷϗετϝτϦοΫͱͯ͠
    .BDLFSFM
    ΁ૹ৴͢Δ
19. maprobe ͸ର৅ϗετͷ૿ݮʹࣗಈ௥ै ຖ෼
    .BDLFSFM
    "1*
    Λୟ͍ͯϗετΛݕࡧ !
    ϗετͷ૿ݮʹࣗಈͰ௥ै %PDLFSίϯςφΞϦ㽂 docker pull fujiwara/maprobe 4
    ʹஔ͍ͨઃఆϑΝΠϧΛࣗಈͰ࠶ಡΈࠐΈ !
    4
    Λߋ৽͢Ε͹ίϯςφϏϧυɾσϓϩΠෆ༻Ͱઃఆ൓ө maprobe agent --config s3://example.com/config.yaml 
    IUUQTIVCEPDLFSDPNSGVKJXBSBNBQSPCF
20. ʲ՝୊ʳconnec%vity Λ΍ΊͨΒ ϗετͷࢮ׆؂ࢹ͸Ͳ͏͢Δʁ
21. ʲղ๏ʳmaprobe ͷϔϧενΣοΫػೳ ૊ΈࠐΈͷϔϧενΣοΫػೳ QJOH
    5$1
    )551
    ͕͋Δ probes: - service: production role: EC2 ping: address: "{{ .Host.IPAddresses.eth0 }}" - service: production role: ElastiCacheRedis tcp: host: "{{ .Host.CustomIdentifier }}" port: 6379 send: "PING\n" expect_pattern: "PONG"
22. maprobe ͰͷϔϧενΣοΫ NBQSPCF
    ͷϔϧενΣοΫ݁Ռ͸ϗετϝτϦοΫʹͳΔ DIFDL
    ؂ࢹͰ͸ͳ͍
23. check ؂ࢹ͕Α͘ͳ͍ͱ͜Ζ(ࢲݟ) ઃఆมߋ͕ϑΝΠϧͷमਖ਼
    
    σϓϩΠ
    ʮͪΐͬͱ͍·͚ͩ؂ࢹ
    P⒎
    
    ᮢ஋มߋʯ͕೉͍͠ ᮢ஋ͷධՁํ๏͕ϓϥάΠϯ͝ͱʹ·ͪ·ͪ
    --critical-under
    ͸ʮҎԼʯ͔ʮະຬʯ͔ʜ Ұ౓ʹଟ਺ͷϗετͰൃใ͕ͪ͠
    ਺ेϗετ͔ΒDIFDL؂ࢹࣦഊ͕དྷͯ΋ݪҼ͸ݸͩͬͨΓ
    OUQͷ࣌ࠁͣΕɺEBFNPO
    ͷઃఆ EFQMPZ ϛεʜ
24. check ؂ࢹ = metric ؂ࢹͷಛघͳύλʔϯ ϝτϦοΫΛอଘͯͦ͠ΕΛධՁ͢Ε͹ಉ͜͡ͱ͕Ͱ͖Δ ϝτϦοΫ؂ࢹɺࣜ؂ࢹΛ׆༻͢Δ
25. ྫɿ ping ʹΑΔࢮ׆؂ࢹ sum(role(production:EC2, ping.count.failure)) QSPDVUDJPO&$
    ͷ͍ͣΕ͔ͷϗετʹQJOH͕ࣦഊͨ͠ΒXBSO Կ୆͔མͪͯ΋αʔϏε͕ఏڙͰ͖͍ͯΕ͹
    $SJUJDBM
    Ͱ͸ͳ͍
26. ྫɿ job queue ͷ଺ཹ job ਺ΛΞϥʔτ sum(role(production:job-queue, custom.gearmand.queue.*.total)) ෳ਺ͷϗετʹKPCRVFVF͕͋Δ ଺ཹ
    KPC
    ਺ΛϝτϦοΫʹ͍ͯ͠Δ ཷ·Δͱ͖͸શͯͷRVFVF͕ཷ·Δ͜ͱ͕ଟ͍ DIFDL
    ؂ࢹͰݸผʹΞϥʔτ͢Δͱશ෦ͷϗετͰൃใ͕ͪ͠ ߹ܭΛݟΔ͜ͱͰશମͷॲཧঢ়گΛ೺Ѳ͢Δ
27. ʲ՝୊ʳୀ໾͢Δͱফ͑ͯ͠·͏ ϗετϝτϦοΫΛ௥͍͍ͨ
28. ΧελϜϝτϦοΫ͸ϗετ͕ୀ໾͢Δͱফ͑ͯ͠·͏
29. ʲղ๏ʳmaprobe ͰϗετϝτϦοΫΛू໿อଘ aggregates: - service: production role: push-server metrics: - name: custom.push.messages.sent outputs: - func: sum name: custom.push.messages.total_sent αʔϏεQSPEVDUJPO
    ϩʔϧQVTITFSWFS
    ʹରͯ͠ ϗετϝτϦοΫͷ
    QVTINFTTBHFTTFOU
    Λશ୆෼औಘ ˠ
    ԋࢉͨ݁͠ՌΛαʔϏεϝτϦοΫͱͯ͠อଘ͢Δ
30. maprobe aggregate func/ons ݱࡏ
    TVN
    NJO
    NBY
    BWFSBHF
    DPVOU
    Λαϙʔτ QFSDFOUJMF΋͋ͬͨ΄͏͕Αͦ͞͏͚ͩͲະ࣮૷ ΧελϜϝτϦοΫ͕ফ͑ͳ͚Ε͹ࣜάϥϑͰ׬݁ͳͷͰԿଔ
31. ʲ՝୊ʳ΋ͬͱΫϥ΢υωΠςΟϒͳ Ϧιʔεͷ؂ࢹ
32. ϚΠΫϩαʔϏεʂ αʔϏεϝογϡʂ &OWPZΛ࢖͍࢝Ί͍ͯΔͷͰɺϝτϦΫεΛऔΓ͍ͨ ͱ͋Δ
    &OWPZ
    ͷ
    /stats
    Λୟ͘ͱʜ $ curl -s x.x.x.x:9901/stats ... cluster.web.default.total_match_count: 1 cluster.web.external.upstream_rq_200: 988 cluster.web.external.upstream_rq_2xx: 988 cluster.web.external.upstream_rq_302: 13 cluster.web.external.upstream_rq_3xx: 13 cluster.web.external.upstream_rq_400: 3 cluster.web.external.upstream_rq_403: 26 cluster.web.external.upstream_rq_404: 5 cluster.web.external.upstream_rq_4xx: 34 .... 
    IUUQTFOWPZQSPYZJP
33. ͜ΕΛશ෦ Mackerel ʹૹΕ͹Α͍ʁ cluster.web.default.total_match_count: 1 cluster.web.external.upstream_rq_200: 988 cluster.web.external.upstream_rq_2xx: 988 cluster.web.external.upstream_rq_302: 13 cluster.web.external.upstream_rq_3xx: 13 cluster.web.external.upstream_rq_400: 3 cluster.web.external.upstream_rq_403: 26 cluster.web.external.upstream_rq_404: 5 cluster.web.external.upstream_rq_4xx: 34 ...
34. ໰͍߹Θͤ ϑΟʔυόοΫϑΥʔϜ͔Βؾܰʹฉ͍ͯΈͨ ۙʑenvoy(https://www.envoyproxy.io/)Λಋೖ༧ఆͳͷͰ͕͢ɺMackerel ެࣜͱͯ͠envoy statsऔಘϓϥάΠϯΛެ։͞ΕΔ༧ఆ͸͋Γ·͢Ͱ͠ΐ͏͔ʁ envoyͷϓϥάΠϯͰ͕͢ɺݱ࣌఺Ͱ͸ެࣜͱͯ͠ެ։͢Δ༧ఆ͸͍͟͝ ·ͤΜɻ !
    ࡞Δ͔ʜʜ
35. ͔͠͠ Envoy ͸େྔͷϝτϦοΫΛు͖ग़͢ $ curl -s x.x.x.x:9901/stats | wc -l 337 ͜ΕΛશ෦
    .BDLFSFM
    ʹ͍࣋ͬͯ͘ͱʜ ϚΠΫϩϗετ͸ϝτϦοΫϗετ
    ˠ
    ϗετ૬౰ &OWPZ
    ͕͍Δϗετ͝ͱʹ
    
    Y
    
    
    ԁ ੫ผ
    
36. ࡞ઓมߋ શ෦Λ͍࣋ͬͯ͘ͷ͸ίετ͕ݫ͍͠ ͱ͸͍͑ͲΜͳ஋͕΄͍͔͠Α͘෼͔͍ͬͯͳ͍ &OWPZӡ༻ܦݧ͕ઙ͍ͷͰɺӡ༻͠ͳ͕ΒݟΔ஋ΛܾΊ͍ͨ QMVHJOΛ࡞ͬͯ༗༻ͳϝτϦοΫΛऔΔʹ͸ӡ༻ܦݧ͕ඞཁ ͭ·ΓͱΓ͋͑ͣ͸શ෦औΓ͍ͨ
37. Prometheus ͸Ͳ͏͔ʁ 1VMM
    ܕϝτϦοΫऩूɾ؂ࢹπʔϧ อଘͨ͠஋Λ
    1SPN2-
    ΫΤϦͰॊೈʹՃ޻ͯ͠औಘͰ͖Δ ௕ظؒͷϝτϦοΫอଘ͸͋·Γߟྀ͞Ε͍ͯͳ͍ (SBGBOBͱ͔ͰՄࢹԽ͕ී௨ʁ
38. ! Envoy → Prometheus → Mackerel զʑ͸
    .BDLFSFM
    ͰΞϥʔτ͍ͨ͠͠άϥϑ΋ݟ͍ͨ Ͳͷ஋ΛऔΔ΂͖͔͕ݟ͑ͳ͍ͷͰɺ௚ۙ͸શ෦औ͓͖͍ͬͯͨ  &OWPZ
    ˠ
    1SPNFUIFVT ͜Ε͸͋Δ  1SPNFUIFVT
    ˠ
    ΫΤϦ݁ՌΛ
    NFUSJD
    QMVHJO
    ܗࣜͰग़ྗ  NFUSJD
    QMVHJO
    ܗࣜͷग़ྗΛ
    .BDLFSFM
    ʹอଘ ͜Ε΋͋Δ ͜ͷ
    
    ͚ͩ࡞Ε͹Α͍ͷͰ͸ʂ ଍Γͳ͍஋͸౎౓
    1SPNFUIFVT
    ʹΫΤϦͯ͠औΕ͹Α͍
39. mackrel-plugin-prometheus-query ࡞Γ·ͨ͠ HJUIVCDPNGVKJXBSBNBDLFSFMQMVHJOQSPNFUIFVTRVFSZ $ mackerel-plugin-prometheus-query \ -query "up" \ -metric-key-format "promq.{job}.{instance}" promq.web.10_1_129_175_9901 1 1575941187 promq.web.10_1_130_170_9901 1 1575941187 promq.web.10_1_131_53_9901 1 1575941187 promq.prometheus.localhost_9090 1 1575941187
40. Prometheus ΁ͷΫΤϦྫ ྫɿ෼ؒͷVQTUSFBN΁ͷϦτϥΠճ਺ΛٻΊΔΫΤϦ sum( delta( envoy_cluster_upstream_rq_retry{envoy_cluster_name="web"}[1m] ) ) .BDLFSFM
    ʹ͸ϗετ୯ҐͰ͸ͳ͘શମͷ஋ΛૹΔ ݸʑͷ஋͸
    1SPNFUIFVT
    ΛݟΕ͹͋ΔͷͰ
41. plugin ͷग़ྗΛ mkr throw Ͱ౤͛Δ $ mackerel-plugin-prometheus-query \ -query 'sum(delta(envoy_cluster_upstream_rq_retry_success {envoy_cluster_name="web"}[1m]))' \ -metric-key-format "envoy.web.upstream.retry.success" \ | mkr throw --service production
42. ·ͱΊ Ϋϥ΢υωΠςΟϒͳ؂ࢹͱ͸ ো֐Λڐ༰͢Δઃܭ
    
    શମͷ݈શੑΛݟΔ ͦͷಓ۩ͱͯ͠ͷܺؒՈ۩044 w NBQSPCF ࣗಈ௥ैɺ֎ܗ؂ࢹɺQMVHJO؂ࢹɺϝτϦοΫू໿ w NBDLFSFMQMVHJOQSPNFUIFVTRVFSZ TIPSU
    UFSN
    ͸
    1SPNFUIFVT
    Ͱ MPOH
    UFSN
    BMFSUJOH
    ͸
    .BDLFSFM
    Ͱ