クラウドネイティブな監視をMackerelで / Mackerel Day#2

Transcript

1. Ϋϥ΢υωΠςΟϒͳ؂ࢹΛ
   Mackerel Ͱ
   2019.12.23 Mackerel Day #2
   @fujiwara
   

   View Slide

2. @fujiwara
   .BDLFSFMΞϯόαμʔ ʙ
   
   HJUIVCDPNLBZBDFDTQSFTTP "NB[PO&$4σϓϩΠπʔϧ
   
   HJUIVCDPNGVKJXBSBMBNCSPMM "84-BNCEBσϓϩΠπʔ
   ϧ
   
   

   View Slide

3. Game & Community
   

   View Slide

4. View Slide

5. Ϋϥ΢υωΠςΟϒʁ
   $/$'$MPVE/BUJWF%FpOJUJPOW
   Ϋϥ΢υωΠςΟϒٕज़͸ɺύϒϦοΫΫϥ΢υɺϓϥΠϕʔτΫϥ΢υɺϋΠϒ
   ϦουΫϥ΢υͳͲͷۙ୅తͰμΠφϛοΫͳ؀ڥʹ͓͍ͯɺεέʔϥϒϧͳΞϓ
   ϦέʔγϣϯΛߏங͓Αͼ࣮ߦ͢ΔͨΊͷೳྗΛ૊৫ʹ΋ͨΒ͠·͢ɻ
   ͜ͷΞϓϩʔνͷ୅දྫʹɺίϯςφɺαʔϏεϝογϡɺϚΠΫϩαʔϏεɺΠ
   ϛϡʔλϒϧΠϯϑϥετϥΫνϟɺ͓ΑͼએݴܕAPI͕͋Γ·͢ɻ
   IUUQTHJUIVCDPNDODGUPDCMPCNBTUFS%&'*/*5*0/NE
   

   View Slide

6. ίϯςφɺαʔϏεϝογϡɺϚΠΫϩαʔϏε…
   ͦΕΒͷٕज़Λ࢖͍ͬͯΔ㱠Ϋϥ΢υωΠςΟϒ
   "Design for Failure" ͱͦΕΛ࣮ݱ͢Δ࢓૊Έ΍ίϯϙʔωϯτɺͦΕΛ࢖͍
   ͜ͳ͢͜ͱ͕ͦ͜Ϋϥ΢υωΠςΟϒ
   ো֐ͷൃੜʹରͯࣗ͠ಈ෮چͰ͖ΔΑ͏ʹσβΠϯ͢Δɻো֐ͷൃੜʹΑΔϢʔ
   βʔӨڹ͕ͦ΋ͦ΋ͳ͍Α͏ʹΞʔΩςΫνϟΛσβΠϯ͢Δ2
   IUUQTTQFBLFSEFDLDPNUPSJDMTEFTJHOGPSGBJMVSFJTUIFUSVFDMPVEOBUJWF
   

   View Slide

7. "Design for Failure"
   ো֐Λ૝ఆͯ͠γεςϜΛσβΠϯ͢Δ
   w Πϯελϯε͸ࢮ͵
   w ϚωʔδυαʔϏε΋ࢮ͵ ͚Ͳେ఍GBJMPWFS͢Δ
   
   w σʔληϯλʔ͝ͱࢮ͵ كʹ͋Δ
   
   Ͳ͜·ͰΛ૝ఆ͠ɺͲ͔͜Β͸ఘΊΔ͔
   ؂ࢹ΋ͦΕΒΛંΓࠐΜͰઃܭ͢ΔˠΫϥ΢υωΠςΟϒͳ؂ࢹ
   

   View Slide

8. ؂ࢹର৅͸ಈతʹ૿͑ͨΓݮͬͨΓ͢Δ
   ίϯςφ͸σϓϩΠ͝ͱʹੜ·ΕมΘΔ
   ίϯςφͰͳͯ͘΋
   w &$ͷΦʔτεέʔϦϯάͰ4QPUΠϯελϯεΛ࢖͏
   w ΘΓͱ͙͢མͪΔɺଞͷ্͕͕Δ
   ϚωʔδυαʔϏε΋ෛՙʹԠͯ͡૿ݮͰ͖ΔΑ͏ʹͳ͖ͬͯͨ
   "VSPSB"VUP4DBMJOHͳͲ
   
   

   View Slide

9. Mackerel ͰΫϥ΢υωΠςΟϒͷୈҰา
   ʮDPOOFDUJWJUZݕ஌Λ΍ΊΔʯ
   

   View Slide

10. connec%vity ݕ஌Λ΍ΊΔ
    ͜ͷΞϥʔτ͸
    !
    $SJUJDBM͔͠ͳ͍
    w ։ൃ؀ڥ͕ਂ໷ʹམͪͯඈͼى͖Δඞཁ͸ͳ͍
    w ຊ൪؀ڥͰ୆མͪͯ΋αʔϏεʹӨڹ͢Δϗετ͸ͳ͍
    Α͏ʹ࡞Δ
    
    ϗετ͕ࢮ͵͜ͱΛલఏʹσβΠϯ͢ΔʹΫϥ΢υωΠςΟϒ
    $SJUJDBMͳΞϥʔτ͸ʮαʔϏεͷܧଓੑʹӨڹ͢Δ΋ͷʯͷΈ
    

    View Slide

11. Mackerel ͷΫϥ΢υωΠςΟϒ͞
    !
    ϗετ͕૿͑ͯ΋ݮͬͯ΋ࣗಈͰ௥ैͰ͖Δ
    NBDLFSFMDPOUBJOFSBHFOU "84"[VSFΠϯςάϨʔγϣϯ
    !
    ୀ໾͢ΔͱϗετϝτϦοΫ͕ݟ͑ͳ͘ͳΔ
    $16ͳͲҰ෦ͷϝτϦοΫͷΈ࢒ΔɺΧελϜϝτϦοΫ͸
    ߹ܭ΍ฏۉͳͲ͸௕ظͰ௥͍͍͕ͨɺফ͑ͯ͠·͏
    !
    ϗετ୯Ґ՝ۚ
    ϚΠΫϩϗετ ԁ݄
    Ͱ΋424΍-BNCEBͷΑ͏ʹ
    ͍҆ɺϝτϦοΫ͕গͳ͍ର৅ͷ؂ࢹʹ͸ͪΐͬͱߴ͍ʜ
    ૯ϝτϦοΫ਺՝ۚϓϥϯ͕΄͍͠
    
    

    View Slide

12. ϚωʔδυαʔϏεͷਐԽʹظ଴ͭͭ͠
    Ϋϥ΢υωΠςΟϒͳ؂ࢹΛਐΊΔͨΊʹ
    ʮܺؒՈ۩044ʯ
    ϚωʔδυαʔϏεͷػೳ΍αʔϏεؒ࿈ܞ͕ࣗ෼ͨͪͷӡ༻ʹ͓͍ͯෆे෼ͳ৔
    ߹ʹɺͦͷ伱ؒΛຒΊͯΑΓΑ͍ӡ༻Λ࣮ݱ͢ΔͨΊʹ։ൃ͞Εͨιϑτ΢ΣΞɻ
    ಛʹOSSͷ΋ͷΛࢦ͢ɻ3
    IUUQTTQFBLFSEFDLDPNGVKJXBSBYJKJBOKJBKVPTTGBMTFTVTVNF
    IUUQTTQFBLFSEFDLDPNGVKJXBSBBXTEFWEBZUPLZP
    

    View Slide

13. ࠓ೔঺հ͢Δ伱ؒՈ۩ OSS
    w NBQSPCF
    w NBDLFSFMQMVHJOQSPNFUIFVTRVFSZ
    IUUQTHJUIVCDPNGVKJXBSBNBDLFSFMQMVHJOQSPNFUIFVTRVFSZ
    IUUQTHJUIVCDPNGVKJXBSBNBQSPCF
    

    View Slide

14. ʲ՝୊ʳAWSΠϯςάϨʔγϣϯͰొ࿥͞ΕͨϗετͰ
    mackerel-plugin ͰͷϝτϦοΫ΋औΓ͍ͨ
    

    View Slide

15. ྫɿAmazon RDS(MySQL)ʹରͯ͠
    mackerel-plugin-mysql Λ࣮ߦ
    

    View Slide

16. ʲղ๏ʳͲ͔͜ͷϗετͷ mackerel-agent Ͱ plugin ࣮ߦʁ
    [plugin.metrics.rds01]
    command = "mackerel-plugin-mysql -host='rds01.***.ap-northeast-1.rds.amazonaws.com' (ུ)"
    custom_identifier = "rds01.***.ap-northeast-1.rds.amazonaws.com"
    [plugin.metrics.rds02]
    command = "mackerel-plugin-mysql -host='rds02.***.ap-northeast-1.rds.amazonaws.com' (ུ)"
    custom_identifier = "rds02.***.ap-northeast-1.rds.amazonaws.com"
    !
    ͜ͷϗετ͕མͪͨΒϝτϦοΫऩू͕ࢭ·Δ
    "
    ͋ͱ͔Β૿͑ͨϗετΛ؂ࢹ͢Δͷʹઃఆมߋ͕໘౗
    #
    ࠷ۙ͸NBDLFSFMBHFOU Λಈ͔͢ϗετ
    ͕ͳ͍͜ͱ΋ʜ
    ؂ࢹର৅ͷ૿ݮʹࣗಈ௥ै͍ͨ͠ʂ
    

    View Slide

17. maprobe
    w .BDLFSFMʹొ࿥͞Εͨϗετʹରͯ͠
    w ֎ܗ؂ࢹQJOHUDQIUUQ
    w NBDLFSFMQMVHJOΛ࣮ߦ
    ϗετϝτϦοΫͱͯ͠౤ߘ
    w ొ࿥ࡁΈͷϗετϝτϦοΫΛू໿͠
    αʔϏεϝτϦοΫͱͯ͠౤ߘ
    Λߦ͏ͨΊͷΤʔδΣϯτ
    

    View Slide

18. ʲղ๏ʳ maprobe Ͱ plugin ࣮ߦ
    probes:
    - service: production
    role: RDS
    command:
    command:
    - 'mackerel-plugin-mysql'
    - '-host={{.Host.CustomIdentifier}}'
    - '-username=root'
    - '-password={{env "RDS_PASSWORD"}}'
    αʔϏεQSPEVDUJPO ϩʔϧ3%4ͷϗετશͯʹରͯ͠
    NBDLFSFMQMVHJONZTRMΛ࣮ߦ
    ݁ՌΛݸʑͷϗετϝτϦοΫͱͯ͠.BDLFSFM΁ૹ৴͢Δ
    

    View Slide

19. maprobe ͸ର৅ϗετͷ૿ݮʹࣗಈ௥ै
    ຖ෼.BDLFSFM"1*Λୟ͍ͯϗετΛݕࡧ
    !
    ϗετͷ૿ݮʹࣗಈͰ௥ै
    %PDLFSίϯςφΞϦ㽂
    docker pull fujiwara/maprobe
    4ʹஔ͍ͨઃఆϑΝΠϧΛࣗಈͰ࠶ಡΈࠐΈ
    !
    4Λߋ৽͢Ε͹ίϯςφϏϧυɾσϓϩΠෆ༻Ͱઃఆ൓ө
    maprobe agent --config s3://example.com/config.yaml
    IUUQTIVCEPDLFSDPNSGVKJXBSBNBQSPCF
    

    View Slide

20. ʲ՝୊ʳconnec%vity Λ΍ΊͨΒ
    ϗετͷࢮ׆؂ࢹ͸Ͳ͏͢Δʁ
    

    View Slide

21. ʲղ๏ʳmaprobe ͷϔϧενΣοΫػೳ
    ૊ΈࠐΈͷϔϧενΣοΫػೳ
    QJOH 5$1 )551͕͋Δ
    probes:
    - service: production
    role: EC2
    ping:
    address: "{{ .Host.IPAddresses.eth0 }}"
    - service: production
    role: ElastiCacheRedis
    tcp:
    host: "{{ .Host.CustomIdentifier }}"
    port: 6379
    send: "PING\n"
    expect_pattern: "PONG"
    

    View Slide

22. maprobe ͰͷϔϧενΣοΫ
    NBQSPCFͷϔϧενΣοΫ݁Ռ͸ϗετϝτϦοΫʹͳΔ
    DIFDL؂ࢹͰ͸ͳ͍
    

    View Slide

23. check ؂ࢹ͕Α͘ͳ͍ͱ͜Ζ(ࢲݟ)
    ઃఆมߋ͕ϑΝΠϧͷमਖ਼σϓϩΠ
    ʮͪΐͬͱ͍·͚ͩ؂ࢹP⒎ᮢ஋มߋʯ͕೉͍͠
    ᮢ஋ͷධՁํ๏͕ϓϥάΠϯ͝ͱʹ·ͪ·ͪ
    --critical-under͸ʮҎԼʯ͔ʮະຬʯ͔ʜ
    Ұ౓ʹଟ਺ͷϗετͰൃใ͕ͪ͠
    ਺ेϗετ͔ΒDIFDL؂ࢹࣦഊ͕དྷͯ΋ݪҼ͸ݸͩͬͨΓ
    OUQͷ࣌ࠁͣΕɺEBFNPOͷઃఆ EFQMPZ
    ϛεʜ
    

    View Slide

24. check ؂ࢹ = metric ؂ࢹͷಛघͳύλʔϯ
    ϝτϦοΫΛอଘͯͦ͠ΕΛධՁ͢Ε͹ಉ͜͡ͱ͕Ͱ͖Δ
    ϝτϦοΫ؂ࢹɺࣜ؂ࢹΛ׆༻͢Δ
    

    View Slide

25. ྫɿ ping ʹΑΔࢮ׆؂ࢹ
    sum(role(production:EC2, ping.count.failure))
    QSPDVUDJPO&$ͷ͍ͣΕ͔ͷϗετʹQJOH͕ࣦഊͨ͠ΒXBSO
    Կ୆͔མͪͯ΋αʔϏε͕ఏڙͰ͖͍ͯΕ͹$SJUJDBMͰ͸ͳ͍
    

    View Slide

26. ྫɿ job queue ͷ଺ཹ job ਺ΛΞϥʔτ
    sum(role(production:job-queue, custom.gearmand.queue.*.total))
    ෳ਺ͷϗετʹKPCRVFVF͕͋Δ
    ଺ཹKPC਺ΛϝτϦοΫʹ͍ͯ͠Δ
    ཷ·Δͱ͖͸શͯͷRVFVF͕ཷ·Δ͜ͱ͕ଟ͍
    
    DIFDL؂ࢹͰݸผʹΞϥʔτ͢Δͱશ෦ͷϗετͰൃใ͕ͪ͠
    ߹ܭΛݟΔ͜ͱͰશମͷॲཧঢ়گΛ೺Ѳ͢Δ
    

    View Slide

27. ʲ՝୊ʳୀ໾͢Δͱফ͑ͯ͠·͏
    ϗετϝτϦοΫΛ௥͍͍ͨ
    

    View Slide

28. ΧελϜϝτϦοΫ͸ϗετ͕ୀ໾͢Δͱফ͑ͯ͠·͏
    

    View Slide

29. ʲղ๏ʳmaprobe ͰϗετϝτϦοΫΛू໿อଘ
    aggregates:
    - service: production
    role: push-server
    metrics:
    - name: custom.push.messages.sent
    outputs:
    - func: sum
    name: custom.push.messages.total_sent
    αʔϏεQSPEVDUJPO ϩʔϧQVTITFSWFSʹରͯ͠
    ϗετϝτϦοΫͷQVTINFTTBHFTTFOUΛશ୆෼औಘ
    ˠԋࢉͨ݁͠ՌΛαʔϏεϝτϦοΫͱͯ͠อଘ͢Δ
    

    View Slide

30. maprobe aggregate func/ons
    ݱࡏTVN NJO NBY BWFSBHF DPVOUΛαϙʔτ
    QFSDFOUJMF΋͋ͬͨ΄͏͕Αͦ͞͏͚ͩͲະ࣮૷
    
    ΧελϜϝτϦοΫ͕ফ͑ͳ͚Ε͹ࣜάϥϑͰ׬݁ͳͷͰԿଔ
    

    View Slide

31. ʲ՝୊ʳ΋ͬͱΫϥ΢υωΠςΟϒͳ
    Ϧιʔεͷ؂ࢹ
    

    View Slide

32. ϚΠΫϩαʔϏεʂ αʔϏεϝογϡʂ
    &OWPZΛ࢖͍࢝Ί͍ͯΔͷͰɺϝτϦΫεΛऔΓ͍ͨ
    ͱ͋Δ&OWPZͷ/statsΛୟ͘ͱʜ
    $ curl -s x.x.x.x:9901/stats
    ...
    cluster.web.default.total_match_count: 1
    cluster.web.external.upstream_rq_200: 988
    cluster.web.external.upstream_rq_2xx: 988
    cluster.web.external.upstream_rq_302: 13
    cluster.web.external.upstream_rq_3xx: 13
    cluster.web.external.upstream_rq_400: 3
    cluster.web.external.upstream_rq_403: 26
    cluster.web.external.upstream_rq_404: 5
    cluster.web.external.upstream_rq_4xx: 34
    ....
    IUUQTFOWPZQSPYZJP
    

    View Slide

33. ͜ΕΛશ෦ Mackerel ʹૹΕ͹Α͍ʁ
    cluster.web.default.total_match_count: 1
    cluster.web.external.upstream_rq_200: 988
    cluster.web.external.upstream_rq_2xx: 988
    cluster.web.external.upstream_rq_302: 13
    cluster.web.external.upstream_rq_3xx: 13
    cluster.web.external.upstream_rq_400: 3
    cluster.web.external.upstream_rq_403: 26
    cluster.web.external.upstream_rq_404: 5
    cluster.web.external.upstream_rq_4xx: 34
    ...
    

    View Slide

34. ໰͍߹Θͤ
    ϑΟʔυόοΫϑΥʔϜ͔Βؾܰʹฉ͍ͯΈͨ
    ۙʑenvoy(https://www.envoyproxy.io/)Λಋೖ༧ఆͳͷͰ͕͢ɺMackerel
    ެࣜͱͯ͠envoy statsऔಘϓϥάΠϯΛެ։͞ΕΔ༧ఆ͸͋Γ·͢Ͱ͠ΐ͏͔ʁ
    envoyͷϓϥάΠϯͰ͕͢ɺݱ࣌఺Ͱ͸ެࣜͱͯ͠ެ։͢Δ༧ఆ͸͍͟͝
    ·ͤΜɻ
    !
    ࡞Δ͔ʜʜ
    

    View Slide

35. ͔͠͠ Envoy ͸େྔͷϝτϦοΫΛు͖ग़͢
    $ curl -s x.x.x.x:9901/stats | wc -l
    337
    ͜ΕΛશ෦.BDLFSFMʹ͍࣋ͬͯ͘ͱʜ
    ϚΠΫϩϗετ͸ϝτϦοΫϗετˠϗετ૬౰
    &OWPZ͕͍Δϗετ͝ͱʹYԁ ੫ผ
    
    

    View Slide

36. ࡞ઓมߋ
    શ෦Λ͍࣋ͬͯ͘ͷ͸ίετ͕ݫ͍͠
    ͱ͸͍͑ͲΜͳ஋͕΄͍͔͠Α͘෼͔͍ͬͯͳ͍
    &OWPZӡ༻ܦݧ͕ઙ͍ͷͰɺӡ༻͠ͳ͕ΒݟΔ஋ΛܾΊ͍ͨ
    QMVHJOΛ࡞ͬͯ༗༻ͳϝτϦοΫΛऔΔʹ͸ӡ༻ܦݧ͕ඞཁ
    ͭ·ΓͱΓ͋͑ͣ͸શ෦औΓ͍ͨ
    

    View Slide

37. Prometheus ͸Ͳ͏͔ʁ
    1VMMܕϝτϦοΫऩूɾ؂ࢹπʔϧ
    อଘͨ͠஋Λ1SPN2-ΫΤϦͰॊೈʹՃ޻ͯ͠औಘͰ͖Δ
    ௕ظؒͷϝτϦοΫอଘ͸͋·Γߟྀ͞Ε͍ͯͳ͍
    (SBGBOBͱ͔ͰՄࢹԽ͕ී௨ʁ
    

    View Slide

38. !
    Envoy → Prometheus → Mackerel
    զʑ͸.BDLFSFMͰΞϥʔτ͍ͨ͠͠άϥϑ΋ݟ͍ͨ
    Ͳͷ஋ΛऔΔ΂͖͔͕ݟ͑ͳ͍ͷͰɺ௚ۙ͸શ෦औ͓͖͍ͬͯͨ
    &OWPZˠ1SPNFUIFVT ͜Ε͸͋Δ
    
    1SPNFUIFVTˠΫΤϦ݁ՌΛNFUSJDQMVHJOܗࣜͰग़ྗ
    NFUSJDQMVHJOܗࣜͷग़ྗΛ.BDLFSFMʹอଘ ͜Ε΋͋Δ
    
    ͜ͷ͚ͩ࡞Ε͹Α͍ͷͰ͸ʂ
    ଍Γͳ͍஋͸౎౓1SPNFUIFVTʹΫΤϦͯ͠औΕ͹Α͍
    

    View Slide

39. mackrel-plugin-prometheus-query
    ࡞Γ·ͨ͠
    HJUIVCDPNGVKJXBSBNBDLFSFMQMVHJOQSPNFUIFVTRVFSZ
    $ mackerel-plugin-prometheus-query \
    -query "up" \
    -metric-key-format "promq.{job}.{instance}"
    promq.web.10_1_129_175_9901 1 1575941187
    promq.web.10_1_130_170_9901 1 1575941187
    promq.web.10_1_131_53_9901 1 1575941187
    promq.prometheus.localhost_9090 1 1575941187
    

    View Slide

40. Prometheus ΁ͷΫΤϦྫ
    ྫɿ෼ؒͷVQTUSFBN΁ͷϦτϥΠճ਺ΛٻΊΔΫΤϦ
    sum(
    delta(
    envoy_cluster_upstream_rq_retry{envoy_cluster_name="web"}[1m]
    )
    )
    .BDLFSFMʹ͸ϗετ୯ҐͰ͸ͳ͘શମͷ஋ΛૹΔ
    ݸʑͷ஋͸1SPNFUIFVTΛݟΕ͹͋ΔͷͰ
    

    View Slide

41. plugin ͷग़ྗΛ mkr throw Ͱ౤͛Δ
    $ mackerel-plugin-prometheus-query \
    -query 'sum(delta(envoy_cluster_upstream_rq_retry_success
    {envoy_cluster_name="web"}[1m]))' \
    -metric-key-format "envoy.web.upstream.retry.success" \
    | mkr throw --service production
    

    View Slide

42. ·ͱΊ
    Ϋϥ΢υωΠςΟϒͳ؂ࢹͱ͸
    ো֐Λڐ༰͢Δઃܭશମͷ݈શੑΛݟΔ
    ͦͷಓ۩ͱͯ͠ͷܺؒՈ۩044
    w NBQSPCF
    ࣗಈ௥ैɺ֎ܗ؂ࢹɺQMVHJO؂ࢹɺϝτϦοΫू໿
    w NBDLFSFMQMVHJOQSPNFUIFVTRVFSZ
    TIPSUUFSN͸1SPNFUIFVTͰ
    MPOHUFSN BMFSUJOH͸.BDLFSFMͰ
    

    View Slide