BASEにおけるMackerel利用上の工夫と困りごとのご紹介 / Mackerel Meetup #13 Tokyo

BASEにおけるMackerel利用上の工夫と困りごとのご紹介 / Mackerel Meetup #13 Tokyo

2019/03/01 Mackerel Meetup #13 Tokyo

B0ecbe74c3fd2be0bbc18cead2d1df2f?s=128

Fumitaka Yamane

March 01, 2019
Tweet

Transcript

  1. BASEʹ͓͚ΔMackerelར༻্ͷ޻෉ͱࠔΓ͝ͱ ͷ͝঺հ #"4&גࣜձࣾ43& ࢁ࢙ࠜਹ .BDLFSFM.FFUVQ5PLZP

  2. ຊ೔͸… BASEͰMackerelΛར༻͍ͯ͠ΔதͰͷɺͪΐͬͱͨ͠ ޻෉ɺࠔͬͨ͜ͱͱͦΕʹର͢ΔϫʔΫΞϥ΢ϯυɺ Mackerel΁ͷཁ๬ͳͲΛ঺հ͠·͢ ࢀߟʹͳΕ͹خ͍͠Ͱ͢

  3. ࣗݾ঺հ ࢁ࢙ࠜਹʢ΍·Ͷ;Έ͔ͨʣ BASEגࣜձࣾɹSREάϧʔϓ 2016೥ೖࣾ ೖࣾ࣌ʹ͸Mackerel͕ಋೖࡁΈͩͬͨ લ৬ʢιʔγϟϧήʔϜʣͰ΋Mackerel࢖ͬͯͨ twitter / ͸ͯͳID: @fumikony

  4. BASEͷϓϩμΫτʹ͍ͭͯ ωοτγϣοϓ࡞੒αʔϏε ʮBASEʯ γϣοϐϯάΞϓϦ ʮBASEʯ Ձ஋ͷަ׵ΛΑΓγϯϓϧʹ͠ɺ ੈքதͷਓʑ͕࠷దͳܦࡁ׆ಈΛߦ͑ΔΑ͏ʹ͢Δɻ MISSION

  5. BASEͷϓϩμΫτʹ͍ͭͯ ωοτγϣοϓ࡞੒αʔϏε ʮBASEʯ γϣοϐϯάΞϓϦ ʮBASEʯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ

  6. BASEͷϓϩμΫτʹ͍ͭͯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ αʔϏεͷن໛֦େʹͱ΋ͳͬͯো֐ͷӨڹ΋େ͖͘

  7. BASEͷϓϩμΫτʹ͍ͭͯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ ҟৗ͕͋ͬͨΒ͙͢ʹؾ͖͍ͮͨ MackerelͰ؂ࢹ

  8. ໨࣍ 1. BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ 2. Mackerelར༻্ͷ޻෉ 3. Mackerelར༻্ͷࠔΓ͝ͱ 4. Mackerelͷ͍͍ͱ͜Ζ

  9. 1. BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ

  10. BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ ֓ཁ ͍͍ͩͨAWS EC2, ECS, ELB, Lambda, ElastiCache, RDS, SQS,

    CloudFront, S3, SES, etc… MackerelͰ؂ࢹ SlackʹΞϥʔτ௨஌
  11. BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ αʔόͷछྨ B/GσϓϩΠ͢Δαʔό σϓϩΠ͢Δͨͼʹ
 Πϯελϯε࡞੒ ݹ͍αʔό͸ফ͍͑ͯ͘ ΞϓϦέʔγϣϯαʔό
 ʢApache+PHPʣ

  12. BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ αʔόͷछྨ B/GσϓϩΠ͠ͳ͍αʔό ͣͬͱ͋Δ όοναʔόͳͲ

  13. 2. Mackerelར༻্ͷ޻෉

  14. Mackerelར༻্ͷ޻෉ /health endpoint pattern OKͩͬͨΒεςʔλε200Λฦ͚ͩ͢ͷURL ֎ܗ؂ࢹͷର৅ /health ͱ /health/deep /health

    ͸αʔό/ίϯςφͷੜଘ /health/deep ͸DB౳΁ͷ઀ଓ·Ͱ֬ೝ ͪ͜ΒΛ֎ܗ؂ࢹͰ࢖༻
 
 See also: ʮೖ໳ ؂ࢹ ―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯʯ
 ɹɹɹɹɹ(https://www.oreilly.co.jp/books/9784873118642/)
 
 ɹɹɹɹɹஶ: Mike Julian 
 ɹɹɹɹɹ7.3 health ΤϯυϙΠϯτύλʔϯ
  15. Mackerelར༻্ͷ޻෉ /health endpoint pattern ֎ܗ؂ࢹ͕Τϥʔʹͳͬͨͱ͖ͷ௨஌

  16. Mackerelར༻্ͷ޻෉ monitoringαʔό MackerelͷΧελϜϝτϦοΫ΍αʔϏεϝτϦοΫΛऔಘɺ౤ ߘ͢ΔͨΊͷαʔό AWSΠϯςάϨʔγϣϯະରԠͷϚωʔδυαʔϏεͷϝτϦο ΫΛऔಘ ର৅ɿSES Ҏલ͸CloudFront΋ର৅͕ͩͬͨɺAWSΠϯςάϨʔγϣϯ͕ ରԠͨ͠ͷͰෆཁʹ

  17. Mackerelར༻্ͷ޻෉ monitoringαʔό AWSΠϯςάϨʔγϣϯ+ϓϥάΠϯ ର৅: RDS AWSΠϯςάϨʔγϣϯͰϝτϦοΫΛऔಘ monitoringαʔόͰ΋mackerel-plugin-mysqlͰϝτϦοΫΛ औಘ custom_identifier Λࢦఆͯ͠ಉ͡ϗετʹू໿

  18. Mackerelར༻্ͷ޻෉ monitoringαʔό [plugin.metrics.mysql-prd-main] command = "mackerel-plugin-mysql -host=db1.xxx.ap-northeast-1.rds.amazonaws.com" custom_identifier = "db1.xxx.ap-northeast-1.rds.amazonaws.com"

    See also: https://mackerel.io/ja/docs/entry/integrations/aws mackerel-agent.confʢൈਮʣ
  19. Mackerelར༻্ͷ޻෉ ࣜʹΑΔ؂ࢹ scale(divide( host(xxxxxxxxxxx, custom.ses.stats.Bounces), host(xxxxxxxxxxx, custom.ses.stats.DeliveryAttempts)), 100) mackerel-plugin-sesͰऔಘͰ͖Δͷ͸഑৴ͱό΢ϯεͦΕͧΕͷ ݅਺ͳͷͰɺࣜͰܭࢉͨ͠஋Λ؂ࢹ

    ό΢ϯε݅਺ / ഑৴݅਺ * 100 See Also: https://mackerel.io/ja/docs/entry/expression-monitoring SESͷBounce Rate (%)
  20. Mackerelར༻্ͷ޻෉ ࣜʹΑΔ؂ࢹ scale(divide( host(xxxxxxxxxxx, custom.ses.stats.Bounces), host(xxxxxxxxxxx, custom.ses.stats.DeliveryAttempts)), 100) SESͷBounce Rate

    (%)
  21. Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ mkr MackerelͷCLIπʔϧ mkr monitors ؂ࢹϧʔϧΛૢ࡞͢ΔαϒίϚϯυ mkr monitors pull

    monitors.json ͱͯ͠؂ࢹϧʔϧΛอଘ mkr monitors diff monitors.json ͱ؂ࢹϧʔϧͷࠩ෼Λදࣔ mkr monitors push monitors.json Λ؂ࢹϧʔϧʹ൓ө
  22. Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ mkr MackerelͷCLIπʔϧ mkr monitors ؂ࢹϧʔϧΛૢ࡞͢ΔαϒίϚϯυ mkr monitors pull

    monitors.json ͱͯ͠؂ࢹϧʔϧΛอଘ mkr monitors diff monitors.json ͱ؂ࢹϧʔϧͷࠩ෼Λදࣔ mkr monitors push monitors.json Λ؂ࢹϧʔϧʹ൓ө
  23. Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ mkr monitors pull → git commit → GitHubʹgit

    push CircleCIͰσΠϦʔ࣮ߦ มߋཤྺΛ௥͑Δ ·͕ͪ͑ͯյͯ͠΋໭ͤΔ όοΫΞοϓతͳ࢖͍ํʹཹΊ͍ͯΔ mkr monitors push͸࢖ΘͣɺWebUIͰͷมߋͷΈ WebUI͸ศར
  24. Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code

    || git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
  25. Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code

    || git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
  26. Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code

    || git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
  27. Mackerelར༻্ͷ޻෉ ໰͍߹Θͤ಺༰ΛSlackͰڞ༗ ʮϝʔϧઃఆʯͰɺ
 4MBDLͷʮ&NBJMʯ"QQͷ
 ϝʔϧΞυϨεΛઃఆ ໰͍߹Θͤͨ಺༰͕
 4MBDLʹྲྀΕ͖ͯͯศར

  28. • ίϯςφΤʔδΣϯτࢼͯ͠Έ·ͨ͠ • ECSͷઃఆϛεΛൃݟʂ • ϝϞϦ͕଍Γͯͳ͔ͬͨʂ

  29. • ϩʔϧ಺ҟৗݕ஌ ࢼͯ͠Έ·ͨ͠

  30. • ෛՙࢼݧͷ࣌ʹ΋࢖ͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 ɹʮEίϚʔεϓϥοτϑΥʔϜͷ੒௕Λࢧ͑Δ
 ɹɹෛՙࢼݧͷऔΓ૊Έʯ •

  31. • ෛՙࢼݧͷ࣌ʹ΋࢖ͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 • ApacheͷίωΫγϣϯ͕ுΓ෇͍ͯͨͷ͕Ϙτϧ ωοΫʹͳ͍ͬͯͨ • MackerelΛΈͨΒ͙͢ʹΘ͔ͬͨʂʢ୲౰ऀஊʣ

  32. • ෛՙࢼݧͷ࣌ʹ΋࢖ͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 • Apache͕ϘτϧωοΫʹͳ͍ͬͯͨ • MackerelΛΈͨΒ͙͢ʹΘ͔ͬͨʂʢ୲౰ऀஊʣ

  33. ʢMackerel౳Λݟͳ͕ΒΘ͍Θ͍΍͍ͬͯΔʣ

  34. 3. Mackerelར༻্ͷࠔΓ͝ͱ

  35. Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̍ʣ ͠ΐͬͪΎ͏ग़ͬͺͳ͠ͷΞϥʔτ र͏΂͖Ξϥʔτ͕ຒ΋ΕΔ ͱ͸͍͑؂ࢹΛશ͘੾Δͷ΋ɻɻɻ ௨஌νϟϯωϧΛ෼͚ΔϫʔΫΞϥ΢ϯυ ຊ౰͸ɺͦΜͳΞϥʔτ͸ແ͍Α͏ʹ͢Δ΂͖͕ͩ…

  36. Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̍ʣ ॏཁ౓ʹԠͯ͡௨஌νϟϯωϧΛ෼͚Δ #base_infra_alert ॏཁ౓ɿߴ #base_infra_alert_warn ॏཁ౓ɿத #base_infra_alert_info ॏཁ౓ɿ௿

  37. Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̎ʣ #base_infra_alert ʹ͸Mackerel͔Βͷ௨஌Ҏ֎ͷ΋ͷ΋ྲྀ͍ͯ͠Δ ϝʔϧͰདྷΔ΋ͷɺRSSϑΟʔυͳͲ ͜ΕΒʹ͸ mention ͸ͳ͍ͷͰɺSlackͰ͜ͷνϟϯωϧͷ௨஌ઃఆ Λڧͯ͘͠ؾ͚ͮΔΑ͏ʹ͢Δ͕…

  38. Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̎ʣ ௨஌͞Ε͗͢Δ #base_infra_alert ্ͷձ࿩͕શ෦௨஌͞ΕΔʂ #base_infra_alert ʹ͸௨஌Λྲྀ͚ͩ͢ʹ͢Δ ٞ࿦͸ผͷνϟϯωϧͰ ʮReacji ChannelerʯͰٞ࿦༻ͷνϟϯωϧʹ


    ϦϯΫΛίϐʔ https://get.slack.help/hc/ja/articles/360000482666-ଞνϟϯωϧ΁ϝοηʔδΛαΫοͱίϐʔ
  39. Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̎ʣ Ξϥʔτ௨஌ʹֆจࣈͰ൓Ԡ͢Δͱ…

  40. Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̎ʣ ٞ࿦༻ͷνϟϯωϧʹϦϯΫ͕ίϐʔ͞ΕΔʂ ௨஌͕ݮͬͯগ͠ฏ࿨ʹ…

  41. Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ αʔϏεͷ੒௕ͱͱ΋ʹαʔόͷछྨ͕૿͑ͨ αϒγεςϜͷ૿Ճ ϩʔϧͷ෼Խ ϚωʔδυαʔϏε౳ΛMackerelͷ؂ࢹର৅ʹ௥Ճ ϩʔϧ/ϗετͷߏ଄ʹ͖Ε͍ʹରԠ෇͚ʹ͍͘ Lambda, SQSͳͲ

  42. Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ MackerelͰ͸ʮϩʔϧʯ͕؂ࢹΛ͢Δ୯ҐͰ͋Γɺά ϥϑΛ·ͱΊΔ୯ҐͰ΋͋Δ ؂ࢹ͍ͨ͠୯ҐͱάϥϑͰݟ͍ͨ୯Ґ͸ඞͣ͠΋Ұக ͠ͳ͍ ؂ࢹ͍ͨ͠୯ҐͰϩʔϧΛ෇͚͍ͯ͘ͱɺ
 ʮαʔϏεʯͷը໘͕ͪ͝Όͪ͝Ό͖ͯͨ͠

  43. Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ

  44. Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ ୞ࠓϩʔϧ࠶ฤ੒த… Ұ؏ੑͷ͋Δϧʔϧʹج͍ͮͯઃܭ͠௚͢ ̎ͭͷʮαʔϏεʯΛ࢖ͬͯϩʔϧΛ੔ཧ͢Δ ϩʔϧάϥϑΛҰཡͰΈ͍ͨϩʔϧ → ϝΠϯͷʮαʔϏεʯ ྫɿdb-primary, db-replica

    ͦΕҎ֎ͷ؂ࢹ୯Ґͱͯ͠ͷϩʔϧ → αϒͷʮαʔϏεʯ ྫɿrds
  45. Mackerelར༻্ͷࠔΓ͝ͱ ΧελϜϝτϦοΫ΋ϩʔϧʹ࢒ͬͯ΄͍͠ αʔό͕ফ͑ΔͱΧελϜϝτϦοΫ͕ফ͑Δ σϓϩΠ͢Δͨͼʹαʔό͕ফ͑ΔͷͰΘΓͱਏ͍ γεςϜϝτϦοΫ͸࢒͍ͬͯΔ͕… ΧελϜϝτϦοΫ΋ϩʔϧʹ࢒ͬͯ΄͍͠ʂ ͳʹͱͧ…

  46. Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ աڈͷάϥϑͷϐʔΫ͕Θ͔Γʹ͍͘ ྫ͑͹ɺաڈ൒೥ͰҰ൪ϦΫΤετ͕དྷͯͨͱ͖ͷ෼ ؒϦΫΤετ਺͸Ͳͷ͘Β͍͔ͩͬͨͳʁͱ͍͏࣌ ͱΓ͋͑ͣʮ1YʯͷάϥϑΛݟͯΈΔ…ʁ

  47. Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ࣌ؒൣғΛ޿͘औΔͱϐʔΫ͕ͳ·Δ

  48. Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ࣌ؒൣғΛ޿͘औΔͱϐʔΫ͕ͳ·Δ

  49. Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ࣌ؒൣғΛ޿͘औΔͱϐʔΫ͕ͳ·Δ

  50. Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ࣌ؒൣғΛ޿͘औΔͱϐʔΫ͕ͳ·Δ

  51. Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ʮ1YʯͷάϥϑͰϐʔΫ͕ݟ͑ͯ΄͍͠ ͳʹͱͧ…

  52. 4. Mackerelͷ͍͍ͱ͜Ζ

  53. Mackerelͷ͍͍ͱ͜Ζ ࣗ෼Ͱӡ༻͠ͳ͍͍ͯ͘ ͲΜͲΜػೳڧԽ͞Ε͍ͯ͘ AWSΠϯςάϨʔγϣϯ͕ศར ϓϥάΠϯͷ௥Ճָ͕ https://github.com/mackerelio/plugin-registry https://github.com/mackerelio/mackerel-agent-plugins ࣗ࡞΋؆୯ ʮαϙʔτνʔϜ΁࿈བྷʯͷϨεϙϯε͕Α͍

  54. Mackerelͷ͍͍ͱ͜Ζ WebUI͕Θ͔Γ΍͍͢ άϥϑ͕͖Ε͍ API͕͋ͬͯϓϩάϥϜ͔Βૢ࡞Ͱ͖Δ CLI͕͋Δ ࢖͍͸͡Ί΍͍͢ ΤʔδΣϯτΛΠϯετʔϧͯ͠ϩʔϧΛઃఆ͢Δ͚ͩͰɺ
 ͍͍ͩͨྑ͍ײ͡ͷάϥϑ͕ग़ͯ͘Δ ೔ຊޠͩ UIɺυΩϡϝϯτ

  55. ࢀߟਤॻ ʮMackerel αʔό؂ࢹ [࣮ફ] ೖ໳ʯ ஶ: Ҫ্େีɼപ୩େีɼਿࢁ޿௨ɼాத৻࢘ɼ௶಺༎थɼদ໦խ޾ (http://gihyo.jp/book/2017/978-4-7741-9213-0) ʮೖ໳ ؂ࢹ

    ―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯʯ ஶ: Mike Julian (https://www.oreilly.co.jp/books/9784873118642/) ʮιϑτ΢ΣΞΤϯδχΞͷͨΊͷɹITΠϯϑϥ؂ࢹʦ࣮ફʧೖ໳ʯ ஶ: ࡈ౻༞Ұ࿠ (https://gihyo.jp/book/2016/978-4-7741-7865-3)
  56. We are hiring! SRE͸΋ͪΖΜɺ༷ʑͳ৬छΛืू͍ͯ͠·͢ ΧδϡΞϧʹձࣾʹ༡ͼʹདྷ͍ͨํ͸๻·Ͱ https://jobs.binc.jp/

  57. ςοΫϒϩάߋ৽ͯ͠·͢ https://devblog.thebase.in/

  58. Φ΢ϯυϝσΟΞߋ৽ͯ͠·͢ https://basebook.binc.jp/

  59. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠