BASEにおけるMackerel利用上の工夫と困りごとのご紹介 / Mackerel Meetup #13 Tokyo

Transcript

1. BASEʹ͓͚ΔMackerelར༻্ͷ޻෉ͱࠔΓ͝ͱ ͷ͝঺հ #"4&גࣜձࣾ
   43&
   ࢁ࢙ࠜਹ 
   
   .BDLFSFM
   .FFUVQ
   
   5PLZP

2. ຊ೔͸… BASEͰMackerelΛར༻͍ͯ͠ΔதͰͷɺͪΐͬͱͨ͠ ޻෉ɺࠔͬͨ͜ͱͱͦΕʹର͢ΔϫʔΫΞϥ΢ϯυɺ Mackerel΁ͷཁ๬ͳͲΛ঺հ͠·͢ ࢀߟʹͳΕ͹خ͍͠Ͱ͢

3. ࣗݾ঺հ ࢁ࢙ࠜਹʢ΍·Ͷ;Έ͔ͨʣ BASEגࣜձࣾɹSREάϧʔϓ 2016೥ೖࣾ ೖࣾ࣌ʹ͸Mackerel͕ಋೖࡁΈͩͬͨ લ৬ʢιʔγϟϧήʔϜʣͰ΋Mackerel࢖ͬͯͨ twitter / ͸ͯͳID: @fumikony

4. BASEͷϓϩμΫτʹ͍ͭͯ ωοτγϣοϓ࡞੒αʔϏε ʮBASEʯ γϣοϐϯάΞϓϦ ʮBASEʯ Ձ஋ͷަ׵ΛΑΓγϯϓϧʹ͠ɺ ੈքதͷਓʑ͕࠷దͳܦࡁ׆ಈΛߦ͑ΔΑ͏ʹ͢Δɻ MISSION

5. BASEͷϓϩμΫτʹ͍ͭͯ ωοτγϣοϓ࡞੒αʔϏε ʮBASEʯ γϣοϐϯάΞϓϦ ʮBASEʯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ

6. BASEͷϓϩμΫτʹ͍ͭͯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ αʔϏεͷن໛֦େʹͱ΋ͳͬͯো֐ͷӨڹ΋େ͖͘

7. BASEͷϓϩμΫτʹ͍ͭͯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ ҟৗ͕͋ͬͨΒ͙͢ʹؾ͖͍ͮͨ MackerelͰ؂ࢹ

8. ໨࣍ 1. BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ 2. Mackerelར༻্ͷ޻෉ 3. Mackerelར༻্ͷࠔΓ͝ͱ 4. Mackerelͷ͍͍ͱ͜Ζ

9. 1. BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ

10. BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ ֓ཁ ͍͍ͩͨAWS EC2, ECS, ELB, Lambda, ElastiCache, RDS, SQS,

    CloudFront, S3, SES, etc… MackerelͰ؂ࢹ SlackʹΞϥʔτ௨஌

11. BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ αʔόͷछྨ B/GσϓϩΠ͢Δαʔό σϓϩΠ͢Δͨͼʹ
 Πϯελϯε࡞੒ ݹ͍αʔό͸ফ͍͑ͯ͘ ΞϓϦέʔγϣϯαʔό
 ʢApache+PHPʣ

12. BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ αʔόͷछྨ B/GσϓϩΠ͠ͳ͍αʔό ͣͬͱ͋Δ όοναʔόͳͲ

13. 2. Mackerelར༻্ͷ޻෉

14. Mackerelར༻্ͷ޻෉ /health endpoint pattern OKͩͬͨΒεςʔλε200Λฦ͚ͩ͢ͷURL ֎ܗ؂ࢹͷର৅ /health ͱ /health/deep /health

    ͸αʔό/ίϯςφͷੜଘ /health/deep ͸DB౳΁ͷ઀ଓ·Ͱ֬ೝ ͪ͜ΒΛ֎ܗ؂ࢹͰ࢖༻
 
 See also: ʮೖ໳ ؂ࢹ ―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯʯ
 ɹɹɹɹɹ(https://www.oreilly.co.jp/books/9784873118642/)
 
 ɹɹɹɹɹஶ: Mike Julian 
 ɹɹɹɹɹ7.3 health ΤϯυϙΠϯτύλʔϯ

15. Mackerelར༻্ͷ޻෉ /health endpoint pattern ֎ܗ؂ࢹ͕Τϥʔʹͳͬͨͱ͖ͷ௨஌

16. Mackerelར༻্ͷ޻෉ monitoringαʔό MackerelͷΧελϜϝτϦοΫ΍αʔϏεϝτϦοΫΛऔಘɺ౤ ߘ͢ΔͨΊͷαʔό AWSΠϯςάϨʔγϣϯະରԠͷϚωʔδυαʔϏεͷϝτϦο ΫΛऔಘ ର৅ɿSES Ҏલ͸CloudFront΋ର৅͕ͩͬͨɺAWSΠϯςάϨʔγϣϯ͕ ରԠͨ͠ͷͰෆཁʹ

17. Mackerelར༻্ͷ޻෉ monitoringαʔό AWSΠϯςάϨʔγϣϯ+ϓϥάΠϯ ର৅: RDS AWSΠϯςάϨʔγϣϯͰϝτϦοΫΛऔಘ monitoringαʔόͰ΋mackerel-plugin-mysqlͰϝτϦοΫΛ औಘ custom_identifier Λࢦఆͯ͠ಉ͡ϗετʹू໿

18. Mackerelར༻্ͷ޻෉ monitoringαʔό [plugin.metrics.mysql-prd-main] command = "mackerel-plugin-mysql -host=db1.xxx.ap-northeast-1.rds.amazonaws.com" custom_identifier = "db1.xxx.ap-northeast-1.rds.amazonaws.com"

    See also: https://mackerel.io/ja/docs/entry/integrations/aws mackerel-agent.confʢൈਮʣ

19. Mackerelར༻্ͷ޻෉ ࣜʹΑΔ؂ࢹ scale(divide( host(xxxxxxxxxxx, custom.ses.stats.Bounces), host(xxxxxxxxxxx, custom.ses.stats.DeliveryAttempts)), 100) mackerel-plugin-sesͰऔಘͰ͖Δͷ͸഑৴ͱό΢ϯεͦΕͧΕͷ ݅਺ͳͷͰɺࣜͰܭࢉͨ͠஋Λ؂ࢹ

    ό΢ϯε݅਺ / ഑৴݅਺ * 100 See Also: https://mackerel.io/ja/docs/entry/expression-monitoring SESͷBounce Rate (%)

20. Mackerelར༻্ͷ޻෉ ࣜʹΑΔ؂ࢹ scale(divide( host(xxxxxxxxxxx, custom.ses.stats.Bounces), host(xxxxxxxxxxx, custom.ses.stats.DeliveryAttempts)), 100) SESͷBounce Rate

    (%)

21. Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ mkr MackerelͷCLIπʔϧ mkr monitors ؂ࢹϧʔϧΛૢ࡞͢ΔαϒίϚϯυ mkr monitors pull

    monitors.json ͱͯ͠؂ࢹϧʔϧΛอଘ mkr monitors diff monitors.json ͱ؂ࢹϧʔϧͷࠩ෼Λදࣔ mkr monitors push monitors.json Λ؂ࢹϧʔϧʹ൓ө

22. Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ mkr MackerelͷCLIπʔϧ mkr monitors ؂ࢹϧʔϧΛૢ࡞͢ΔαϒίϚϯυ mkr monitors pull

    monitors.json ͱͯ͠؂ࢹϧʔϧΛอଘ mkr monitors diff monitors.json ͱ؂ࢹϧʔϧͷࠩ෼Λදࣔ mkr monitors push monitors.json Λ؂ࢹϧʔϧʹ൓ө

23. Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ mkr monitors pull → git commit → GitHubʹgit

    push CircleCIͰσΠϦʔ࣮ߦ มߋཤྺΛ௥͑Δ ·͕ͪ͑ͯյͯ͠΋໭ͤΔ όοΫΞοϓతͳ࢖͍ํʹཹΊ͍ͯΔ mkr monitors push͸࢖ΘͣɺWebUIͰͷมߋͷΈ WebUI͸ศར

24. Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code

    || git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ

25. Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code

    || git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ

26. Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code

    || git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ

27. Mackerelར༻্ͷ޻෉ ໰͍߹Θͤ಺༰ΛSlackͰڞ༗ ʮϝʔϧઃఆʯͰɺ
 4MBDLͷʮ&NBJMʯ"QQͷ
 ϝʔϧΞυϨεΛઃఆ
    ໰͍߹Θͤͨ಺༰͕
 4MBDLʹྲྀΕ͖ͯͯศར

28. • ίϯςφΤʔδΣϯτࢼͯ͠Έ·ͨ͠ • ECSͷઃఆϛεΛൃݟʂ • ϝϞϦ͕଍Γͯͳ͔ͬͨʂ

29. • ϩʔϧ಺ҟৗݕ஌ ࢼͯ͠Έ·ͨ͠

30. • ෛՙࢼݧͷ࣌ʹ΋࢖ͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 ɹʮEίϚʔεϓϥοτϑΥʔϜͷ੒௕Λࢧ͑Δ
 ɹɹෛՙࢼݧͷऔΓ૊Έʯ •

31. • ෛՙࢼݧͷ࣌ʹ΋࢖ͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 • ApacheͷίωΫγϣϯ͕ுΓ෇͍ͯͨͷ͕Ϙτϧ ωοΫʹͳ͍ͬͯͨ • MackerelΛΈͨΒ͙͢ʹΘ͔ͬͨʂʢ୲౰ऀஊʣ

32. • ෛՙࢼݧͷ࣌ʹ΋࢖ͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 • Apache͕ϘτϧωοΫʹͳ͍ͬͯͨ • MackerelΛΈͨΒ͙͢ʹΘ͔ͬͨʂʢ୲౰ऀஊʣ

33. ʢMackerel౳Λݟͳ͕ΒΘ͍Θ͍΍͍ͬͯΔʣ

34. 3. Mackerelར༻্ͷࠔΓ͝ͱ

35. Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̍ʣ ͠ΐͬͪΎ͏ग़ͬͺͳ͠ͷΞϥʔτ र͏΂͖Ξϥʔτ͕ຒ΋ΕΔ ͱ͸͍͑؂ࢹΛશ͘੾Δͷ΋ɻɻɻ ௨஌νϟϯωϧΛ෼͚ΔϫʔΫΞϥ΢ϯυ ຊ౰͸ɺͦΜͳΞϥʔτ͸ແ͍Α͏ʹ͢Δ΂͖͕ͩ…

36. Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̍ʣ ॏཁ౓ʹԠͯ͡௨஌νϟϯωϧΛ෼͚Δ #base_infra_alert ॏཁ౓ɿߴ #base_infra_alert_warn ॏཁ౓ɿத #base_infra_alert_info ॏཁ౓ɿ௿

37. Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̎ʣ #base_infra_alert ʹ͸Mackerel͔Βͷ௨஌Ҏ֎ͷ΋ͷ΋ྲྀ͍ͯ͠Δ ϝʔϧͰདྷΔ΋ͷɺRSSϑΟʔυͳͲ ͜ΕΒʹ͸ mention ͸ͳ͍ͷͰɺSlackͰ͜ͷνϟϯωϧͷ௨஌ઃఆ Λڧͯ͘͠ؾ͚ͮΔΑ͏ʹ͢Δ͕…

38. Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̎ʣ ௨஌͞Ε͗͢Δ #base_infra_alert ্ͷձ࿩͕શ෦௨஌͞ΕΔʂ #base_infra_alert ʹ͸௨஌Λྲྀ͚ͩ͢ʹ͢Δ ٞ࿦͸ผͷνϟϯωϧͰ ʮReacji ChannelerʯͰٞ࿦༻ͷνϟϯωϧʹ


    ϦϯΫΛίϐʔ https://get.slack.help/hc/ja/articles/360000482666-ଞνϟϯωϧ΁ϝοηʔδΛαΫοͱίϐʔ

39. Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̎ʣ Ξϥʔτ௨஌ʹֆจࣈͰ൓Ԡ͢Δͱ…

40. Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̎ʣ ٞ࿦༻ͷνϟϯωϧʹϦϯΫ͕ίϐʔ͞ΕΔʂ ௨஌͕ݮͬͯগ͠ฏ࿨ʹ…

41. Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ αʔϏεͷ੒௕ͱͱ΋ʹαʔόͷछྨ͕૿͑ͨ αϒγεςϜͷ૿Ճ ϩʔϧͷ෼Խ ϚωʔδυαʔϏε౳ΛMackerelͷ؂ࢹର৅ʹ௥Ճ ϩʔϧ/ϗετͷߏ଄ʹ͖Ε͍ʹରԠ෇͚ʹ͍͘ Lambda, SQSͳͲ

42. Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ MackerelͰ͸ʮϩʔϧʯ͕؂ࢹΛ͢Δ୯ҐͰ͋Γɺά ϥϑΛ·ͱΊΔ୯ҐͰ΋͋Δ ؂ࢹ͍ͨ͠୯ҐͱάϥϑͰݟ͍ͨ୯Ґ͸ඞͣ͠΋Ұக ͠ͳ͍ ؂ࢹ͍ͨ͠୯ҐͰϩʔϧΛ෇͚͍ͯ͘ͱɺ
 ʮαʔϏεʯͷը໘͕ͪ͝Όͪ͝Ό͖ͯͨ͠

43. Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ

44. Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ ୞ࠓϩʔϧ࠶ฤ੒த… Ұ؏ੑͷ͋Δϧʔϧʹج͍ͮͯઃܭ͠௚͢ ̎ͭͷʮαʔϏεʯΛ࢖ͬͯϩʔϧΛ੔ཧ͢Δ ϩʔϧάϥϑΛҰཡͰΈ͍ͨϩʔϧ → ϝΠϯͷʮαʔϏεʯ ྫɿdb-primary, db-replica

    ͦΕҎ֎ͷ؂ࢹ୯Ґͱͯ͠ͷϩʔϧ → αϒͷʮαʔϏεʯ ྫɿrds

45. Mackerelར༻্ͷࠔΓ͝ͱ ΧελϜϝτϦοΫ΋ϩʔϧʹ࢒ͬͯ΄͍͠ αʔό͕ফ͑ΔͱΧελϜϝτϦοΫ͕ফ͑Δ σϓϩΠ͢Δͨͼʹαʔό͕ফ͑ΔͷͰΘΓͱਏ͍ γεςϜϝτϦοΫ͸࢒͍ͬͯΔ͕… ΧελϜϝτϦοΫ΋ϩʔϧʹ࢒ͬͯ΄͍͠ʂ ͳʹͱͧ…

46. Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ աڈͷάϥϑͷϐʔΫ͕Θ͔Γʹ͍͘ ྫ͑͹ɺաڈ൒೥ͰҰ൪ϦΫΤετ͕དྷͯͨͱ͖ͷ෼ ؒϦΫΤετ਺͸Ͳͷ͘Β͍͔ͩͬͨͳʁͱ͍͏࣌ ͱΓ͋͑ͣʮ1YʯͷάϥϑΛݟͯΈΔ…ʁ

47. Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ࣌ؒൣғΛ޿͘औΔͱϐʔΫ͕ͳ·Δ

48. Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ࣌ؒൣғΛ޿͘औΔͱϐʔΫ͕ͳ·Δ

49. Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ࣌ؒൣғΛ޿͘औΔͱϐʔΫ͕ͳ·Δ

50. Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ࣌ؒൣғΛ޿͘औΔͱϐʔΫ͕ͳ·Δ

51. Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ʮ1YʯͷάϥϑͰϐʔΫ͕ݟ͑ͯ΄͍͠ ͳʹͱͧ…

52. 4. Mackerelͷ͍͍ͱ͜Ζ

53. Mackerelͷ͍͍ͱ͜Ζ ࣗ෼Ͱӡ༻͠ͳ͍͍ͯ͘ ͲΜͲΜػೳڧԽ͞Ε͍ͯ͘ AWSΠϯςάϨʔγϣϯ͕ศར ϓϥάΠϯͷ௥Ճָ͕ https://github.com/mackerelio/plugin-registry https://github.com/mackerelio/mackerel-agent-plugins ࣗ࡞΋؆୯ ʮαϙʔτνʔϜ΁࿈བྷʯͷϨεϙϯε͕Α͍

54. Mackerelͷ͍͍ͱ͜Ζ WebUI͕Θ͔Γ΍͍͢ άϥϑ͕͖Ε͍ API͕͋ͬͯϓϩάϥϜ͔Βૢ࡞Ͱ͖Δ CLI͕͋Δ ࢖͍͸͡Ί΍͍͢ ΤʔδΣϯτΛΠϯετʔϧͯ͠ϩʔϧΛઃఆ͢Δ͚ͩͰɺ
 ͍͍ͩͨྑ͍ײ͡ͷάϥϑ͕ग़ͯ͘Δ ೔ຊޠͩ UIɺυΩϡϝϯτ

55. ࢀߟਤॻ ʮMackerel αʔό؂ࢹ [࣮ફ] ೖ໳ʯ ஶ: Ҫ্େีɼപ୩େีɼਿࢁ޿௨ɼాத৻࢘ɼ௶಺༎थɼদ໦խ޾ (http://gihyo.jp/book/2017/978-4-7741-9213-0) ʮೖ໳ ؂ࢹ

    ―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯʯ ஶ: Mike Julian (https://www.oreilly.co.jp/books/9784873118642/) ʮιϑτ΢ΣΞΤϯδχΞͷͨΊͷɹITΠϯϑϥ؂ࢹʦ࣮ફʧೖ໳ʯ ஶ: ࡈ౻༞Ұ࿠ (https://gihyo.jp/book/2016/978-4-7741-7865-3)

56. We are hiring! SRE͸΋ͪΖΜɺ༷ʑͳ৬छΛืू͍ͯ͠·͢ ΧδϡΞϧʹձࣾʹ༡ͼʹདྷ͍ͨํ͸๻·Ͱ https://jobs.binc.jp/

57. ςοΫϒϩάߋ৽ͯ͠·͢ https://devblog.thebase.in/

58. Φ΢ϯυϝσΟΞߋ৽ͯ͠·͢ https://basebook.binc.jp/

59. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠