BASEにおけるMackerel利用上の工夫と困りごとのご紹介 / Mackerel Meetup #13 Tokyo

2019/03/01 Mackerel Meetup #13 Tokyo

Transcript

1. BASEʹ͓͚ΔMackerelར༻্ͷ޻෉ͱࠔΓ͝ͱ ͷ͝঺հ #"4&גࣜձࣾ
   43&
   ࢁ࢙ࠜਹ 
   
   .BDLFSFM
   .FFUVQ
   
   5PLZP
2. ຊ೔͸… BASEͰMackerelΛར༻͍ͯ͠ΔதͰͷɺͪΐͬͱͨ͠ ޻෉ɺࠔͬͨ͜ͱͱͦΕʹର͢ΔϫʔΫΞϥ΢ϯυɺ Mackerel΁ͷཁ๬ͳͲΛ঺հ͠·͢ ࢀߟʹͳΕ͹خ͍͠Ͱ͢
3. ࣗݾ঺հ ࢁ࢙ࠜਹʢ΍·Ͷ;Έ͔ͨʣ BASEגࣜձࣾɹSREάϧʔϓ 2016೥ೖࣾ ೖࣾ࣌ʹ͸Mackerel͕ಋೖࡁΈͩͬͨ લ৬ʢιʔγϟϧήʔϜʣͰ΋Mackerel࢖ͬͯͨ twitter / ͸ͯͳID: @fumikony
4. BASEͷϓϩμΫτʹ͍ͭͯ ωοτγϣοϓ࡞੒αʔϏε ʮBASEʯ γϣοϐϯάΞϓϦ ʮBASEʯ Ձ஋ͷަ׵ΛΑΓγϯϓϧʹ͠ɺ ੈքதͷਓʑ͕࠷దͳܦࡁ׆ಈΛߦ͑ΔΑ͏ʹ͢Δɻ MISSION
5. BASEͷϓϩμΫτʹ͍ͭͯ ωοτγϣοϓ࡞੒αʔϏε ʮBASEʯ γϣοϐϯάΞϓϦ ʮBASEʯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ
6. BASEͷϓϩμΫτʹ͍ͭͯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ αʔϏεͷن໛֦େʹͱ΋ͳͬͯো֐ͷӨڹ΋େ͖͘
7. BASEͷϓϩμΫτʹ͍ͭͯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ ҟৗ͕͋ͬͨΒ͙͢ʹؾ͖͍ͮͨ MackerelͰ؂ࢹ
8. ໨࣍ 1. BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ 2. Mackerelར༻্ͷ޻෉ 3. Mackerelར༻্ͷࠔΓ͝ͱ 4. Mackerelͷ͍͍ͱ͜Ζ
9. 1. BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ
10. BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ ֓ཁ ͍͍ͩͨAWS EC2, ECS, ELB, Lambda, ElastiCache, RDS, SQS, CloudFront, S3, SES, etc… MackerelͰ؂ࢹ SlackʹΞϥʔτ௨஌
11. BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ αʔόͷछྨ B/GσϓϩΠ͢Δαʔό σϓϩΠ͢Δͨͼʹ
 Πϯελϯε࡞੒ ݹ͍αʔό͸ফ͍͑ͯ͘ ΞϓϦέʔγϣϯαʔό
 ʢApache+PHPʣ
12. BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ αʔόͷछྨ B/GσϓϩΠ͠ͳ͍αʔό ͣͬͱ͋Δ όοναʔόͳͲ
13. 2. Mackerelར༻্ͷ޻෉
14. Mackerelར༻্ͷ޻෉ /health endpoint pattern OKͩͬͨΒεςʔλε200Λฦ͚ͩ͢ͷURL ֎ܗ؂ࢹͷର৅ /health ͱ /health/deep /health ͸αʔό/ίϯςφͷੜଘ /health/deep ͸DB౳΁ͷ઀ଓ·Ͱ֬ೝ ͪ͜ΒΛ֎ܗ؂ࢹͰ࢖༻
 
 See also: ʮೖ໳ ؂ࢹ ―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯʯ
 ɹɹɹɹɹ(https://www.oreilly.co.jp/books/9784873118642/)
 
 ɹɹɹɹɹஶ: Mike Julian 
 ɹɹɹɹɹ7.3 health ΤϯυϙΠϯτύλʔϯ
15. Mackerelར༻্ͷ޻෉ /health endpoint pattern ֎ܗ؂ࢹ͕Τϥʔʹͳͬͨͱ͖ͷ௨஌
16. Mackerelར༻্ͷ޻෉ monitoringαʔό MackerelͷΧελϜϝτϦοΫ΍αʔϏεϝτϦοΫΛऔಘɺ౤ ߘ͢ΔͨΊͷαʔό AWSΠϯςάϨʔγϣϯະରԠͷϚωʔδυαʔϏεͷϝτϦο ΫΛऔಘ ର৅ɿSES Ҏલ͸CloudFront΋ର৅͕ͩͬͨɺAWSΠϯςάϨʔγϣϯ͕ ରԠͨ͠ͷͰෆཁʹ
17. Mackerelར༻্ͷ޻෉ monitoringαʔό AWSΠϯςάϨʔγϣϯ+ϓϥάΠϯ ର৅: RDS AWSΠϯςάϨʔγϣϯͰϝτϦοΫΛऔಘ monitoringαʔόͰ΋mackerel-plugin-mysqlͰϝτϦοΫΛ औಘ custom_identifier Λࢦఆͯ͠ಉ͡ϗετʹू໿
18. Mackerelར༻্ͷ޻෉ monitoringαʔό [plugin.metrics.mysql-prd-main] command = "mackerel-plugin-mysql -host=db1.xxx.ap-northeast-1.rds.amazonaws.com" custom_identifier = "db1.xxx.ap-northeast-1.rds.amazonaws.com" See also: https://mackerel.io/ja/docs/entry/integrations/aws mackerel-agent.confʢൈਮʣ
19. Mackerelར༻্ͷ޻෉ ࣜʹΑΔ؂ࢹ scale(divide( host(xxxxxxxxxxx, custom.ses.stats.Bounces), host(xxxxxxxxxxx, custom.ses.stats.DeliveryAttempts)), 100) mackerel-plugin-sesͰऔಘͰ͖Δͷ͸഑৴ͱό΢ϯεͦΕͧΕͷ ݅਺ͳͷͰɺࣜͰܭࢉͨ͠஋Λ؂ࢹ ό΢ϯε݅਺ / ഑৴݅਺ * 100 See Also: https://mackerel.io/ja/docs/entry/expression-monitoring SESͷBounce Rate (%)
20. Mackerelར༻্ͷ޻෉ ࣜʹΑΔ؂ࢹ scale(divide( host(xxxxxxxxxxx, custom.ses.stats.Bounces), host(xxxxxxxxxxx, custom.ses.stats.DeliveryAttempts)), 100) SESͷBounce Rate (%)
21. Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ mkr MackerelͷCLIπʔϧ mkr monitors ؂ࢹϧʔϧΛૢ࡞͢ΔαϒίϚϯυ mkr monitors pull monitors.json ͱͯ͠؂ࢹϧʔϧΛอଘ mkr monitors diff monitors.json ͱ؂ࢹϧʔϧͷࠩ෼Λදࣔ mkr monitors push monitors.json Λ؂ࢹϧʔϧʹ൓ө
22. Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ mkr MackerelͷCLIπʔϧ mkr monitors ؂ࢹϧʔϧΛૢ࡞͢ΔαϒίϚϯυ mkr monitors pull monitors.json ͱͯ͠؂ࢹϧʔϧΛอଘ mkr monitors diff monitors.json ͱ؂ࢹϧʔϧͷࠩ෼Λදࣔ mkr monitors push monitors.json Λ؂ࢹϧʔϧʹ൓ө
23. Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ mkr monitors pull → git commit → GitHubʹgit push CircleCIͰσΠϦʔ࣮ߦ มߋཤྺΛ௥͑Δ ·͕ͪ͑ͯյͯ͠΋໭ͤΔ όοΫΞοϓతͳ࢖͍ํʹཹΊ͍ͯΔ mkr monitors push͸࢖ΘͣɺWebUIͰͷมߋͷΈ WebUI͸ศར
24. Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code || git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
25. Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code || git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
26. Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code || git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
27. Mackerelར༻্ͷ޻෉ ໰͍߹Θͤ಺༰ΛSlackͰڞ༗ ʮϝʔϧઃఆʯͰɺ
 4MBDLͷʮ&NBJMʯ"QQͷ
 ϝʔϧΞυϨεΛઃఆ
    ໰͍߹Θͤͨ಺༰͕
 4MBDLʹྲྀΕ͖ͯͯศར
28. • ίϯςφΤʔδΣϯτࢼͯ͠Έ·ͨ͠ • ECSͷઃఆϛεΛൃݟʂ • ϝϞϦ͕଍Γͯͳ͔ͬͨʂ
29. • ϩʔϧ಺ҟৗݕ஌ ࢼͯ͠Έ·ͨ͠
30. • ෛՙࢼݧͷ࣌ʹ΋࢖ͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 ɹʮEίϚʔεϓϥοτϑΥʔϜͷ੒௕Λࢧ͑Δ
 ɹɹෛՙࢼݧͷऔΓ૊Έʯ •
31. • ෛՙࢼݧͷ࣌ʹ΋࢖ͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 • ApacheͷίωΫγϣϯ͕ுΓ෇͍ͯͨͷ͕Ϙτϧ ωοΫʹͳ͍ͬͯͨ • MackerelΛΈͨΒ͙͢ʹΘ͔ͬͨʂʢ୲౰ऀஊʣ
32. • ෛՙࢼݧͷ࣌ʹ΋࢖ͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 • Apache͕ϘτϧωοΫʹͳ͍ͬͯͨ • MackerelΛΈͨΒ͙͢ʹΘ͔ͬͨʂʢ୲౰ऀஊʣ
33. ʢMackerel౳Λݟͳ͕ΒΘ͍Θ͍΍͍ͬͯΔʣ
34. 3. Mackerelར༻্ͷࠔΓ͝ͱ
35. Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̍ʣ ͠ΐͬͪΎ͏ग़ͬͺͳ͠ͷΞϥʔτ र͏΂͖Ξϥʔτ͕ຒ΋ΕΔ ͱ͸͍͑؂ࢹΛશ͘੾Δͷ΋ɻɻɻ ௨஌νϟϯωϧΛ෼͚ΔϫʔΫΞϥ΢ϯυ ຊ౰͸ɺͦΜͳΞϥʔτ͸ແ͍Α͏ʹ͢Δ΂͖͕ͩ…
36. Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̍ʣ ॏཁ౓ʹԠͯ͡௨஌νϟϯωϧΛ෼͚Δ #base_infra_alert ॏཁ౓ɿߴ #base_infra_alert_warn ॏཁ౓ɿத #base_infra_alert_info ॏཁ౓ɿ௿
37. Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̎ʣ #base_infra_alert ʹ͸Mackerel͔Βͷ௨஌Ҏ֎ͷ΋ͷ΋ྲྀ͍ͯ͠Δ ϝʔϧͰདྷΔ΋ͷɺRSSϑΟʔυͳͲ ͜ΕΒʹ͸ mention ͸ͳ͍ͷͰɺSlackͰ͜ͷνϟϯωϧͷ௨஌ઃఆ Λڧͯ͘͠ؾ͚ͮΔΑ͏ʹ͢Δ͕…
38. Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̎ʣ ௨஌͞Ε͗͢Δ #base_infra_alert ্ͷձ࿩͕શ෦௨஌͞ΕΔʂ #base_infra_alert ʹ͸௨஌Λྲྀ͚ͩ͢ʹ͢Δ ٞ࿦͸ผͷνϟϯωϧͰ ʮReacji ChannelerʯͰٞ࿦༻ͷνϟϯωϧʹ
 ϦϯΫΛίϐʔ https://get.slack.help/hc/ja/articles/360000482666-ଞνϟϯωϧ΁ϝοηʔδΛαΫοͱίϐʔ
39. Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̎ʣ Ξϥʔτ௨஌ʹֆจࣈͰ൓Ԡ͢Δͱ…
40. Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̎ʣ ٞ࿦༻ͷνϟϯωϧʹϦϯΫ͕ίϐʔ͞ΕΔʂ ௨஌͕ݮͬͯগ͠ฏ࿨ʹ…
41. Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ αʔϏεͷ੒௕ͱͱ΋ʹαʔόͷछྨ͕૿͑ͨ αϒγεςϜͷ૿Ճ ϩʔϧͷ෼Խ ϚωʔδυαʔϏε౳ΛMackerelͷ؂ࢹର৅ʹ௥Ճ ϩʔϧ/ϗετͷߏ଄ʹ͖Ε͍ʹରԠ෇͚ʹ͍͘ Lambda, SQSͳͲ
42. Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ MackerelͰ͸ʮϩʔϧʯ͕؂ࢹΛ͢Δ୯ҐͰ͋Γɺά ϥϑΛ·ͱΊΔ୯ҐͰ΋͋Δ ؂ࢹ͍ͨ͠୯ҐͱάϥϑͰݟ͍ͨ୯Ґ͸ඞͣ͠΋Ұக ͠ͳ͍ ؂ࢹ͍ͨ͠୯ҐͰϩʔϧΛ෇͚͍ͯ͘ͱɺ
 ʮαʔϏεʯͷը໘͕ͪ͝Όͪ͝Ό͖ͯͨ͠
43. Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ
44. Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ ୞ࠓϩʔϧ࠶ฤ੒த… Ұ؏ੑͷ͋Δϧʔϧʹج͍ͮͯઃܭ͠௚͢ ̎ͭͷʮαʔϏεʯΛ࢖ͬͯϩʔϧΛ੔ཧ͢Δ ϩʔϧάϥϑΛҰཡͰΈ͍ͨϩʔϧ → ϝΠϯͷʮαʔϏεʯ ྫɿdb-primary, db-replica ͦΕҎ֎ͷ؂ࢹ୯Ґͱͯ͠ͷϩʔϧ → αϒͷʮαʔϏεʯ ྫɿrds
45. Mackerelར༻্ͷࠔΓ͝ͱ ΧελϜϝτϦοΫ΋ϩʔϧʹ࢒ͬͯ΄͍͠ αʔό͕ফ͑ΔͱΧελϜϝτϦοΫ͕ফ͑Δ σϓϩΠ͢Δͨͼʹαʔό͕ফ͑ΔͷͰΘΓͱਏ͍ γεςϜϝτϦοΫ͸࢒͍ͬͯΔ͕… ΧελϜϝτϦοΫ΋ϩʔϧʹ࢒ͬͯ΄͍͠ʂ ͳʹͱͧ…
46. Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ աڈͷάϥϑͷϐʔΫ͕Θ͔Γʹ͍͘ ྫ͑͹ɺաڈ൒೥ͰҰ൪ϦΫΤετ͕དྷͯͨͱ͖ͷ෼ ؒϦΫΤετ਺͸Ͳͷ͘Β͍͔ͩͬͨͳʁͱ͍͏࣌ ͱΓ͋͑ͣʮ1YʯͷάϥϑΛݟͯΈΔ…ʁ
47. Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ࣌ؒൣғΛ޿͘औΔͱϐʔΫ͕ͳ·Δ
48. Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ࣌ؒൣғΛ޿͘औΔͱϐʔΫ͕ͳ·Δ
49. Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ࣌ؒൣғΛ޿͘औΔͱϐʔΫ͕ͳ·Δ
50. Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ࣌ؒൣғΛ޿͘औΔͱϐʔΫ͕ͳ·Δ
51. Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ʮ1YʯͷάϥϑͰϐʔΫ͕ݟ͑ͯ΄͍͠ ͳʹͱͧ…
52. 4. Mackerelͷ͍͍ͱ͜Ζ
53. Mackerelͷ͍͍ͱ͜Ζ ࣗ෼Ͱӡ༻͠ͳ͍͍ͯ͘ ͲΜͲΜػೳڧԽ͞Ε͍ͯ͘ AWSΠϯςάϨʔγϣϯ͕ศར ϓϥάΠϯͷ௥Ճָ͕ https://github.com/mackerelio/plugin-registry https://github.com/mackerelio/mackerel-agent-plugins ࣗ࡞΋؆୯ ʮαϙʔτνʔϜ΁࿈བྷʯͷϨεϙϯε͕Α͍
54. Mackerelͷ͍͍ͱ͜Ζ WebUI͕Θ͔Γ΍͍͢ άϥϑ͕͖Ε͍ API͕͋ͬͯϓϩάϥϜ͔Βૢ࡞Ͱ͖Δ CLI͕͋Δ ࢖͍͸͡Ί΍͍͢ ΤʔδΣϯτΛΠϯετʔϧͯ͠ϩʔϧΛઃఆ͢Δ͚ͩͰɺ
 ͍͍ͩͨྑ͍ײ͡ͷάϥϑ͕ग़ͯ͘Δ ೔ຊޠͩ UIɺυΩϡϝϯτ
55. ࢀߟਤॻ ʮMackerel αʔό؂ࢹ [࣮ફ] ೖ໳ʯ ஶ: Ҫ্େีɼപ୩େีɼਿࢁ޿௨ɼాத৻࢘ɼ௶಺༎थɼদ໦խ޾ (http://gihyo.jp/book/2017/978-4-7741-9213-0) ʮೖ໳ ؂ࢹ ―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯʯ ஶ: Mike Julian (https://www.oreilly.co.jp/books/9784873118642/) ʮιϑτ΢ΣΞΤϯδχΞͷͨΊͷɹITΠϯϑϥ؂ࢹʦ࣮ફʧೖ໳ʯ ஶ: ࡈ౻༞Ұ࿠ (https://gihyo.jp/book/2016/978-4-7741-7865-3)
56. We are hiring! SRE͸΋ͪΖΜɺ༷ʑͳ৬छΛืू͍ͯ͠·͢ ΧδϡΞϧʹձࣾʹ༡ͼʹདྷ͍ͨํ͸๻·Ͱ https://jobs.binc.jp/
57. ςοΫϒϩάߋ৽ͯ͠·͢ https://devblog.thebase.in/
58. Φ΢ϯυϝσΟΞߋ৽ͯ͠·͢ https://basebook.binc.jp/
59. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠