BASEにおけるMackerel利用上の工夫と困りごとのご紹介 / Mackerel Meetup #13 Tokyo

by Fumitaka Yamane

Slide 1

Slide 1 text

BASEʹ͓͚ΔMackerelར༻্ͷ޻෉ͱࠔΓ͝ͱ ͷ͝঺հ #"4&גࣜձࣾ43& ࢁ࢙ࠜਹ .BDLFSFM.FFUVQ5PLZP

Slide 2

Slide 2 text

ຊ೔͸… BASEͰMackerelΛར༻͍ͯ͠ΔதͰͷɺͪΐͬͱͨ͠ ޻෉ɺࠔͬͨ͜ͱͱͦΕʹର͢ΔϫʔΫΞϥ΢ϯυɺ Mackerel΁ͷཁ๬ͳͲΛ঺հ͠·͢ ࢀߟʹͳΕ͹خ͍͠Ͱ͢

Slide 3

Slide 3 text

ࣗݾ঺հ ࢁ࢙ࠜਹʢ΍·Ͷ;Έ͔ͨʣ BASEגࣜձࣾɹSREάϧʔϓ 2016೥ೖࣾ ೖࣾ࣌ʹ͸Mackerel͕ಋೖࡁΈͩͬͨ લ৬ʢιʔγϟϧήʔϜʣͰ΋Mackerel࢖ͬͯͨ twitter / ͸ͯͳID: @fumikony

Slide 4

Slide 4 text

BASEͷϓϩμΫτʹ͍ͭͯ ωοτγϣοϓ࡞੒αʔϏε ʮBASEʯ γϣοϐϯάΞϓϦ ʮBASEʯ Ձ஋ͷަ׵ΛΑΓγϯϓϧʹ͠ɺ ੈքதͷਓʑ͕࠷దͳܦࡁ׆ಈΛߦ͑ΔΑ͏ʹ͢Δɻ MISSION

Slide 5

Slide 5 text

BASEͷϓϩμΫτʹ͍ͭͯ ωοτγϣοϓ࡞੒αʔϏε ʮBASEʯ γϣοϐϯάΞϓϦ ʮBASEʯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ

Slide 6

Slide 6 text

BASEͷϓϩμΫτʹ͍ͭͯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ αʔϏεͷن໛֦େʹͱ΋ͳͬͯো֐ͷӨڹ΋େ͖͘

Slide 7

Slide 7 text

BASEͷϓϩμΫτʹ͍ͭͯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ ҟৗ͕͋ͬͨΒ͙͢ʹؾ͖͍ͮͨ MackerelͰ؂ࢹ

Slide 8

Slide 8 text

໨࣍ 1. BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ 2. Mackerelར༻্ͷ޻෉ 3. Mackerelར༻্ͷࠔΓ͝ͱ 4. Mackerelͷ͍͍ͱ͜Ζ

Slide 9

Slide 9 text

1. BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ

Slide 10

Slide 10 text

BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ ֓ཁ ͍͍ͩͨAWS EC2, ECS, ELB, Lambda, ElastiCache, RDS, SQS, CloudFront, S3, SES, etc… MackerelͰ؂ࢹ SlackʹΞϥʔτ௨஌

Slide 11

Slide 11 text

BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ αʔόͷछྨ B/GσϓϩΠ͢Δαʔό σϓϩΠ͢Δͨͼʹ  Πϯελϯε࡞੒ ݹ͍αʔό͸ফ͍͑ͯ͘ ΞϓϦέʔγϣϯαʔό  ʢApache+PHPʣ

Slide 12

Slide 12 text

BASEͷΠϯϑϥߏ੒ʹ͍ͭͯ αʔόͷछྨ B/GσϓϩΠ͠ͳ͍αʔό ͣͬͱ͋Δ όοναʔόͳͲ

Slide 13

Slide 13 text

2. Mackerelར༻্ͷ޻෉

Slide 14

Slide 14 text

Mackerelར༻্ͷ޻෉ /health endpoint pattern OKͩͬͨΒεςʔλε200Λฦ͚ͩ͢ͷURL ֎ܗ؂ࢹͷର৅ /health ͱ /health/deep /health ͸αʔό/ίϯςφͷੜଘ /health/deep ͸DB౳΁ͷ઀ଓ·Ͱ֬ೝ ͪ͜ΒΛ֎ܗ؂ࢹͰ࢖༻    See also: ʮೖ໳ ؂ࢹ ―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯʯ  ɹɹɹɹɹ(https://www.oreilly.co.jp/books/9784873118642/)    ɹɹɹɹɹஶ: Mike Julian   ɹɹɹɹɹ7.3 health ΤϯυϙΠϯτύλʔϯ

Slide 15

Slide 15 text

Mackerelར༻্ͷ޻෉ /health endpoint pattern ֎ܗ؂ࢹ͕Τϥʔʹͳͬͨͱ͖ͷ௨஌

Slide 16

Slide 16 text

Mackerelར༻্ͷ޻෉ monitoringαʔό MackerelͷΧελϜϝτϦοΫ΍αʔϏεϝτϦοΫΛऔಘɺ౤ ߘ͢ΔͨΊͷαʔό AWSΠϯςάϨʔγϣϯະରԠͷϚωʔδυαʔϏεͷϝτϦο ΫΛऔಘ ର৅ɿSES Ҏલ͸CloudFront΋ର৅͕ͩͬͨɺAWSΠϯςάϨʔγϣϯ͕ ରԠͨ͠ͷͰෆཁʹ

Slide 17

Slide 17 text

Mackerelར༻্ͷ޻෉ monitoringαʔό AWSΠϯςάϨʔγϣϯ+ϓϥάΠϯ ର৅: RDS AWSΠϯςάϨʔγϣϯͰϝτϦοΫΛऔಘ monitoringαʔόͰ΋mackerel-plugin-mysqlͰϝτϦοΫΛ औಘ custom_identiﬁer Λࢦఆͯ͠ಉ͡ϗετʹू໿

Slide 18

Slide 18 text

Mackerelར༻্ͷ޻෉ monitoringαʔό [plugin.metrics.mysql-prd-main] command = "mackerel-plugin-mysql -host=db1.xxx.ap-northeast-1.rds.amazonaws.com" custom_identifier = "db1.xxx.ap-northeast-1.rds.amazonaws.com" See also: https://mackerel.io/ja/docs/entry/integrations/aws mackerel-agent.confʢൈਮʣ

Slide 19

Slide 19 text

Mackerelར༻্ͷ޻෉ ࣜʹΑΔ؂ࢹ scale(divide( host(xxxxxxxxxxx, custom.ses.stats.Bounces), host(xxxxxxxxxxx, custom.ses.stats.DeliveryAttempts)), 100) mackerel-plugin-sesͰऔಘͰ͖Δͷ͸഑৴ͱό΢ϯεͦΕͧΕͷ ݅਺ͳͷͰɺࣜͰܭࢉͨ͠஋Λ؂ࢹ ό΢ϯε݅਺ / ഑৴݅਺ * 100 See Also: https://mackerel.io/ja/docs/entry/expression-monitoring SESͷBounce Rate (%)

Slide 20

Slide 20 text

Mackerelར༻্ͷ޻෉ ࣜʹΑΔ؂ࢹ scale(divide( host(xxxxxxxxxxx, custom.ses.stats.Bounces), host(xxxxxxxxxxx, custom.ses.stats.DeliveryAttempts)), 100) SESͷBounce Rate (%)

Slide 21

Slide 21 text

Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ mkr MackerelͷCLIπʔϧ mkr monitors ؂ࢹϧʔϧΛૢ࡞͢ΔαϒίϚϯυ mkr monitors pull monitors.json ͱͯ͠؂ࢹϧʔϧΛอଘ mkr monitors diff monitors.json ͱ؂ࢹϧʔϧͷࠩ෼Λදࣔ mkr monitors push monitors.json Λ؂ࢹϧʔϧʹ൓ө

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ mkr monitors pull → git commit → GitHubʹgit push CircleCIͰσΠϦʔ࣮ߦ มߋཤྺΛ௥͑Δ ·͕ͪ͑ͯյͯ͠΋໭ͤΔ όοΫΞοϓతͳ࢖͍ํʹཹΊ͍ͯΔ mkr monitors push͸࢖ΘͣɺWebUIͰͷมߋͷΈ WebUI͸ศར

Slide 24

Slide 24 text

Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code || git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ

Slide 25

Slide 25 text

Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code || git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ

Slide 26

Slide 26 text

Mackerelར༻্ͷ޻෉ ؂ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code || git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ

Slide 27

Slide 27 text

Mackerelར༻্ͷ޻෉ ໰͍߹Θͤ಺༰ΛSlackͰڞ༗ ʮϝʔϧઃఆʯͰɺ  4MBDLͷʮ&NBJMʯ"QQͷ  ϝʔϧΞυϨεΛઃఆ ໰͍߹Θͤͨ಺༰͕  4MBDLʹྲྀΕ͖ͯͯศར

Slide 28

Slide 28 text

• ίϯςφΤʔδΣϯτࢼͯ͠Έ·ͨ͠ • ECSͷઃఆϛεΛൃݟʂ • ϝϞϦ͕଍Γͯͳ͔ͬͨʂ

Slide 29

Slide 29 text

• ϩʔϧ಺ҟৗݕ஌ ࢼͯ͠Έ·ͨ͠

Slide 30

Slide 30 text

• ෛՙࢼݧͷ࣌ʹ΋࢖ͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 ɹʮEίϚʔεϓϥοτϑΥʔϜͷ੒௕Λࢧ͑Δ  ɹɹෛՙࢼݧͷऔΓ૊Έʯ •

Slide 31

Slide 31 text

• ෛՙࢼݧͷ࣌ʹ΋࢖ͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 • ApacheͷίωΫγϣϯ͕ுΓ෇͍ͯͨͷ͕Ϙτϧ ωοΫʹͳ͍ͬͯͨ • MackerelΛΈͨΒ͙͢ʹΘ͔ͬͨʂʢ୲౰ऀஊʣ

Slide 32

Slide 32 text

• ෛՙࢼݧͷ࣌ʹ΋࢖ͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 • Apache͕ϘτϧωοΫʹͳ͍ͬͯͨ • MackerelΛΈͨΒ͙͢ʹΘ͔ͬͨʂʢ୲౰ऀஊʣ

Slide 33

Slide 33 text

ʢMackerel౳Λݟͳ͕ΒΘ͍Θ͍΍͍ͬͯΔʣ

Slide 34

Slide 34 text

3. Mackerelར༻্ͷࠔΓ͝ͱ

Slide 35

Slide 35 text

Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̍ʣ ͠ΐͬͪΎ͏ग़ͬͺͳ͠ͷΞϥʔτ र͏΂͖Ξϥʔτ͕ຒ΋ΕΔ ͱ͸͍͑؂ࢹΛશ͘੾Δͷ΋ɻɻɻ ௨஌νϟϯωϧΛ෼͚ΔϫʔΫΞϥ΢ϯυ ຊ౰͸ɺͦΜͳΞϥʔτ͸ແ͍Α͏ʹ͢Δ΂͖͕ͩ…

Slide 36

Slide 36 text

Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̍ʣ ॏཁ౓ʹԠͯ͡௨஌νϟϯωϧΛ෼͚Δ #base_infra_alert ॏཁ౓ɿߴ #base_infra_alert_warn ॏཁ౓ɿத #base_infra_alert_info ॏཁ౓ɿ௿

Slide 37

Slide 37 text

Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̎ʣ #base_infra_alert ʹ͸Mackerel͔Βͷ௨஌Ҏ֎ͷ΋ͷ΋ྲྀ͍ͯ͠Δ ϝʔϧͰདྷΔ΋ͷɺRSSϑΟʔυͳͲ ͜ΕΒʹ͸ mention ͸ͳ͍ͷͰɺSlackͰ͜ͷνϟϯωϧͷ௨஌ઃఆ Λڧͯ͘͠ؾ͚ͮΔΑ͏ʹ͢Δ͕…

Slide 38

Slide 38 text

Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̎ʣ ௨஌͞Ε͗͢Δ #base_infra_alert ্ͷձ࿩͕શ෦௨஌͞ΕΔʂ #base_infra_alert ʹ͸௨஌Λྲྀ͚ͩ͢ʹ͢Δ ٞ࿦͸ผͷνϟϯωϧͰ ʮReacji ChannelerʯͰٞ࿦༻ͷνϟϯωϧʹ  ϦϯΫΛίϐʔ https://get.slack.help/hc/ja/articles/360000482666-ଞνϟϯωϧ΁ϝοηʔδΛαΫοͱίϐʔ

Slide 39

Slide 39 text

Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̎ʣ Ξϥʔτ௨஌ʹֆจࣈͰ൓Ԡ͢Δͱ…

Slide 40

Slide 40 text

Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগ೥໰୊ʢ̎ʣ ٞ࿦༻ͷνϟϯωϧʹϦϯΫ͕ίϐʔ͞ΕΔʂ ௨஌͕ݮͬͯগ͠ฏ࿨ʹ…

Slide 41

Slide 41 text

Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ αʔϏεͷ੒௕ͱͱ΋ʹαʔόͷछྨ͕૿͑ͨ αϒγεςϜͷ૿Ճ ϩʔϧͷ෼Խ ϚωʔδυαʔϏε౳ΛMackerelͷ؂ࢹର৅ʹ௥Ճ ϩʔϧ/ϗετͷߏ଄ʹ͖Ε͍ʹରԠ෇͚ʹ͍͘ Lambda, SQSͳͲ

Slide 42

Slide 42 text

Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ MackerelͰ͸ʮϩʔϧʯ͕؂ࢹΛ͢Δ୯ҐͰ͋Γɺά ϥϑΛ·ͱΊΔ୯ҐͰ΋͋Δ ؂ࢹ͍ͨ͠୯ҐͱάϥϑͰݟ͍ͨ୯Ґ͸ඞͣ͠΋Ұக ͠ͳ͍ ؂ࢹ͍ͨ͠୯ҐͰϩʔϧΛ෇͚͍ͯ͘ͱɺ  ʮαʔϏεʯͷը໘͕ͪ͝Όͪ͝Ό͖ͯͨ͠

Slide 43

Slide 43 text

Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ

Slide 44

Slide 44 text

Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ ୞ࠓϩʔϧ࠶ฤ੒த… Ұ؏ੑͷ͋Δϧʔϧʹج͍ͮͯઃܭ͠௚͢ ̎ͭͷʮαʔϏεʯΛ࢖ͬͯϩʔϧΛ੔ཧ͢Δ ϩʔϧάϥϑΛҰཡͰΈ͍ͨϩʔϧ → ϝΠϯͷʮαʔϏεʯ ྫɿdb-primary, db-replica ͦΕҎ֎ͷ؂ࢹ୯Ґͱͯ͠ͷϩʔϧ → αϒͷʮαʔϏεʯ ྫɿrds

Slide 45

Slide 45 text

Mackerelར༻্ͷࠔΓ͝ͱ ΧελϜϝτϦοΫ΋ϩʔϧʹ࢒ͬͯ΄͍͠ αʔό͕ফ͑ΔͱΧελϜϝτϦοΫ͕ফ͑Δ σϓϩΠ͢Δͨͼʹαʔό͕ফ͑ΔͷͰΘΓͱਏ͍ γεςϜϝτϦοΫ͸࢒͍ͬͯΔ͕… ΧελϜϝτϦοΫ΋ϩʔϧʹ࢒ͬͯ΄͍͠ʂ ͳʹͱͧ…

Slide 46

Slide 46 text

Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ աڈͷάϥϑͷϐʔΫ͕Θ͔Γʹ͍͘ ྫ͑͹ɺաڈ൒೥ͰҰ൪ϦΫΤετ͕དྷͯͨͱ͖ͷ෼ ؒϦΫΤετ਺͸Ͳͷ͘Β͍͔ͩͬͨͳʁͱ͍͏࣌ ͱΓ͋͑ͣʮ1YʯͷάϥϑΛݟͯΈΔ…ʁ

Slide 47

Slide 47 text

Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ࣌ؒൣғΛ޿͘औΔͱϐʔΫ͕ͳ·Δ

Slide 48

Slide 48 text

Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ࣌ؒൣғΛ޿͘औΔͱϐʔΫ͕ͳ·Δ

Slide 49

Slide 49 text

Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ࣌ؒൣғΛ޿͘औΔͱϐʔΫ͕ͳ·Δ

Slide 50

Slide 50 text

Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ࣌ؒൣғΛ޿͘औΔͱϐʔΫ͕ͳ·Δ

Slide 51

Slide 51 text

Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γ΍ͯ͘͢͠΄͍͠ ʮ1YʯͷάϥϑͰϐʔΫ͕ݟ͑ͯ΄͍͠ ͳʹͱͧ…

Slide 52

Slide 52 text

4. Mackerelͷ͍͍ͱ͜Ζ

Slide 53

Slide 53 text

Mackerelͷ͍͍ͱ͜Ζ ࣗ෼Ͱӡ༻͠ͳ͍͍ͯ͘ ͲΜͲΜػೳڧԽ͞Ε͍ͯ͘ AWSΠϯςάϨʔγϣϯ͕ศར ϓϥάΠϯͷ௥Ճָ͕ https://github.com/mackerelio/plugin-registry https://github.com/mackerelio/mackerel-agent-plugins ࣗ࡞΋؆୯ ʮαϙʔτνʔϜ΁࿈བྷʯͷϨεϙϯε͕Α͍

Slide 54

Slide 54 text

Mackerelͷ͍͍ͱ͜Ζ WebUI͕Θ͔Γ΍͍͢ άϥϑ͕͖Ε͍ API͕͋ͬͯϓϩάϥϜ͔Βૢ࡞Ͱ͖Δ CLI͕͋Δ ࢖͍͸͡Ί΍͍͢ ΤʔδΣϯτΛΠϯετʔϧͯ͠ϩʔϧΛઃఆ͢Δ͚ͩͰɺ  ͍͍ͩͨྑ͍ײ͡ͷάϥϑ͕ग़ͯ͘Δ ೔ຊޠͩ UIɺυΩϡϝϯτ

Slide 55

Slide 55 text

ࢀߟਤॻ ʮMackerel αʔό؂ࢹ [࣮ફ] ೖ໳ʯ ஶ: Ҫ্େีɼപ୩େีɼਿࢁ޿௨ɼాத৻࢘ɼ௶಺༎थɼদ໦խ޾ (http://gihyo.jp/book/2017/978-4-7741-9213-0) ʮೖ໳ ؂ࢹ ―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯʯ ஶ: Mike Julian (https://www.oreilly.co.jp/books/9784873118642/) ʮιϑτ΢ΣΞΤϯδχΞͷͨΊͷɹITΠϯϑϥ؂ࢹʦ࣮ફʧೖ໳ʯ ஶ: ࡈ౻༞Ұ࿠ (https://gihyo.jp/book/2016/978-4-7741-7865-3)

Slide 56

Slide 56 text

We are hiring! SRE͸΋ͪΖΜɺ༷ʑͳ৬छΛืू͍ͯ͠·͢ ΧδϡΞϧʹձࣾʹ༡ͼʹདྷ͍ͨํ͸๻·Ͱ https://jobs.binc.jp/