seesaa meets Mackerel

Mackerel Meetup #11 Tokyo #mackerelio

https://mackerelio.connpass.com/event/76678/

Transcript

1. seesaa meets Mackerel γʔαʔגࣜձࣾ ੉ށ༞थ 
2. ࣗݾ঺հ • ੉ށ༞थ • γʔαʔגࣜձࣾΤϯδχΞ • த్ೖࣾ4೥໨ • Mackerel͸2015೥͔Β • ϑϩϯτɺόοΫΤϯυɺӡ༻ 
3. ձࣾͷ঺հ • γʔαʔגࣜձࣾ • 2003೥10݄૑ۀ • γʔαʔϒϩάɺͨ͠Β͹ܝࣔ൘ɺγʔαʔ wikiͳͲͷϝσΟΞӡӦ • ϒϩάγεςϜͷOEMఏڙ 
4. ໨࣍ • Πϯϑϥӡ༻ͷ࢓ࣄʹ͍ͭͯ • MackerelҎલɾҎޙ • งғؾͰ؂ࢹ͍ͯͨ͠ɾ͍ͯ͠Δ • ؂ࢹ͢ΔྖҬ͸૿͍͑ͯͬͨ • ͜Μͳ؂ࢹΛ͍ͯ͠·͢ 
5. Πϯϑϥɾӡ༻ʹ͍ͭͯ • ΦϯϓϨϛεɺAWSΛओʹར༻͍ͯ͠Δ • ओʹӡ༻Λ΍͍ͬͯΔͷ͸3໊ఔ౓ • ख़࿅ऀ+एऀͷ૊Έ߹Θͤ • ࡉ͔͍ͱ͜Ζʹख͕ճΒͳ͍ • ࣌ʑൃੜ͢ΔϏοάόϯతΠϕϯτͰࠞཚ 
6. ӡ༻ϝϯόʔͷओͳ࢓ࣄ • ೔ࠒͷΞϥʔτରԠ • Mackerel͔ΒͷΞϥʔτΛݟͯҟৗ͔Ͳ͏͔ͷݟۃΊ • ӡ༻໘͕ڧ͍։ൃ • طଘΞϓϦέʔγϣϯͷDockerԽͷςετ • ࣗࣾϝσΟΞαʔϏεʢϒϩάɾܝࣔ൘ɾWikiʣ͸ DockerԽࡁΈ
7. ΦϯϓϨϛε • DBͳͲͷෛՙ͕༧૝͞ΕΔαʔόʔ͸CentOSΛಋೖͯ͠ӡ ༻ • ΞϓϦέʔγϣϯαʔόʔͳͲ͸ɺXen ServerΛ༻͍ͯɺVM ʢCentOSʣʹͯӡ༻ • ݹ͔͘ΒͷαʔϏε͕ଟ͘ɺػثϦϓϨʔεͳͲʹ߹Θͤͯ VMɺmysqlͷઃఆͳͲฏ४Խ͍ͤͯ͞Δ • ҆ఆՔಇͯ͠΍ͬͱฏۉ఺ 
8. AWS • ৽نαʔϏεͳͲ͸ੵۃతʹ࠾༻͢Δ܏޲ • mackerel-agent-pluginsʹAWSؔ࿈͕ॆ࣮͍ͯ͠ ΔͷͰ҆৺ • ΠϯςάϨʔγϣϯػೳ͸ঢ়گʹԠͯ͡࢖͍෼͚ • ࢖Θͳ͍৔߹͸EC2ͷϗετϝτϦοΫʹ಺แ 
9. MackerelҎલɾҎޙ 
10. MackerelҎલ • ࣾ಺ΦϯϓϨϛε؀ڥ؂ࢹ • savacan ʢಡΈ:͞͹͔ΜʣͰ؂ࢹ͍ͯͨ͠ • cacti Ͱάϥϑ • 1ਓӡ༻୲౰ऀ͕ுΓ෇͖ 
11. savacanͷ؆୯ͳ঺հ • αʔόʔ୆ா • ϥοΫɺVMͷऩ༰ϗετ୆ா • ϩʔυόϥϯαͷίϯτϩʔϥత໾ׂ • ϑϩϯτͷwebαʔόʔ͕ઃఆΛಡΈࠐΉ • Ξϥʔτൃใʢϝʔϧʣ • ϩάϩʔςʔγϣϯิॿ 
12. savacanͷΞʔΩςΫνϟ • ؂ࢹ͢ΔϗετʹεΫϦϓτΛΠϯετʔϧɺperl੡ • CartonͳͲΛߟྀͯ͠ͳ͍࡞ΓɺΠϯετʔϧ͕ෳࡶ • ϝτϦοΫ͸ू໿͢ΔmysqlʹmysqlͰ௨৴ • ؂ࢹ͸γϯάϧεϨου • ࣌ʑ٧·ͬͯ؂ࢹ͕ࢮʹ͕ͪ… • εέʔϧ͠ͳ͍ɺಠࣗCLI͸ॳ৺ऀʹͱ͔ͬͯͳΓϋʔυϧ 
13. savacanͷCLIʢྫʣ sdev@xxxx$ svcn stat bl-app21 bl-app21 bl app active 1 20.1 N/A N/A sdev@xxxx$ svcn scan stat bl-app21 bl-app21 df 0 95 on bl-app21 loads 0 35 on 35 bl-app21 nfs_mount 0 3 on bl-app21 ping 0 0 on bl-app21 port 0 3 on 5000 sdev@xxxx$ svcn update bl-app21 status active sdev@xxxx$ svcn update bl-app21 status ignored 
14. savacanͷCLIߏจ • svcn update HOSTNAME KEY1 VALUE1 • σʔλΛೖΕΔmysqlͷߏ଄͕ͦͷ··࢖ΘΕ͍ͯΔ • ςʔϒϧߏ଄͕Θ͔Βͳ͍ͱ࢖͑ͳ͍ • ӡ༻͢Δਓ͔͠Θ͔ͬͯͳ͍ • ӡ༻ϝϯόʔҎ֎ͷࢀೖোนʹͳ͍ͬͯΔ
15. savacanͷcliͰݟΕΔ৘ใ • load averageɺdfͷঢ়ଶ • pingɺϙʔτ؂ࢹɺNFSϚ΢ϯτͷ؂ࢹঢ়ଶ • ϩʔυόϥϯεର৅Ͱ͋Δ͔ͷ༗ແ • load averageɺdfͷঢ়ଶͳͲʹΑͬͯ͸঎༻ αʔϏε͔Βࣗಈతʹ཭୤͠ɺΞϥʔτൃใ 
16. savacanͰݟΕΔաڈ • աڈͷ஋͸ࣺ͍ͯͯͨͷͰɺܧଓͨ͠ϝτϦοΫ؂ࢹ͸ Ͱ͖͍ͯͳ͔ͬͨ • ॏཁੑΛೝࣝͰ͖͍ͯͳ͔ͬͨͷͩͱࢥ͏ • MackerelʹΑͬͯաڈ͕ܧଓͯ͠ݟΕΔΑ͏ʹͳΓɺӡ ༻ෛՙΛ၆ᛌతʹݟ΍͘͢ͳͬͨ • cactiͰ͸ɺload averageఔ౓͔͠औಘ͍ͯ͠ͳ͔ͬͨ ͨΊ 
17. ͋Δౙɺӡ༻୲౰ऀ͕ࣙΊͨ 
18. cactiΛҾ͖ܧ͝͏ͱͨ͠ • snmpͷmibΛ֦ு͠ɺएׯΦϨΦϨͳ؂ࢹΛ ͍ͯͨ͠ɻ • ଐਓత͗͢Δ… • cactiͷੵۃతͳܧଓӡ༻Λஅ೦ • ژ౎ͷํ޲͔Βޫ͕ࠩ͠ࠐΜͰ͖ͨ 
19. MackerelΛࢼ༻ 
20. ΤʔδΣϯτͷηοτΞοϓ • ଟछଟ༷ͳ؀ڥʹࡶʹΤʔδΣϯτΛೖΕͯ άϥϑͷงғؾʹײಈ • ଈܾͰ໿200ϗετͷ؂ࢹΛ࢝ΊΔ • ઃఆͷࣽ٧Ί͕؁͘ɺΤʔδΣϯτΛೖΕͨ ͚ͩɻͱ͍͏ࡶ͗͢Δϗετ΋͋ͬͨ 
21. MackerelΛೖΕͯؾ͍ͮͨ͜ͱ • VMͷੑೳઃܭ͕ద੾Ͱͳ͘ɺΧπΧπؾຯ • ֎ܗ؂ࢹʹΑͬͯࡉ͔͍εϩʔμ΢ϯͳͲݟ ͔ͭΔ • ࠓ·Ͱͷπέ͕શ෦ݟ͑ΔԽ͞ΕΔ 
22. MackerelʹΑͬͯಘΒΕͨ؂ࢹ • ֎ܗ؂ࢹ • Ҏલ͔Βଘࡏ͸͍ͯͨ͠΋ͷͷࡉ͔͍࣌ؒ ࣠ͷ৘ใ͕ܽམ͕ͪ͠ • طଘ؂ࢹͰ͸ɺଟ͘ͷ৘ใΛऔಘ͍ͯ͠ͳ͔ͬ ͨͷͰɺࠓࢥ͑͹ɺงғؾͰରࡦͳͲ͍ͯ͠ ͨͷ͔΋͠Εͳ͍
23. Mackerelͷ͍͍ͱ͜Ζ • ·ͣ͸ΤʔδΣϯτΛೖΕͯঢ়گΛݟΔɻͱ͍͏ͱ͜Ζ ͔Β࢝ΊΒΕΔɻ • ΤʔδΣϯτ͸yumͳͲͰΠϯετʔϧͰ͖Δɻߋ৽΋ ׆ൃ • ෆ۩߹ɺෆ໌఺͸͙͢ʹղܾ͞ΕΔ • TwitterͰࠔͬͨɻͱݴΘͣʹʮαϙʔτνʔϜ΁࿈བྷʯ 
24. զʑ͸งғؾͰ؂ࢹ͍ͯͨ͠ 
25. ؾ͍͖͔͚ͮͨͬ • ಥવ͸ͯͳ͞Μ͔Βɺ࢖͍উखͳͲΠϯλϏϡʔ ͍ͨ͠ͷͰ๚໰ͯ͠ྑ͍͔?ͱͷ͓੠͕͔͔Δ • ओʹɺ͜Ε·ͰʹϦϦʔεͨ͠ػೳͷར༻ͷ༗ ແɺ࢖͍উखͷώΞϦϯά • a-know͞ΜʮདྷिɺΠϕϯτ͕͋ΔͷͰͥͻ དྷͯ΄͍͠ʯ 
26. 2017/10/5
27. Driving Mercari with 50+ custom plugins / Mackerel DAY by kazeburo
28. Mackerel DAY • ͦΕͳΓʹ؂ࢹϧʔϧͷݟ௚͠ͳͲ͸͍ͯ͠ ͨͭ΋Γͩͬͨ • ͔͠͠ɺ΄ͱΜͲ͕৔౰ͨΓతͳݟ௚͠ • ࡉ͔͍ͱ͜Ζ·Ͱࢹ໺͕ಧ͍ͯͳ͔ͬͨ • ඦฉ͸Ұݟʹ೗͔ͣɻงғؾͰ؂ࢹ͍ͯͨ͠
29. PDCA͕ճͬͯͳ͔ͬͨ • ઃఆϑΝΠϧ͸αʔϏε͝ͱʹ؅ཧ • ࠷ॳʹ࡞ͬͨαʔϏεͷઃఆϑΝΠϧΛίϐʔͯ͠ɺઃ ఆΛࣽ٧ΊΔ • ޙͷ΄͏ʹͳΔͱɺઃఆ߲໨͕ॆ࣮ • ઃఆͷݟ௚͕͠ͳ͍ͷͰɺ࠷ॳʹೖΕͨͷ͸๨ΕΒΕΔ • ʮ͋Εɺ͜Εʹ͸͜ͷϝτϦοΫ͕ͳ͍Ͷʯ 
30. Mackerelͷઃఆݟ௚͠ • งғؾա͗ͨઃఆͩͬͨͱ൓ল • mackerel-agentΛೖΕ͍ͯͨ໿300ϗετͷઃఆ ݟ௚͠Λܾஅ • mkrίϚϯυͰαʔόʔҰཡΛ࡞ΓɺઃఆϑΝΠϧ Λμ΢ϯϩʔυ • શ਺ݕࠪΛߦ͏͜ͱʹ 
31. ઃఆΛ௚ͨ͠ͱ͜Ζ • ϓϩηεͷ؂ࢹΛ͍͕ͨ͠ɺΑ͘Θ͔Βͳ͍ͷͰɺ શ෦άϥϑʹग़ͤ͹͍͍Μ͡ΌͶʁ • ͦΕɺcheck-procsͰͰ͖ΔΑ • ͦ΋ͦ΋γεςϜʹඞཁෆՄܽͳϓϩηεΛશ෦ ؂ࢹͰ͖ͯͳ͔ͬͨ • ྫ͑͹ɺϩʔΧϧͷDNSΩϟογϡͳͲ 
32. աڈ΍͍ࣦͬͯͨഊ [plugin.checks.xxxx] command = "TIME=`date +%s`; for x in xxx1 xxx2; do printf 'XXXXX.%s %d %d\n' $x `command $x` $TIME ; done” 
33. ΍ͬͯ͠·͍͕ͪ • ϫϯϥΠφʔ • ΍͚ͬͭͰॻ͍ͨΓ͢Δ • ޙ͔ΒɺಡΈͮΒ͍ • ͜͏͍ͬͨ΋ͷ͸શͯγΣϧεΫϦϓτԽ • ϓϥάΠϯ͕͋ͬͨΒͦͬͪʹࡌͤସ͑ 
34. Mackerel DAYͰͷऩ֭ • ϩʔϧ͝ͱʹ෼͚Δ֓೦Λਖ਼͘͠஌Δ • ϩʔϧͷઃఆ=include͢ΔઃఆϑΝΠϧΛ ಉҰͱ͢ΔελΠϧ • ΍͹͍ɺ͔͍͍ͬ͜ 
35. MackerelͰݟ͍ͨ৘ใ • ྫ͑͹AWSͳͲͩͱɺAZͳͲ͕ݟΕΔɻ • XenαʔόʔͷVM͸Ͳ͜ʹϗετ͍ͯ͠Δͷ ͔Θ͔Βͳ͍ • savacanͱͷೋॏ؅ཧɺMackerelݟ͚ͨͩͰ ೺Ѳ͍ͨ͠ 
36. Whitebait • ͠Β͢ - VM͕Ͳ͜ʹډΔ͔஌Β͢ • Xen ServerͷRPCΛୟ͍ͯVMͷҐஔ৘ใͱIPΞυϨε ҰཡΛΩϟογϡ • mackerel-agentͷchecksϓϥάΠϯͷ݁ՌͰऩ༰ϗε τΛࣔͤΕΔΑ͏ʹ • https://github.com/yseto/whitebait 
37. ऩ༰ϗετ͕Θ͔Δ 
38. MackerelͷઃఆϨϙδτϦ • Mackerelͷઃఆ͸ࣾ಺GitͰ؅ཧ͞Ε͍ͯΔ • ͦ΋ͦ΋ɺϗετ͝ͱʹ֎෦ͷαʔόʔͷϨϙδτϦ ʹऔΓ͜Έʹߦ͘ͷ͸credential؅ཧͳͲ໰୊͕͋Δ • ސ٬ͷػີ৘ใ͕ೖͬͯͳ͚Ε͹໰୊ͳ͍ͱ൑அ͠ɺ WhitebaitʹGitϨϙδτϦͷϗεςΟϯάػೳΛ࣋ͨ ͤΔʢpull͚ͩͰ͖Δʣ 
39. ૿͑Δ؂ࢹྖҬ 
40. ΦϯϓϨϛεεΠον • ΦϯϓϨϛε؀ڥͷεΠονɺࠓ·Ͱৄ͘͠τϥ ϑΟοΫͷ֬ೝͳͲ͍ͯ͠ͳ͔ͬͨ • τϥϒϧ͕͋ͬͨͷͰɺMackerelͰ؂ࢹ͢Δ͜ͱ ʹ… • ͦ΋ͦ΋ɺMackerelͰεΠονͷάϥϑग़ͨ͠ ͍͚ͩͳͷ͕ͩ… 
41. OSSΛௐ΂Δ • https://github.com/sorah/snmp2mkr • CookpadͷͦΒ͸͞Μ͕࡞͍ͬͯΔOSS • SNMPΛू໿ͯ͠ɺϗετͱͯ͠POSTͯ͘͠ΕΔ • ݩ͸ɺΠϕϯτͷWi-Fi؂ࢹ༻ͷΑ͏͕ͩɺͱΓ͋͑ͣઃఆ • ର৅εΠον͸ϙʔτ਺͕ଟ͘ɺऔΓ͜΅͕͋͠Δ΋ͷͷɺ࣌ ؒ࣠Λେ͖͘ͱΕ͹܏޲͸͔ͭΊΔ • ͪΌΜͱه࿥͍ͨ͠ͳΒɺ͕Μ͹Δ΂͖ 
42. ʮϓϦϯλಈ͔ͳ͍Μ͚ͩͲʁʯ ϑϦʔૉࡐͺͨͦ͘ ΑΓ 
43. ΦϑΟεͰ͋Δ͋Δ • ؾܰʹ੠͕͔͚ΒΕΔΤϯδχΞʹॿ͚Λٻ Ίͯ͘Δύλʔϯ • ΦϑΟεͷॄث΋؂ࢹͰ͖Δ΍ͭ͸ͯ͠͠· ͓͏ • ΞϥʔτൃใͰɺτϥϒϧ͕ൃੜ͢Δલʹ ରॲ͢Δ 
44. ؂ࢹ͞ΕΔΦϑΟεػث 
45. ΦϑΟε಺ͷ؂ࢹ • pingͷԠ౴ΛݟΔ͜ͱͰ؂ࢹ • ΋ͪΖΜɺϧʔλʔͳͲ΋؂ࢹ͍ͯ͠Δ • ԕִ஍ΦϑΟεͷϧʔλʔ΋τϥϑΟοΫ؂ ࢹ͠ɺτϥϒϧରԠ͕Ͱ͖ΔΑ͏ʹ͍ͯ͠Δ 
46. ͜Μͳ؂ࢹ΋͍ͯ͠·͢ ͜Μͳ͜ͱ΋͍ͯ͠·͢ 
47. mysqlεϨʔϒ • Slave_IO_RunningͳͲΛ؂ࢹ • ఆ࣌όοΫΞοϓͳͲͷؒ͸εςʔλεΛૢ ࡞ • Ξϥʔτ͕ෆඞཁʹ໐Βͳ͍Α͏ʹ͢Δ • ॲཧʹࣦഊ͔ͨ͠Λcheck-file-ageͳͲͰ؂ࢹ 
48. keepalivedͷ؂ࢹ • ϓϩηε؂ࢹ • ঢ়ଶ؂ࢹʢΞΫςΟϒɾελϯόΠʣ • γΣϧεΫϦϓτͰIPΞυϨεͷଘࡏΛνΣοΫ ͠ɺঢ়ଶ͕੾ΓସΘͬͨ࣌ʹΞϥʔτΛൃใ • ো֐ɺఆܕ࡞ۀ࣌ͷ੾ΓସΘΓ͕ɺӡ༻ϝϯόʔ શମʹ఻ΘΔ 
49. ϋʔυ΢ΣΞRAIDͷ؂ࢹ • DELL OpenManageͳͲͷCLI݁ՌΛύʔε • σΟεΫނোΛݕ஌ • ·ͨɺOpenManage͕ಥવࢮ͍ͯͨ͜͠ͱ΋ ͋ΔͷͰɺϓϩηεͷࢮ׆؂ࢹ • ஌ݟ͕͋·Γཷ·͍ͬͯͳ͍ 
50. ͘͞ΒͷΫϥ΢υͷ՝ۚ؂ࢹ • ࡞ۀͰΠϯελϯεΛ࡞ͬͯͦͷ··ɻͷΑ ͏ͳ͜ͱ͕ͳ͍Α͏ʹɺ՝ۚͷάϥϑΛ MackerelʹPOST • ΠϯελϯεͷλάͰαʔϏεΛ͚ͭͯɺαʔ Ϗε͝ͱͷ՝ֹۚΛ͓͓ΑͦͰݟ͑ΔԽ͢Δ 
51. lambdaɺCloudFrontͷ؂ࢹ • lambda΍CloudFrontɺAmazon Elasticsearch ServiceͳͲ࢖༻͍ͯ͠ΔͷͰɺmackerel- agent-pluginsͰ؂ࢹ • Elasticsearch Service͸ɺϗετͱͯ͠ొ࿥͠ ͍ͯΔ 
52. Dockerͷ؂ࢹ • ίϯςφͷىಈ͍ͯ͠Δ͔Λdocker psΛύʔ εͯ͠؂ࢹ • ·ͨ͸ɺϙʔτΛ࢖͍ͬͯΔͷͰͦΕΛ؂ࢹ ͢ΔͳͲ • ෳ਺ίϯςφΛ1ϗετͰ͋͛ΔΫϥελͷΑ ͏ͳӡ༻͸·͍ͩͯ͠ͳ͍ 
53. ঎༻؀ڥͷdockerͷঢ়ଶ 
54. ฐࣾɺChatWork࢖ͬͯ·͢ • ChatWorkɺը૾ͷURLΛషͬͯ΋ల։ͯ͘͠Ε ͳ͍… • ChatWorkͷAPIʹը૾Λड͚෇͚ΔAPI͕ͳ͍ • MackerelͷάϥϑΛChatWorkͰݟ͍ͨ • https://github.com/yseto/slack_to_chatwork 
55. ࢓૊Έ • Mackerel͔Βม׵༻ͷΞϓϦΛSlackͷΤϯυ ϙΠϯτͱͯ͠ొ࿥ • chatworkͷwebͷૢ࡞Λௐ΂ͯɺΞοϓϩʔ υૢ࡞Λ૊Έཱͯͯૹ৴ 
56. chatworkͰMackerelͷάϥϑΛோΊΔ άϥϑ͕ݟΕΔ͜ͱͰٞ࿦͕νϟοτʹͪΌΜͱ࢒Δ 
57. ·ͱΊ 
58. ·ͱΊ • ΦϯϓϨϛεͰ͋ͬͯ΋AWSͰ͋ͬͯ΋ɺΠ ϯετʔϧ͢Δ͚ͩͰ࢖͑ΔMackerel͸ӡ༻ ͷຯํ • ΦϑΟεͷτϥϒϧ΋MackerelͰ؂ࢹ͢Δͱ ḿΔ 
59. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠