seesaa meets Mackerel

726899505fa048a1bce68537a337dbd1?s=47 yseto
February 05, 2018

seesaa meets Mackerel

Mackerel Meetup #11 Tokyo #mackerelio

https://mackerelio.connpass.com/event/76678/

726899505fa048a1bce68537a337dbd1?s=128

yseto

February 05, 2018
Tweet

Transcript

  1. seesaa meets Mackerel γʔαʔגࣜձࣾ ੉ށ༞थ 

  2. ࣗݾ঺հ • ੉ށ༞थ • γʔαʔגࣜձࣾΤϯδχΞ • த్ೖࣾ4೥໨ • Mackerel͸2015೥͔Β •

    ϑϩϯτɺόοΫΤϯυɺӡ༻ 
  3. ձࣾͷ঺հ • γʔαʔגࣜձࣾ • 2003೥10݄૑ۀ • γʔαʔϒϩάɺͨ͠Β͹ܝࣔ൘ɺγʔαʔ wikiͳͲͷϝσΟΞӡӦ • ϒϩάγεςϜͷOEMఏڙ

    
  4. ໨࣍ • Πϯϑϥӡ༻ͷ࢓ࣄʹ͍ͭͯ • MackerelҎલɾҎޙ • งғؾͰ؂ࢹ͍ͯͨ͠ɾ͍ͯ͠Δ • ؂ࢹ͢ΔྖҬ͸૿͍͑ͯͬͨ •

    ͜Μͳ؂ࢹΛ͍ͯ͠·͢ 
  5. Πϯϑϥɾӡ༻ʹ͍ͭͯ • ΦϯϓϨϛεɺAWSΛओʹར༻͍ͯ͠Δ • ओʹӡ༻Λ΍͍ͬͯΔͷ͸3໊ఔ౓ • ख़࿅ऀ+एऀͷ૊Έ߹Θͤ • ࡉ͔͍ͱ͜Ζʹख͕ճΒͳ͍ •

    ࣌ʑൃੜ͢ΔϏοάόϯతΠϕϯτͰࠞཚ 
  6. ӡ༻ϝϯόʔͷओͳ࢓ࣄ • ೔ࠒͷΞϥʔτରԠ • Mackerel͔ΒͷΞϥʔτΛݟͯҟৗ͔Ͳ͏͔ͷݟۃΊ • ӡ༻໘͕ڧ͍։ൃ • طଘΞϓϦέʔγϣϯͷDockerԽͷςετ •

    ࣗࣾϝσΟΞαʔϏεʢϒϩάɾܝࣔ൘ɾWikiʣ͸ DockerԽࡁΈ
  7. ΦϯϓϨϛε • DBͳͲͷෛՙ͕༧૝͞ΕΔαʔόʔ͸CentOSΛಋೖͯ͠ӡ ༻ • ΞϓϦέʔγϣϯαʔόʔͳͲ͸ɺXen ServerΛ༻͍ͯɺVM ʢCentOSʣʹͯӡ༻ • ݹ͔͘ΒͷαʔϏε͕ଟ͘ɺػثϦϓϨʔεͳͲʹ߹Θͤͯ

    VMɺmysqlͷઃఆͳͲฏ४Խ͍ͤͯ͞Δ • ҆ఆՔಇͯ͠΍ͬͱฏۉ఺ 
  8. AWS • ৽نαʔϏεͳͲ͸ੵۃతʹ࠾༻͢Δ܏޲ • mackerel-agent-pluginsʹAWSؔ࿈͕ॆ࣮͍ͯ͠ ΔͷͰ҆৺ • ΠϯςάϨʔγϣϯػೳ͸ঢ়گʹԠͯ͡࢖͍෼͚ • ࢖Θͳ͍৔߹͸EC2ͷϗετϝτϦοΫʹ಺แ

    
  9. MackerelҎલɾҎޙ 

  10. MackerelҎલ • ࣾ಺ΦϯϓϨϛε؀ڥ؂ࢹ • savacan ʢಡΈ:͞͹͔ΜʣͰ؂ࢹ͍ͯͨ͠ • cacti Ͱάϥϑ •

    1ਓӡ༻୲౰ऀ͕ுΓ෇͖ 
  11. savacanͷ؆୯ͳ঺հ • αʔόʔ୆ா • ϥοΫɺVMͷऩ༰ϗετ୆ா • ϩʔυόϥϯαͷίϯτϩʔϥత໾ׂ • ϑϩϯτͷwebαʔόʔ͕ઃఆΛಡΈࠐΉ •

    Ξϥʔτൃใʢϝʔϧʣ • ϩάϩʔςʔγϣϯิॿ 
  12. savacanͷΞʔΩςΫνϟ • ؂ࢹ͢ΔϗετʹεΫϦϓτΛΠϯετʔϧɺperl੡ • CartonͳͲΛߟྀͯ͠ͳ͍࡞ΓɺΠϯετʔϧ͕ෳࡶ • ϝτϦοΫ͸ू໿͢ΔmysqlʹmysqlͰ௨৴ • ؂ࢹ͸γϯάϧεϨου •

    ࣌ʑ٧·ͬͯ؂ࢹ͕ࢮʹ͕ͪ… • εέʔϧ͠ͳ͍ɺಠࣗCLI͸ॳ৺ऀʹͱ͔ͬͯͳΓϋʔυϧ 
  13. savacanͷCLIʢྫʣ sdev@xxxx$ svcn stat bl-app21 bl-app21 bl app active 1

    20.1 N/A N/A sdev@xxxx$ svcn scan stat bl-app21 bl-app21 df 0 95 on bl-app21 loads 0 35 on 35 bl-app21 nfs_mount 0 3 on bl-app21 ping 0 0 on bl-app21 port 0 3 on 5000 sdev@xxxx$ svcn update bl-app21 status active sdev@xxxx$ svcn update bl-app21 status ignored 
  14. savacanͷCLIߏจ • svcn update HOSTNAME KEY1 VALUE1 • σʔλΛೖΕΔmysqlͷߏ଄͕ͦͷ··࢖ΘΕ͍ͯΔ •

    ςʔϒϧߏ଄͕Θ͔Βͳ͍ͱ࢖͑ͳ͍ • ӡ༻͢Δਓ͔͠Θ͔ͬͯͳ͍ • ӡ༻ϝϯόʔҎ֎ͷࢀೖোนʹͳ͍ͬͯΔ
  15. savacanͷcliͰݟΕΔ৘ใ • load averageɺdfͷঢ়ଶ • pingɺϙʔτ؂ࢹɺNFSϚ΢ϯτͷ؂ࢹঢ়ଶ • ϩʔυόϥϯεର৅Ͱ͋Δ͔ͷ༗ແ • load

    averageɺdfͷঢ়ଶͳͲʹΑͬͯ͸঎༻ αʔϏε͔Βࣗಈతʹ཭୤͠ɺΞϥʔτൃใ 
  16. savacanͰݟΕΔաڈ • աڈͷ஋͸ࣺ͍ͯͯͨͷͰɺܧଓͨ͠ϝτϦοΫ؂ࢹ͸ Ͱ͖͍ͯͳ͔ͬͨ • ॏཁੑΛೝࣝͰ͖͍ͯͳ͔ͬͨͷͩͱࢥ͏ • MackerelʹΑͬͯաڈ͕ܧଓͯ͠ݟΕΔΑ͏ʹͳΓɺӡ ༻ෛՙΛ၆ᛌతʹݟ΍͘͢ͳͬͨ •

    cactiͰ͸ɺload averageఔ౓͔͠औಘ͍ͯ͠ͳ͔ͬͨ ͨΊ 
  17. ͋Δౙɺӡ༻୲౰ऀ͕ࣙΊͨ 

  18. cactiΛҾ͖ܧ͝͏ͱͨ͠ • snmpͷmibΛ֦ு͠ɺएׯΦϨΦϨͳ؂ࢹΛ ͍ͯͨ͠ɻ • ଐਓత͗͢Δ… • cactiͷੵۃతͳܧଓӡ༻Λஅ೦ • ژ౎ͷํ޲͔Βޫ͕ࠩ͠ࠐΜͰ͖ͨ

    
  19. MackerelΛࢼ༻ 

  20. ΤʔδΣϯτͷηοτΞοϓ • ଟछଟ༷ͳ؀ڥʹࡶʹΤʔδΣϯτΛೖΕͯ άϥϑͷงғؾʹײಈ • ଈܾͰ໿200ϗετͷ؂ࢹΛ࢝ΊΔ • ઃఆͷࣽ٧Ί͕؁͘ɺΤʔδΣϯτΛೖΕͨ ͚ͩɻͱ͍͏ࡶ͗͢Δϗετ΋͋ͬͨ 

  21. MackerelΛೖΕͯؾ͍ͮͨ͜ͱ • VMͷੑೳઃܭ͕ద੾Ͱͳ͘ɺΧπΧπؾຯ • ֎ܗ؂ࢹʹΑͬͯࡉ͔͍εϩʔμ΢ϯͳͲݟ ͔ͭΔ • ࠓ·Ͱͷπέ͕શ෦ݟ͑ΔԽ͞ΕΔ 

  22. MackerelʹΑͬͯಘΒΕͨ؂ࢹ • ֎ܗ؂ࢹ • Ҏલ͔Βଘࡏ͸͍ͯͨ͠΋ͷͷࡉ͔͍࣌ؒ ࣠ͷ৘ใ͕ܽམ͕ͪ͠ • طଘ؂ࢹͰ͸ɺଟ͘ͷ৘ใΛऔಘ͍ͯ͠ͳ͔ͬ ͨͷͰɺࠓࢥ͑͹ɺงғؾͰରࡦͳͲ͍ͯ͠ ͨͷ͔΋͠Εͳ͍

  23. Mackerelͷ͍͍ͱ͜Ζ • ·ͣ͸ΤʔδΣϯτΛೖΕͯঢ়گΛݟΔɻͱ͍͏ͱ͜Ζ ͔Β࢝ΊΒΕΔɻ • ΤʔδΣϯτ͸yumͳͲͰΠϯετʔϧͰ͖Δɻߋ৽΋ ׆ൃ • ෆ۩߹ɺෆ໌఺͸͙͢ʹղܾ͞ΕΔ •

    TwitterͰࠔͬͨɻͱݴΘͣʹʮαϙʔτνʔϜ΁࿈བྷʯ 
  24. զʑ͸งғؾͰ؂ࢹ͍ͯͨ͠ 

  25. ؾ͍͖͔͚ͮͨͬ • ಥવ͸ͯͳ͞Μ͔Βɺ࢖͍উखͳͲΠϯλϏϡʔ ͍ͨ͠ͷͰ๚໰ͯ͠ྑ͍͔?ͱͷ͓੠͕͔͔Δ • ओʹɺ͜Ε·ͰʹϦϦʔεͨ͠ػೳͷར༻ͷ༗ ແɺ࢖͍উखͷώΞϦϯά • a-know͞ΜʮདྷिɺΠϕϯτ͕͋ΔͷͰͥͻ དྷͯ΄͍͠ʯ

    
  26. 2017/10/5

  27. Driving Mercari with 50+ custom plugins / Mackerel DAY by

    kazeburo
  28. Mackerel DAY • ͦΕͳΓʹ؂ࢹϧʔϧͷݟ௚͠ͳͲ͸͍ͯ͠ ͨͭ΋Γͩͬͨ • ͔͠͠ɺ΄ͱΜͲ͕৔౰ͨΓతͳݟ௚͠ • ࡉ͔͍ͱ͜Ζ·Ͱࢹ໺͕ಧ͍ͯͳ͔ͬͨ •

    ඦฉ͸Ұݟʹ೗͔ͣɻงғؾͰ؂ࢹ͍ͯͨ͠
  29. PDCA͕ճͬͯͳ͔ͬͨ • ઃఆϑΝΠϧ͸αʔϏε͝ͱʹ؅ཧ • ࠷ॳʹ࡞ͬͨαʔϏεͷઃఆϑΝΠϧΛίϐʔͯ͠ɺઃ ఆΛࣽ٧ΊΔ • ޙͷ΄͏ʹͳΔͱɺઃఆ߲໨͕ॆ࣮ • ઃఆͷݟ௚͕͠ͳ͍ͷͰɺ࠷ॳʹೖΕͨͷ͸๨ΕΒΕΔ

    • ʮ͋Εɺ͜Εʹ͸͜ͷϝτϦοΫ͕ͳ͍Ͷʯ 
  30. Mackerelͷઃఆݟ௚͠ • งғؾա͗ͨઃఆͩͬͨͱ൓ল • mackerel-agentΛೖΕ͍ͯͨ໿300ϗετͷઃఆ ݟ௚͠Λܾஅ • mkrίϚϯυͰαʔόʔҰཡΛ࡞ΓɺઃఆϑΝΠϧ Λμ΢ϯϩʔυ •

    શ਺ݕࠪΛߦ͏͜ͱʹ 
  31. ઃఆΛ௚ͨ͠ͱ͜Ζ • ϓϩηεͷ؂ࢹΛ͍͕ͨ͠ɺΑ͘Θ͔Βͳ͍ͷͰɺ શ෦άϥϑʹग़ͤ͹͍͍Μ͡ΌͶʁ • ͦΕɺcheck-procsͰͰ͖ΔΑ • ͦ΋ͦ΋γεςϜʹඞཁෆՄܽͳϓϩηεΛશ෦ ؂ࢹͰ͖ͯͳ͔ͬͨ •

    ྫ͑͹ɺϩʔΧϧͷDNSΩϟογϡͳͲ 
  32. աڈ΍͍ࣦͬͯͨഊ [plugin.checks.xxxx] command = "TIME=`date +%s`; for x in xxx1

    xxx2; do printf 'XXXXX.%s %d %d\n' $x `command $x` $TIME ; done” 
  33. ΍ͬͯ͠·͍͕ͪ • ϫϯϥΠφʔ • ΍͚ͬͭͰॻ͍ͨΓ͢Δ • ޙ͔ΒɺಡΈͮΒ͍ • ͜͏͍ͬͨ΋ͷ͸શͯγΣϧεΫϦϓτԽ •

    ϓϥάΠϯ͕͋ͬͨΒͦͬͪʹࡌͤସ͑ 
  34. Mackerel DAYͰͷऩ֭ • ϩʔϧ͝ͱʹ෼͚Δ֓೦Λਖ਼͘͠஌Δ • ϩʔϧͷઃఆ=include͢ΔઃఆϑΝΠϧΛ ಉҰͱ͢ΔελΠϧ • ΍͹͍ɺ͔͍͍ͬ͜ 

  35. MackerelͰݟ͍ͨ৘ใ • ྫ͑͹AWSͳͲͩͱɺAZͳͲ͕ݟΕΔɻ • XenαʔόʔͷVM͸Ͳ͜ʹϗετ͍ͯ͠Δͷ ͔Θ͔Βͳ͍ • savacanͱͷೋॏ؅ཧɺMackerelݟ͚ͨͩͰ ೺Ѳ͍ͨ͠ 

  36. Whitebait • ͠Β͢ - VM͕Ͳ͜ʹډΔ͔஌Β͢ • Xen ServerͷRPCΛୟ͍ͯVMͷҐஔ৘ใͱIPΞυϨε ҰཡΛΩϟογϡ •

    mackerel-agentͷchecksϓϥάΠϯͷ݁ՌͰऩ༰ϗε τΛࣔͤΕΔΑ͏ʹ • https://github.com/yseto/whitebait 
  37. ऩ༰ϗετ͕Θ͔Δ 

  38. MackerelͷઃఆϨϙδτϦ • Mackerelͷઃఆ͸ࣾ಺GitͰ؅ཧ͞Ε͍ͯΔ • ͦ΋ͦ΋ɺϗετ͝ͱʹ֎෦ͷαʔόʔͷϨϙδτϦ ʹऔΓ͜Έʹߦ͘ͷ͸credential؅ཧͳͲ໰୊͕͋Δ • ސ٬ͷػີ৘ใ͕ೖͬͯͳ͚Ε͹໰୊ͳ͍ͱ൑அ͠ɺ WhitebaitʹGitϨϙδτϦͷϗεςΟϯάػೳΛ࣋ͨ ͤΔʢpull͚ͩͰ͖Δʣ

    
  39. ૿͑Δ؂ࢹྖҬ 

  40. ΦϯϓϨϛεεΠον • ΦϯϓϨϛε؀ڥͷεΠονɺࠓ·Ͱৄ͘͠τϥ ϑΟοΫͷ֬ೝͳͲ͍ͯ͠ͳ͔ͬͨ • τϥϒϧ͕͋ͬͨͷͰɺMackerelͰ؂ࢹ͢Δ͜ͱ ʹ… • ͦ΋ͦ΋ɺMackerelͰεΠονͷάϥϑग़ͨ͠ ͍͚ͩͳͷ͕ͩ…

    
  41. OSSΛௐ΂Δ • https://github.com/sorah/snmp2mkr • CookpadͷͦΒ͸͞Μ͕࡞͍ͬͯΔOSS • SNMPΛू໿ͯ͠ɺϗετͱͯ͠POSTͯ͘͠ΕΔ • ݩ͸ɺΠϕϯτͷWi-Fi؂ࢹ༻ͷΑ͏͕ͩɺͱΓ͋͑ͣઃఆ •

    ର৅εΠον͸ϙʔτ਺͕ଟ͘ɺऔΓ͜΅͕͋͠Δ΋ͷͷɺ࣌ ؒ࣠Λେ͖͘ͱΕ͹܏޲͸͔ͭΊΔ • ͪΌΜͱه࿥͍ͨ͠ͳΒɺ͕Μ͹Δ΂͖ 
  42. ʮϓϦϯλಈ͔ͳ͍Μ͚ͩͲʁʯ ϑϦʔૉࡐͺͨͦ͘ ΑΓ 

  43. ΦϑΟεͰ͋Δ͋Δ • ؾܰʹ੠͕͔͚ΒΕΔΤϯδχΞʹॿ͚Λٻ Ίͯ͘Δύλʔϯ • ΦϑΟεͷॄث΋؂ࢹͰ͖Δ΍ͭ͸ͯ͠͠· ͓͏ • ΞϥʔτൃใͰɺτϥϒϧ͕ൃੜ͢Δલʹ ରॲ͢Δ

    
  44. ؂ࢹ͞ΕΔΦϑΟεػث 

  45. ΦϑΟε಺ͷ؂ࢹ • pingͷԠ౴ΛݟΔ͜ͱͰ؂ࢹ • ΋ͪΖΜɺϧʔλʔͳͲ΋؂ࢹ͍ͯ͠Δ • ԕִ஍ΦϑΟεͷϧʔλʔ΋τϥϑΟοΫ؂ ࢹ͠ɺτϥϒϧରԠ͕Ͱ͖ΔΑ͏ʹ͍ͯ͠Δ 

  46. ͜Μͳ؂ࢹ΋͍ͯ͠·͢ ͜Μͳ͜ͱ΋͍ͯ͠·͢ 

  47. mysqlεϨʔϒ • Slave_IO_RunningͳͲΛ؂ࢹ • ఆ࣌όοΫΞοϓͳͲͷؒ͸εςʔλεΛૢ ࡞ • Ξϥʔτ͕ෆඞཁʹ໐Βͳ͍Α͏ʹ͢Δ • ॲཧʹࣦഊ͔ͨ͠Λcheck-file-ageͳͲͰ؂ࢹ

    
  48. keepalivedͷ؂ࢹ • ϓϩηε؂ࢹ • ঢ়ଶ؂ࢹʢΞΫςΟϒɾελϯόΠʣ • γΣϧεΫϦϓτͰIPΞυϨεͷଘࡏΛνΣοΫ ͠ɺঢ়ଶ͕੾ΓସΘͬͨ࣌ʹΞϥʔτΛൃใ • ো֐ɺఆܕ࡞ۀ࣌ͷ੾ΓସΘΓ͕ɺӡ༻ϝϯόʔ

    શମʹ఻ΘΔ 
  49. ϋʔυ΢ΣΞRAIDͷ؂ࢹ • DELL OpenManageͳͲͷCLI݁ՌΛύʔε • σΟεΫނোΛݕ஌ • ·ͨɺOpenManage͕ಥવࢮ͍ͯͨ͜͠ͱ΋ ͋ΔͷͰɺϓϩηεͷࢮ׆؂ࢹ •

    ஌ݟ͕͋·Γཷ·͍ͬͯͳ͍ 
  50. ͘͞ΒͷΫϥ΢υͷ՝ۚ؂ࢹ • ࡞ۀͰΠϯελϯεΛ࡞ͬͯͦͷ··ɻͷΑ ͏ͳ͜ͱ͕ͳ͍Α͏ʹɺ՝ۚͷάϥϑΛ MackerelʹPOST • ΠϯελϯεͷλάͰαʔϏεΛ͚ͭͯɺαʔ Ϗε͝ͱͷ՝ֹۚΛ͓͓ΑͦͰݟ͑ΔԽ͢Δ 

  51. lambdaɺCloudFrontͷ؂ࢹ • lambda΍CloudFrontɺAmazon Elasticsearch ServiceͳͲ࢖༻͍ͯ͠ΔͷͰɺmackerel- agent-pluginsͰ؂ࢹ • Elasticsearch Service͸ɺϗετͱͯ͠ొ࿥͠ ͍ͯΔ

    
  52. Dockerͷ؂ࢹ • ίϯςφͷىಈ͍ͯ͠Δ͔Λdocker psΛύʔ εͯ͠؂ࢹ • ·ͨ͸ɺϙʔτΛ࢖͍ͬͯΔͷͰͦΕΛ؂ࢹ ͢ΔͳͲ • ෳ਺ίϯςφΛ1ϗετͰ͋͛ΔΫϥελͷΑ

    ͏ͳӡ༻͸·͍ͩͯ͠ͳ͍ 
  53. ঎༻؀ڥͷdockerͷঢ়ଶ 

  54. ฐࣾɺChatWork࢖ͬͯ·͢ • ChatWorkɺը૾ͷURLΛషͬͯ΋ల։ͯ͘͠Ε ͳ͍… • ChatWorkͷAPIʹը૾Λड͚෇͚ΔAPI͕ͳ͍ • MackerelͷάϥϑΛChatWorkͰݟ͍ͨ • https://github.com/yseto/slack_to_chatwork

    
  55. ࢓૊Έ • Mackerel͔Βม׵༻ͷΞϓϦΛSlackͷΤϯυ ϙΠϯτͱͯ͠ొ࿥ • chatworkͷwebͷૢ࡞Λௐ΂ͯɺΞοϓϩʔ υૢ࡞Λ૊Έཱͯͯૹ৴ 

  56. chatworkͰMackerelͷάϥϑΛோΊΔ άϥϑ͕ݟΕΔ͜ͱͰٞ࿦͕νϟοτʹͪΌΜͱ࢒Δ 

  57. ·ͱΊ 

  58. ·ͱΊ • ΦϯϓϨϛεͰ͋ͬͯ΋AWSͰ͋ͬͯ΋ɺΠ ϯετʔϧ͢Δ͚ͩͰ࢖͑ΔMackerel͸ӡ༻ ͷຯํ • ΦϑΟεͷτϥϒϧ΋MackerelͰ؂ࢹ͢Δͱ ḿΔ 

  59. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠