seesaa meets Mackerel - Speaker Deck

seesaa meets Mackerel

by yseto

Slide 1

Slide 1 text

seesaa meets Mackerel γʔαʔגࣜձࣾ ੉ށ༞थ

Slide 2

Slide 2 text

ࣗݾ঺հ • ੉ށ༞थ • γʔαʔגࣜձࣾΤϯδχΞ • த్ೖࣾ4೥໨ • Mackerel͸2015೥͔Β • ϑϩϯτɺόοΫΤϯυɺӡ༻

Slide 3

Slide 3 text

ձࣾͷ঺հ • γʔαʔגࣜձࣾ • 2003೥10݄૑ۀ • γʔαʔϒϩάɺͨ͠Β͹ܝࣔ൘ɺγʔαʔ wikiͳͲͷϝσΟΞӡӦ • ϒϩάγεςϜͷOEMఏڙ

Slide 4

Slide 4 text

໨࣍ • Πϯϑϥӡ༻ͷ࢓ࣄʹ͍ͭͯ • MackerelҎલɾҎޙ • งғؾͰ؂ࢹ͍ͯͨ͠ɾ͍ͯ͠Δ • ؂ࢹ͢ΔྖҬ͸૿͍͑ͯͬͨ • ͜Μͳ؂ࢹΛ͍ͯ͠·͢

Slide 5

Slide 5 text

Πϯϑϥɾӡ༻ʹ͍ͭͯ • ΦϯϓϨϛεɺAWSΛओʹར༻͍ͯ͠Δ • ओʹӡ༻Λ΍͍ͬͯΔͷ͸3໊ఔ౓ • ख़࿅ऀ+एऀͷ૊Έ߹Θͤ • ࡉ͔͍ͱ͜Ζʹख͕ճΒͳ͍ • ࣌ʑൃੜ͢ΔϏοάόϯతΠϕϯτͰࠞཚ

Slide 6

Slide 6 text

ӡ༻ϝϯόʔͷओͳ࢓ࣄ • ೔ࠒͷΞϥʔτରԠ • Mackerel͔ΒͷΞϥʔτΛݟͯҟৗ͔Ͳ͏͔ͷݟۃΊ • ӡ༻໘͕ڧ͍։ൃ • طଘΞϓϦέʔγϣϯͷDockerԽͷςετ • ࣗࣾϝσΟΞαʔϏεʢϒϩάɾܝࣔ൘ɾWikiʣ͸ DockerԽࡁΈ

Slide 7

Slide 7 text

ΦϯϓϨϛε • DBͳͲͷෛՙ͕༧૝͞ΕΔαʔόʔ͸CentOSΛಋೖͯ͠ӡ ༻ • ΞϓϦέʔγϣϯαʔόʔͳͲ͸ɺXen ServerΛ༻͍ͯɺVM ʢCentOSʣʹͯӡ༻ • ݹ͔͘ΒͷαʔϏε͕ଟ͘ɺػثϦϓϨʔεͳͲʹ߹Θͤͯ VMɺmysqlͷઃఆͳͲฏ४Խ͍ͤͯ͞Δ • ҆ఆՔಇͯ͠΍ͬͱฏۉ఺

Slide 8

Slide 8 text

AWS • ৽نαʔϏεͳͲ͸ੵۃతʹ࠾༻͢Δ܏޲ • mackerel-agent-pluginsʹAWSؔ࿈͕ॆ࣮͍ͯ͠ ΔͷͰ҆৺ • ΠϯςάϨʔγϣϯػೳ͸ঢ়گʹԠͯ͡࢖͍෼͚ • ࢖Θͳ͍৔߹͸EC2ͷϗετϝτϦοΫʹ಺แ

Slide 9

Slide 9 text

MackerelҎલɾҎޙ

Slide 10

Slide 10 text

MackerelҎલ • ࣾ಺ΦϯϓϨϛε؀ڥ؂ࢹ • savacan ʢಡΈ:͞͹͔ΜʣͰ؂ࢹ͍ͯͨ͠ • cacti Ͱάϥϑ • 1ਓӡ༻୲౰ऀ͕ுΓ෇͖

Slide 11

Slide 11 text

savacanͷ؆୯ͳ঺հ • αʔόʔ୆ா • ϥοΫɺVMͷऩ༰ϗετ୆ா • ϩʔυόϥϯαͷίϯτϩʔϥత໾ׂ • ϑϩϯτͷwebαʔόʔ͕ઃఆΛಡΈࠐΉ • Ξϥʔτൃใʢϝʔϧʣ • ϩάϩʔςʔγϣϯิॿ

Slide 12

Slide 12 text

savacanͷΞʔΩςΫνϟ • ؂ࢹ͢ΔϗετʹεΫϦϓτΛΠϯετʔϧɺperl੡ • CartonͳͲΛߟྀͯ͠ͳ͍࡞ΓɺΠϯετʔϧ͕ෳࡶ • ϝτϦοΫ͸ू໿͢ΔmysqlʹmysqlͰ௨৴ • ؂ࢹ͸γϯάϧεϨου • ࣌ʑ٧·ͬͯ؂ࢹ͕ࢮʹ͕ͪ… • εέʔϧ͠ͳ͍ɺಠࣗCLI͸ॳ৺ऀʹͱ͔ͬͯͳΓϋʔυϧ

Slide 13

Slide 13 text

savacanͷCLIʢྫʣ sdev@xxxx$ svcn stat bl-app21 bl-app21 bl app active 1 20.1 N/A N/A sdev@xxxx$ svcn scan stat bl-app21 bl-app21 df 0 95 on bl-app21 loads 0 35 on 35 bl-app21 nfs_mount 0 3 on bl-app21 ping 0 0 on bl-app21 port 0 3 on 5000 sdev@xxxx$ svcn update bl-app21 status active sdev@xxxx$ svcn update bl-app21 status ignored

Slide 14

Slide 14 text

savacanͷCLIߏจ • svcn update HOSTNAME KEY1 VALUE1 • σʔλΛೖΕΔmysqlͷߏ଄͕ͦͷ··࢖ΘΕ͍ͯΔ • ςʔϒϧߏ଄͕Θ͔Βͳ͍ͱ࢖͑ͳ͍ • ӡ༻͢Δਓ͔͠Θ͔ͬͯͳ͍ • ӡ༻ϝϯόʔҎ֎ͷࢀೖোนʹͳ͍ͬͯΔ

Slide 15

Slide 15 text

savacanͷcliͰݟΕΔ৘ใ • load averageɺdfͷঢ়ଶ • pingɺϙʔτ؂ࢹɺNFSϚ΢ϯτͷ؂ࢹঢ়ଶ • ϩʔυόϥϯεର৅Ͱ͋Δ͔ͷ༗ແ • load averageɺdfͷঢ়ଶͳͲʹΑͬͯ͸঎༻ αʔϏε͔Βࣗಈతʹ཭୤͠ɺΞϥʔτൃใ

Slide 16

Slide 16 text

savacanͰݟΕΔաڈ • աڈͷ஋͸ࣺ͍ͯͯͨͷͰɺܧଓͨ͠ϝτϦοΫ؂ࢹ͸ Ͱ͖͍ͯͳ͔ͬͨ • ॏཁੑΛೝࣝͰ͖͍ͯͳ͔ͬͨͷͩͱࢥ͏ • MackerelʹΑͬͯաڈ͕ܧଓͯ͠ݟΕΔΑ͏ʹͳΓɺӡ ༻ෛՙΛ၆ᛌతʹݟ΍͘͢ͳͬͨ • cactiͰ͸ɺload averageఔ౓͔͠औಘ͍ͯ͠ͳ͔ͬͨ ͨΊ

Slide 17

Slide 17 text

͋Δౙɺӡ༻୲౰ऀ͕ࣙΊͨ

Slide 18

Slide 18 text

cactiΛҾ͖ܧ͝͏ͱͨ͠ • snmpͷmibΛ֦ு͠ɺएׯΦϨΦϨͳ؂ࢹΛ ͍ͯͨ͠ɻ • ଐਓత͗͢Δ… • cactiͷੵۃతͳܧଓӡ༻Λஅ೦ • ژ౎ͷํ޲͔Βޫ͕ࠩ͠ࠐΜͰ͖ͨ

Slide 19

Slide 19 text

MackerelΛࢼ༻

Slide 20

Slide 20 text

ΤʔδΣϯτͷηοτΞοϓ • ଟछଟ༷ͳ؀ڥʹࡶʹΤʔδΣϯτΛೖΕͯ άϥϑͷงғؾʹײಈ • ଈܾͰ໿200ϗετͷ؂ࢹΛ࢝ΊΔ • ઃఆͷࣽ٧Ί͕؁͘ɺΤʔδΣϯτΛೖΕͨ ͚ͩɻͱ͍͏ࡶ͗͢Δϗετ΋͋ͬͨ

Slide 21

Slide 21 text

MackerelΛೖΕͯؾ͍ͮͨ͜ͱ • VMͷੑೳઃܭ͕ద੾Ͱͳ͘ɺΧπΧπؾຯ • ֎ܗ؂ࢹʹΑͬͯࡉ͔͍εϩʔμ΢ϯͳͲݟ ͔ͭΔ • ࠓ·Ͱͷπέ͕શ෦ݟ͑ΔԽ͞ΕΔ

Slide 22

Slide 22 text

MackerelʹΑͬͯಘΒΕͨ؂ࢹ • ֎ܗ؂ࢹ • Ҏલ͔Βଘࡏ͸͍ͯͨ͠΋ͷͷࡉ͔͍࣌ؒ ࣠ͷ৘ใ͕ܽམ͕ͪ͠ • طଘ؂ࢹͰ͸ɺଟ͘ͷ৘ใΛऔಘ͍ͯ͠ͳ͔ͬ ͨͷͰɺࠓࢥ͑͹ɺงғؾͰରࡦͳͲ͍ͯ͠ ͨͷ͔΋͠Εͳ͍

Slide 23

Slide 23 text

Mackerelͷ͍͍ͱ͜Ζ • ·ͣ͸ΤʔδΣϯτΛೖΕͯঢ়گΛݟΔɻͱ͍͏ͱ͜Ζ ͔Β࢝ΊΒΕΔɻ • ΤʔδΣϯτ͸yumͳͲͰΠϯετʔϧͰ͖Δɻߋ৽΋ ׆ൃ • ෆ۩߹ɺෆ໌఺͸͙͢ʹղܾ͞ΕΔ • TwitterͰࠔͬͨɻͱݴΘͣʹʮαϙʔτνʔϜ΁࿈བྷʯ

Slide 24

Slide 24 text

զʑ͸งғؾͰ؂ࢹ͍ͯͨ͠

Slide 25

Slide 25 text

ؾ͍͖͔͚ͮͨͬ • ಥવ͸ͯͳ͞Μ͔Βɺ࢖͍উखͳͲΠϯλϏϡʔ ͍ͨ͠ͷͰ๚໰ͯ͠ྑ͍͔?ͱͷ͓੠͕͔͔Δ • ओʹɺ͜Ε·ͰʹϦϦʔεͨ͠ػೳͷར༻ͷ༗ ແɺ࢖͍উखͷώΞϦϯά • a-know͞ΜʮདྷिɺΠϕϯτ͕͋ΔͷͰͥͻ དྷͯ΄͍͠ʯ

Slide 26

Slide 26 text

2017/10/5

Slide 27

Slide 27 text

Driving Mercari with 50+ custom plugins / Mackerel DAY by kazeburo

Slide 28

Slide 28 text

Mackerel DAY • ͦΕͳΓʹ؂ࢹϧʔϧͷݟ௚͠ͳͲ͸͍ͯ͠ ͨͭ΋Γͩͬͨ • ͔͠͠ɺ΄ͱΜͲ͕৔౰ͨΓతͳݟ௚͠ • ࡉ͔͍ͱ͜Ζ·Ͱࢹ໺͕ಧ͍ͯͳ͔ͬͨ • ඦฉ͸Ұݟʹ೗͔ͣɻงғؾͰ؂ࢹ͍ͯͨ͠

Slide 29

Slide 29 text

PDCA͕ճͬͯͳ͔ͬͨ • ઃఆϑΝΠϧ͸αʔϏε͝ͱʹ؅ཧ • ࠷ॳʹ࡞ͬͨαʔϏεͷઃఆϑΝΠϧΛίϐʔͯ͠ɺઃ ఆΛࣽ٧ΊΔ • ޙͷ΄͏ʹͳΔͱɺઃఆ߲໨͕ॆ࣮ • ઃఆͷݟ௚͕͠ͳ͍ͷͰɺ࠷ॳʹೖΕͨͷ͸๨ΕΒΕΔ • ʮ͋Εɺ͜Εʹ͸͜ͷϝτϦοΫ͕ͳ͍Ͷʯ

Slide 30

Slide 30 text

Mackerelͷઃఆݟ௚͠ • งғؾա͗ͨઃఆͩͬͨͱ൓ল • mackerel-agentΛೖΕ͍ͯͨ໿300ϗετͷઃఆ ݟ௚͠Λܾஅ • mkrίϚϯυͰαʔόʔҰཡΛ࡞ΓɺઃఆϑΝΠϧ Λμ΢ϯϩʔυ • શ਺ݕࠪΛߦ͏͜ͱʹ

Slide 31

Slide 31 text

ઃఆΛ௚ͨ͠ͱ͜Ζ • ϓϩηεͷ؂ࢹΛ͍͕ͨ͠ɺΑ͘Θ͔Βͳ͍ͷͰɺ શ෦άϥϑʹग़ͤ͹͍͍Μ͡ΌͶʁ • ͦΕɺcheck-procsͰͰ͖ΔΑ • ͦ΋ͦ΋γεςϜʹඞཁෆՄܽͳϓϩηεΛશ෦ ؂ࢹͰ͖ͯͳ͔ͬͨ • ྫ͑͹ɺϩʔΧϧͷDNSΩϟογϡͳͲ

Slide 32

Slide 32 text

աڈ΍͍ࣦͬͯͨഊ [plugin.checks.xxxx] command = "TIME=`date +%s`; for x in xxx1 xxx2; do printf 'XXXXX.%s %d %d\n' $x `command $x` $TIME ; done”

Slide 33

Slide 33 text

΍ͬͯ͠·͍͕ͪ • ϫϯϥΠφʔ • ΍͚ͬͭͰॻ͍ͨΓ͢Δ • ޙ͔ΒɺಡΈͮΒ͍ • ͜͏͍ͬͨ΋ͷ͸શͯγΣϧεΫϦϓτԽ • ϓϥάΠϯ͕͋ͬͨΒͦͬͪʹࡌͤସ͑

Slide 34

Slide 34 text

Mackerel DAYͰͷऩ֭ • ϩʔϧ͝ͱʹ෼͚Δ֓೦Λਖ਼͘͠஌Δ • ϩʔϧͷઃఆ=include͢ΔઃఆϑΝΠϧΛ ಉҰͱ͢ΔελΠϧ • ΍͹͍ɺ͔͍͍ͬ͜

Slide 35

Slide 35 text

MackerelͰݟ͍ͨ৘ใ • ྫ͑͹AWSͳͲͩͱɺAZͳͲ͕ݟΕΔɻ • XenαʔόʔͷVM͸Ͳ͜ʹϗετ͍ͯ͠Δͷ ͔Θ͔Βͳ͍ • savacanͱͷೋॏ؅ཧɺMackerelݟ͚ͨͩͰ ೺Ѳ͍ͨ͠

Slide 36

Slide 36 text

Whitebait • ͠Β͢ - VM͕Ͳ͜ʹډΔ͔஌Β͢ • Xen ServerͷRPCΛୟ͍ͯVMͷҐஔ৘ใͱIPΞυϨε ҰཡΛΩϟογϡ • mackerel-agentͷchecksϓϥάΠϯͷ݁ՌͰऩ༰ϗε τΛࣔͤΕΔΑ͏ʹ • https://github.com/yseto/whitebait

Slide 37

Slide 37 text

ऩ༰ϗετ͕Θ͔Δ

Slide 38

Slide 38 text

MackerelͷઃఆϨϙδτϦ • Mackerelͷઃఆ͸ࣾ಺GitͰ؅ཧ͞Ε͍ͯΔ • ͦ΋ͦ΋ɺϗετ͝ͱʹ֎෦ͷαʔόʔͷϨϙδτϦ ʹऔΓ͜Έʹߦ͘ͷ͸credential؅ཧͳͲ໰୊͕͋Δ • ސ٬ͷػີ৘ใ͕ೖͬͯͳ͚Ε͹໰୊ͳ͍ͱ൑அ͠ɺ WhitebaitʹGitϨϙδτϦͷϗεςΟϯάػೳΛ࣋ͨ ͤΔʢpull͚ͩͰ͖Δʣ

Slide 39

Slide 39 text

૿͑Δ؂ࢹྖҬ

Slide 40

Slide 40 text

ΦϯϓϨϛεεΠον • ΦϯϓϨϛε؀ڥͷεΠονɺࠓ·Ͱৄ͘͠τϥ ϑΟοΫͷ֬ೝͳͲ͍ͯ͠ͳ͔ͬͨ • τϥϒϧ͕͋ͬͨͷͰɺMackerelͰ؂ࢹ͢Δ͜ͱ ʹ… • ͦ΋ͦ΋ɺMackerelͰεΠονͷάϥϑग़ͨ͠ ͍͚ͩͳͷ͕ͩ…

Slide 41

Slide 41 text

OSSΛௐ΂Δ • https://github.com/sorah/snmp2mkr • CookpadͷͦΒ͸͞Μ͕࡞͍ͬͯΔOSS • SNMPΛू໿ͯ͠ɺϗετͱͯ͠POSTͯ͘͠ΕΔ • ݩ͸ɺΠϕϯτͷWi-Fi؂ࢹ༻ͷΑ͏͕ͩɺͱΓ͋͑ͣઃఆ • ର৅εΠον͸ϙʔτ਺͕ଟ͘ɺऔΓ͜΅͕͋͠Δ΋ͷͷɺ࣌ ؒ࣠Λେ͖͘ͱΕ͹܏޲͸͔ͭΊΔ • ͪΌΜͱه࿥͍ͨ͠ͳΒɺ͕Μ͹Δ΂͖

Slide 42

Slide 42 text

ʮϓϦϯλಈ͔ͳ͍Μ͚ͩͲʁʯ ϑϦʔૉࡐͺͨͦ͘ ΑΓ

Slide 43

Slide 43 text

ΦϑΟεͰ͋Δ͋Δ • ؾܰʹ੠͕͔͚ΒΕΔΤϯδχΞʹॿ͚Λٻ Ίͯ͘Δύλʔϯ • ΦϑΟεͷॄث΋؂ࢹͰ͖Δ΍ͭ͸ͯ͠͠· ͓͏ • ΞϥʔτൃใͰɺτϥϒϧ͕ൃੜ͢Δલʹ ରॲ͢Δ

Slide 44

Slide 44 text

؂ࢹ͞ΕΔΦϑΟεػث

Slide 45

Slide 45 text

ΦϑΟε಺ͷ؂ࢹ • pingͷԠ౴ΛݟΔ͜ͱͰ؂ࢹ • ΋ͪΖΜɺϧʔλʔͳͲ΋؂ࢹ͍ͯ͠Δ • ԕִ஍ΦϑΟεͷϧʔλʔ΋τϥϑΟοΫ؂ ࢹ͠ɺτϥϒϧରԠ͕Ͱ͖ΔΑ͏ʹ͍ͯ͠Δ

Slide 46

Slide 46 text

͜Μͳ؂ࢹ΋͍ͯ͠·͢ ͜Μͳ͜ͱ΋͍ͯ͠·͢

Slide 47

Slide 47 text

mysqlεϨʔϒ • Slave_IO_RunningͳͲΛ؂ࢹ • ఆ࣌όοΫΞοϓͳͲͷؒ͸εςʔλεΛૢ ࡞ • Ξϥʔτ͕ෆඞཁʹ໐Βͳ͍Α͏ʹ͢Δ • ॲཧʹࣦഊ͔ͨ͠Λcheck-ﬁle-ageͳͲͰ؂ࢹ

Slide 48

Slide 48 text

keepalivedͷ؂ࢹ • ϓϩηε؂ࢹ • ঢ়ଶ؂ࢹʢΞΫςΟϒɾελϯόΠʣ • γΣϧεΫϦϓτͰIPΞυϨεͷଘࡏΛνΣοΫ ͠ɺঢ়ଶ͕੾ΓସΘͬͨ࣌ʹΞϥʔτΛൃใ • ো֐ɺఆܕ࡞ۀ࣌ͷ੾ΓସΘΓ͕ɺӡ༻ϝϯόʔ શମʹ఻ΘΔ

Slide 49

Slide 49 text

ϋʔυ΢ΣΞRAIDͷ؂ࢹ • DELL OpenManageͳͲͷCLI݁ՌΛύʔε • σΟεΫނোΛݕ஌ • ·ͨɺOpenManage͕ಥવࢮ͍ͯͨ͜͠ͱ΋ ͋ΔͷͰɺϓϩηεͷࢮ׆؂ࢹ • ஌ݟ͕͋·Γཷ·͍ͬͯͳ͍

Slide 50

Slide 50 text

͘͞ΒͷΫϥ΢υͷ՝ۚ؂ࢹ • ࡞ۀͰΠϯελϯεΛ࡞ͬͯͦͷ··ɻͷΑ ͏ͳ͜ͱ͕ͳ͍Α͏ʹɺ՝ۚͷάϥϑΛ MackerelʹPOST • ΠϯελϯεͷλάͰαʔϏεΛ͚ͭͯɺαʔ Ϗε͝ͱͷ՝ֹۚΛ͓͓ΑͦͰݟ͑ΔԽ͢Δ

Slide 51

Slide 51 text

lambdaɺCloudFrontͷ؂ࢹ • lambda΍CloudFrontɺAmazon Elasticsearch ServiceͳͲ࢖༻͍ͯ͠ΔͷͰɺmackerel- agent-pluginsͰ؂ࢹ • Elasticsearch Service͸ɺϗετͱͯ͠ొ࿥͠ ͍ͯΔ

Slide 52

Slide 52 text

Dockerͷ؂ࢹ • ίϯςφͷىಈ͍ͯ͠Δ͔Λdocker psΛύʔ εͯ͠؂ࢹ • ·ͨ͸ɺϙʔτΛ࢖͍ͬͯΔͷͰͦΕΛ؂ࢹ ͢ΔͳͲ • ෳ਺ίϯςφΛ1ϗετͰ͋͛ΔΫϥελͷΑ ͏ͳӡ༻͸·͍ͩͯ͠ͳ͍

Slide 53

Slide 53 text

঎༻؀ڥͷdockerͷঢ়ଶ

Slide 54

Slide 54 text

ฐࣾɺChatWork࢖ͬͯ·͢ • ChatWorkɺը૾ͷURLΛషͬͯ΋ల։ͯ͘͠Ε ͳ͍… • ChatWorkͷAPIʹը૾Λड͚෇͚ΔAPI͕ͳ͍ • MackerelͷάϥϑΛChatWorkͰݟ͍ͨ • https://github.com/yseto/slack_to_chatwork

Slide 55

Slide 55 text

࢓૊Έ • Mackerel͔Βม׵༻ͷΞϓϦΛSlackͷΤϯυ ϙΠϯτͱͯ͠ొ࿥ • chatworkͷwebͷૢ࡞Λௐ΂ͯɺΞοϓϩʔ υૢ࡞Λ૊Έཱͯͯૹ৴

Slide 56

Slide 56 text

chatworkͰMackerelͷάϥϑΛோΊΔ άϥϑ͕ݟΕΔ͜ͱͰٞ࿦͕νϟοτʹͪΌΜͱ࢒Δ

Slide 57

Slide 57 text

·ͱΊ

Slide 58

Slide 58 text

·ͱΊ • ΦϯϓϨϛεͰ͋ͬͯ΋AWSͰ͋ͬͯ΋ɺΠ ϯετʔϧ͢Δ͚ͩͰ࢖͑ΔMackerel͸ӡ༻ ͷຯํ • ΦϑΟεͷτϥϒϧ΋MackerelͰ؂ࢹ͢Δͱ ḿΔ

Slide 59

Slide 59 text

͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠