20+ Custom mackerel Plugins in 20 minutes mackerel Meetup#7 Masahiro Nagano @kazeburo

Me • Masahiro Nagano • @kazeburo • Mercari, Inc.
 Principal Engineer
 Site Reliability Engineering (SRE) Team • BASE, Inc. Technical Advisor

We’re Hiring!! ΑΖ͓͘͠ئ͍͠·͢ʂ

Mercari Your Friendly Mobile MarketPlace JP US

Infrastructure ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥ΢υ Amazon Web Service US West (Oregon) Region JP US

Monitoring (past) ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥ΢υ Amazon Web Service US West (Oregon) Region JP US Z ZABBIX ҟͳΔzabbixͷόʔδϣϯ.. WebUIΆͪΆͪർΕ.. ؂ࢹ௥Ճ·Ͱͷεςοϓͷଟ͞.. ؂ࢹ࿙ΕɺҟͳΔ؂ࢹ಺༰.. Z ZABBIX

Monitoring (now) ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥ΢υ Amazon Web Service US West (Oregon) Region JP US JP/USͷҰׅ؂ࢹ Service & RoleʹΑΔ؅ཧ Nagios & Sensu ϕʔεͷPlugin

Migration Items, Graphs Screens, Triggers Auto Discovery, UserParameters Official mackerel plugins Official check plugins Nagios plugins Custom mackerel Plugins Custom check Pluginss + Simple Monitoring Rules Z ZABBIX ࣜʹΑΔ ؂ࢹ จࣈྻɺ਺஋ɺ ࠩ෼ɺܭࢉ݁ՌͳͲͷ σʔλܗࣜ ΧελϜσʔλͷ औಘ

Custom Plugins my-ec2-tag[go], jmx-get[go], diff-detector[go], delay-checker, interval-checker, periodic-checker, check-mysql-uptime[go], check-memcached-uptime[go], check-conntrack-free, check-crt-expiration, check-dns-rr, check-hydra-pos, check-inode, check-iptables, check-myip, check-solr-replication, check-solr-update, check-uptime, check-mysql-msr[go], mackerel-plugin-accelmail-counter, mackerel-plugin-gaurun-usage, mackerel-plugin-linux-lite, mackerel-plugin-msr[go] mackerel-plugin-ntpq, mackerel-plugin-search, mackerel-plugin-postfix mackerel-plugin-php-and-accesslog, mackerel-plugin-php-version and MORE! and utilitiy commands

my-ec2-tags $ my-ec2-tag --tag Role production EC2ΠϯελϯεͷλάΛऔಘ awsίϚϯυ஗͍ͷͰGoͰ࡞੒ ޙड़ͷdiff-detectorͱ૊Έ߹Θͤͯ࢖͏ Utility

delay-checker $ delay-checker -c 3 -- check-tcp -p 80 ࢦఆͨ͠ճ਺Τϥʔ͕࿈ଓ͢ΔͱNG max_check_attempts Φϓγϣϯ͕Ͱ͖ͨ check ؂ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins

interval-checker $ interval-checker --interval 300 -- \ my-ec2-tag --tag Role ࢦఆִͨؒ͠Ͱ؂ࢹίϚϯυ࣮ߦ ࢦఆͨ࣌ؒ͠·Ͱ͸લճͷίϚϯυ࣮ߦ݁ՌΛฦ͢ check ؂ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins

periodic-checker $ periodic-checker --range 00:00-06:00,06:30-24:00 \ -- check-tcp -p 80 ࢦఆͨ͠ൣғͷ͚࣌ؒͩ؂ࢹ ͦΕҎ֎͸OKΛฦ͢ Daily ϝϯςφϯεʗॏ͍batch͕͋Δͱ͖ʹ check ؂ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins

diff-detector $ diff-detector -- date NG: detect difference: ```@@ -1 +1 @@ -Tue May 10 08:11:42 UTC 2016 +Tue May 10 08:11:43 UTC 2016``` ίϚϯυ࣮ߦ݁Ռͷࠩ෼͕͋ΔͱΤϥʔ ec2 tagɺ/etc/passwdϑΝΠϧͷ؂ࢹ ▶ https://github.com/kazeburo/diff-detector check ؂ࢹ

check-conntrack-free $ check-conntrack-free -w 10000 -c 2000 OK: free:596376 max:655360 count:58984 `nf_conntrack_max - nf_conntrack_count` ͷ ݁Ռͷ؂ࢹɻiptables͕༗ޮͳ࣌ʹ check ؂ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins

check-crt-expiration $ check-crt-expiration -w 7 -c 5 -f /path/cert.pem OK: Expiration date: 'Jun 1 10:38:57 2016 GMT'. 20 days left ূ໌ॻͷ༗ޮظݶͷ֬ೝ check-cert-file͕ެࣜʹ௥Ճ͞Εͨ check ؂ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins

check-dns-rr $ check-dns-rr -w 3 -c 2 \ --host prod.example.service.consul OK: 3 dns-rr hosts found DNS-RRͷ݁Ռʹؚ·ΕΔIP਺ͷ؂ࢹ Consul/DNSͰ෼ࢄ͍ͯ͠Δ࣌ʹ check ؂ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins

check-inode $ check-inode -w 10 -c 5
 OK: /:1%, /dev:1%, /dev/shm:1%, /run:1%, /sys/fs/cgroup:1%, /boot: 1%, /run/user/1037:1% ͢΂ͯͷfilesystemͷinodeΛ؂ࢹ Ұ൪গͳ͍΋ͷ͕ᮢ஋ΛԼճΔͱNG mackerel-plugin-inode ͸ `inode.percentage.*.used` ʹରͯ͠؂ࢹ͕Ͱ͖ͳ͍ͷͰ࢖͍ʹ͍͘ check ؂ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins

check-memcached-uptime check ؂ࢹ $ check-memcached-uptime \ -p 11211 -w 110 -c 110 memcached Uptime OK: up 240 days, 03:08:01 ▶ https://github.com/kazeburo/go-check-memcached-uptime memcachedͷಥવࢮ => ࠶ىಈΛݕ஌͢Δ check-memcached ͸keyͷऔಘΛߦ͏

check-mysql-uptime check ؂ࢹ $ check-mysql-uptime --host=127.0.0.1 \ -w 110 -c 110 MySQL Uptime OK: up 71 days, 17:47:39 pingͰ͸λΠϛϯάʹΑΓҾ͔͔ͬΒͳ͍࠶ىಈΛݕ஌ MySQL 4.0.xͳͲʹ༗ޮ `check-mysql uptime` Ͱ΋Մೳ ▶ https://github.com/kazeburo/go-check-mysql-uptime

check-mysql-msr $ check-mysql-msr --host localhost -w 5 -c 10 MySQL Multi Source Replication OK: [O]foo-db=io:Yes,sql:Yes,behind:0 bar-db=io:Yes,sql:Yes,behind:0 main-db=io:Yes,sql:Yes,behind:0 baz- db=io:Yes,sql:Yes,behind:0 MySQL 5.7ͷMulti Source Replicationͷ༻ ͲΕ͔ҰͭͰ΋Replicationͷఀࢭ͋Δ͍͸ɺ஗Ԇ͕ᮢ஋Λ্ճΔͱNG mackerel-plugin-msr ΋ผʹ࡞ͬͨ inodeಉ༷ `mysql-msr.*.beind` ʹରͯ͠؂ࢹ͕Ͱ͖ͳ͍ͷͰ࢖͍ʹ͍͘ check ؂ࢹ ▶ https://github.com/kazeburo/go-check-mysql-msr

mackerel-plugin-linux-lite Free Memory, 100%্͕ݶͷCPU Usage CPUίΞ͋ͨΓͷϩʔυΞϕϨʔδ, ϓϩηε਺ Ұ؏ͨ͠ϝτϦΫεͷ൑அɺ؂ࢹͷᮢ஋ͷઃఆ͕Ͱ͖Δ Agent Plugin ▶ https://github.com/kazeburo/custom-mackerel-plugins

mackerel-plugin-ntpq ࣌ؒͷͣΕ(ઈର஋)ɺϦϞʔτͱͷSyncͷ༗ແ check-ntpoffset ΋͋Δ Agent Plugin ▶ https://github.com/kazeburo/custom-mackerel-plugins

mackerel-plugin-postfix Postfixͷqueue mackerel-plugin-mailq ΋͋Δ Agent Plugin ▶ https://github.com/kazeburo/custom-mackerel-plugins

mackerel-plugin-mysql-lite ίωΫγϣϯ਺/max_connectionͷׂ߹ɺϨϓϦέʔγϣϯ஗Ԇ Slave SQL Thread/IO Threadͷಈ࡞༗ແɺ εϨου਺ KuradoͰ΋ϝτϦΫεΛݟ͍ͯΔͷͰɺ؂ࢹʹඞཁͳάϥϑʹߜΔ ▶ https://github.com/kazeburo/go-mackerel-plugin-mysql-lite Agent Plugin

mackerel-plugin-php-version ݹ͍PHPΛݕ஌^^ Agent Plugin

Plugins & Language • ͍͔ͭ͘ͷίϚϯυͷݺͼग़͠ͷΈ
 = > Shell Script • ίϚϯυͷݺͼग़͠ͱ݁Ռͷॲཧ => LL (Perl) • HTTPϦΫΤετͱJSONॲཧ => LL (Perl) • AWS APIɺMySQL΁ͷ઀ଓ =>Golang ͜ΕΒΛ࢖͍׳Ε͍ͯΔ͜ͱ͕ॏཁ

ίʔυΛॻ͍ͯ໰୊Λղܾ͢Δͷ͸ָ͍͠!! mackerel͸࢖͍͕͍ͷ͋Δπʔϧ