Upgrade to Pro — share decks privately, control downloads, hide ads and more …

20+ custom mackerel plugins in 20 minutes

20+ custom mackerel plugins in 20 minutes

mackerel meetup #7 Tokyo
mercariでmackerelを導入したときに作ったpluginの紹介です #mackerelio

kazeburo

May 12, 2016
Tweet

More Decks by kazeburo

Other Decks in Technology

Transcript

  1. Me • Masahiro Nagano • @kazeburo • Mercari, Inc.
 Principal

    Engineer
 Site Reliability Engineering (SRE) Team • BASE, Inc. Technical Advisor
  2. Monitoring (past) ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥ΢υ Amazon Web Service US West

    (Oregon) Region JP US Z ZABBIX ҟͳΔzabbixͷόʔδϣϯ.. WebUIΆͪΆͪർΕ.. ؂ࢹ௥Ճ·Ͱͷεςοϓͷଟ͞.. ؂ࢹ࿙ΕɺҟͳΔ؂ࢹ಺༰.. Z ZABBIX
  3. Monitoring (now) ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥ΢υ Amazon Web Service US West

    (Oregon) Region JP US JP/USͷҰׅ؂ࢹ Service & RoleʹΑΔ؅ཧ Nagios & Sensu ϕʔεͷPlugin
  4. Migration Items, Graphs Screens, Triggers Auto Discovery, UserParameters Official mackerel

    plugins Official check plugins Nagios plugins Custom mackerel Plugins Custom check Pluginss + Simple Monitoring Rules Z ZABBIX ࣜʹΑΔ ؂ࢹ จࣈྻɺ਺஋ɺ ࠩ෼ɺܭࢉ݁ՌͳͲͷ σʔλܗࣜ ΧελϜσʔλͷ औಘ
  5. Custom Plugins my-ec2-tag[go], jmx-get[go], diff-detector[go], delay-checker, interval-checker, periodic-checker, check-mysql-uptime[go], check-memcached-uptime[go],

    check-conntrack-free, check-crt-expiration, check-dns-rr, check-hydra-pos, check-inode, check-iptables, check-myip, check-solr-replication, check-solr-update, check-uptime, check-mysql-msr[go], mackerel-plugin-accelmail-counter, mackerel-plugin-gaurun-usage, mackerel-plugin-linux-lite, mackerel-plugin-msr[go] mackerel-plugin-ntpq, mackerel-plugin-search, mackerel-plugin-postfix mackerel-plugin-php-and-accesslog, mackerel-plugin-php-version and MORE! and utilitiy commands
  6. delay-checker $ delay-checker -c 3 -- check-tcp -p 80 ࢦఆͨ͠ճ਺Τϥʔ͕࿈ଓ͢ΔͱNG

    max_check_attempts Φϓγϣϯ͕Ͱ͖ͨ check ؂ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins
  7. interval-checker $ interval-checker --interval 300 -- \ my-ec2-tag --tag Role

    ࢦఆִͨؒ͠Ͱ؂ࢹίϚϯυ࣮ߦ ࢦఆͨ࣌ؒ͠·Ͱ͸લճͷίϚϯυ࣮ߦ݁ՌΛฦ͢ check ؂ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins
  8. periodic-checker $ periodic-checker --range 00:00-06:00,06:30-24:00 \ -- check-tcp -p 80

    ࢦఆͨ͠ൣғͷ͚࣌ؒͩ؂ࢹ ͦΕҎ֎͸OKΛฦ͢ Daily ϝϯςφϯεʗॏ͍batch͕͋Δͱ͖ʹ check ؂ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins
  9. diff-detector $ diff-detector -- date NG: detect difference: ```@@ -1

    +1 @@ -Tue May 10 08:11:42 UTC 2016 +Tue May 10 08:11:43 UTC 2016``` ίϚϯυ࣮ߦ݁Ռͷࠩ෼͕͋ΔͱΤϥʔ ec2 tagɺ/etc/passwdϑΝΠϧͷ؂ࢹ ▶ https://github.com/kazeburo/diff-detector check ؂ࢹ
  10. check-conntrack-free $ check-conntrack-free -w 10000 -c 2000 OK: free:596376 max:655360

    count:58984 `nf_conntrack_max - nf_conntrack_count` ͷ ݁Ռͷ؂ࢹɻiptables͕༗ޮͳ࣌ʹ check ؂ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
  11. check-crt-expiration $ check-crt-expiration -w 7 -c 5 -f /path/cert.pem OK:

    Expiration date: 'Jun 1 10:38:57 2016 GMT'. 20 days left ূ໌ॻͷ༗ޮظݶͷ֬ೝ check-cert-file͕ެࣜʹ௥Ճ͞Εͨ check ؂ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
  12. check-dns-rr $ check-dns-rr -w 3 -c 2 \ --host prod.example.service.consul

    OK: 3 dns-rr hosts found DNS-RRͷ݁Ռʹؚ·ΕΔIP਺ͷ؂ࢹ Consul/DNSͰ෼ࢄ͍ͯ͠Δ࣌ʹ check ؂ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
  13. check-inode $ check-inode -w 10 -c 5
 OK: /:1%, /dev:1%,

    /dev/shm:1%, /run:1%, /sys/fs/cgroup:1%, /boot: 1%, /run/user/1037:1% ͢΂ͯͷfilesystemͷinodeΛ؂ࢹ Ұ൪গͳ͍΋ͷ͕ᮢ஋ΛԼճΔͱNG mackerel-plugin-inode ͸ `inode.percentage.*.used` ʹରͯ͠؂ࢹ͕Ͱ͖ͳ͍ͷͰ࢖͍ʹ͍͘ check ؂ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
  14. check-memcached-uptime check ؂ࢹ $ check-memcached-uptime \ -p 11211 -w 110

    -c 110 memcached Uptime OK: up 240 days, 03:08:01 ▶ https://github.com/kazeburo/go-check-memcached-uptime memcachedͷಥવࢮ => ࠶ىಈΛݕ஌͢Δ check-memcached ͸keyͷऔಘΛߦ͏
  15. check-mysql-uptime check ؂ࢹ $ check-mysql-uptime --host=127.0.0.1 \ -w 110 -c

    110 MySQL Uptime OK: up 71 days, 17:47:39 pingͰ͸λΠϛϯάʹΑΓҾ͔͔ͬΒͳ͍࠶ىಈΛݕ஌ MySQL 4.0.xͳͲʹ༗ޮ `check-mysql uptime` Ͱ΋Մೳ ▶ https://github.com/kazeburo/go-check-mysql-uptime
  16. check-mysql-msr $ check-mysql-msr --host localhost -w 5 -c 10 MySQL

    Multi Source Replication OK: [O]foo-db=io:Yes,sql:Yes,behind:0 bar-db=io:Yes,sql:Yes,behind:0 main-db=io:Yes,sql:Yes,behind:0 baz- db=io:Yes,sql:Yes,behind:0 MySQL 5.7ͷMulti Source Replicationͷ༻ ͲΕ͔ҰͭͰ΋Replicationͷఀࢭ͋Δ͍͸ɺ஗Ԇ͕ᮢ஋Λ্ճΔͱNG mackerel-plugin-msr ΋ผʹ࡞ͬͨ inodeಉ༷ `mysql-msr.*.beind` ʹରͯ͠؂ࢹ͕Ͱ͖ͳ͍ͷͰ࢖͍ʹ͍͘ check ؂ࢹ ▶ https://github.com/kazeburo/go-check-mysql-msr
  17. Plugins & Language • ͍͔ͭ͘ͷίϚϯυͷݺͼग़͠ͷΈ
 = > Shell Script •

    ίϚϯυͷݺͼग़͠ͱ݁Ռͷॲཧ => LL (Perl) • HTTPϦΫΤετͱJSONॲཧ => LL (Perl) • AWS APIɺMySQL΁ͷ઀ଓ =>Golang ͜ΕΒΛ࢖͍׳Ε͍ͯΔ͜ͱ͕ॏཁ