Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
20+ custom mackerel plugins in 20 minutes
Search
kazeburo
May 12, 2016
Technology
2
3.4k
20+ custom mackerel plugins in 20 minutes
mackerel meetup #7 Tokyo
mercariでmackerelを導入したときに作ったpluginの紹介です #mackerelio
kazeburo
May 12, 2016
Tweet
Share
More Decks by kazeburo
See All by kazeburo
DNS水責め攻撃と監視 / DNS water torture attack Monitoring and SLO
kazeburo
4
3.6k
DBやめてみた / DNS water torture attack and countermeasures
kazeburo
13
11k
IaaSにおけるPlatform Engineeringとこれから / Platform engineering in IaaS
kazeburo
2
1.1k
高信頼IaaSを実現するDevOps / DevOps for Highly Reliable IaaS
kazeburo
1
460
権威DNSサービスへのDDoSと ハイパフォーマンスなベンチマーカ / DNS Pseudo random subdomain attack and High performance Benchmarker
kazeburo
3
4.5k
DNS権威サーバのクラウドサービス向けに行われた攻撃および対策 / DNS Pseudo-Random Subdomain Attack and mitigations
kazeburo
7
11k
sacloudns
kazeburo
2
280
「orchestratorとGTID運用を支える監視」の勉強 / Monitoring orchestrator and GTID operation
kazeburo
2
1.2k
最近の監視(仮)/Recent system monitoring with mackerel
kazeburo
3
4.5k
Other Decks in Technology
See All in Technology
Cracking the KubeCon CfP
inductor
2
250
Compose Compiler Metricsを使った実践的なコードレビュー
tomorrowkey
1
220
Delivering Millions of Messages within seconds @ Duolingo
pelelgrino
0
350
いつか使うかも貯金してたらめちゃめちゃ機能が増えてた話
riyaamemiya
0
290
サーバー間 GraphQL と webmock-graphql の話 / server-to-server graphql and webmock-graphql
qsona
2
190
プラットフォームってつくることより計測することが重要なんじゃないかという話 / Platform Engineering Meetup #8
taishin
1
370
Meta Quest 3 で動く桜マシマシ WebXR アプリを IBM Cloud Code Engine と Babylon.js で作った話
1ftseabass
PRO
0
120
JAWS-UG Bedrock Claude Night
yamahiro
3
610
Building a RAG-poweredAI chat appwith Python and VS Code
pamelafox
0
100
ExaDB-D dbaascli で出来ること
oracle4engineer
PRO
0
2.1k
Gitlab本から学んだこと - そーだいなるプレイバック / gitlab-book
soudai
4
440
Reducing Cross-Zone Egress at Spotify with Custom gRPC Load Balancing Recap
koh_naga
0
210
Featured
See All Featured
The Invisible Side of Design
smashingmag
294
49k
The MySQL Ecosystem @ GitHub 2015
samlambert
243
12k
Rails Girls Zürich Keynote
gr2m
91
13k
The Power of CSS Pseudo Elements
geoffreycrofte
60
5k
Art, The Web, and Tiny UX
lynnandtonic
289
19k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
21
1.6k
The Pragmatic Product Professional
lauravandoore
25
5.8k
Designing for Performance
lara
601
67k
Why You Should Never Use an ORM
jnunemaker
PRO
51
8.6k
What's new in Ruby 2.0
geeforr
337
31k
Being A Developer After 40
akosma
57
580k
Designing Experiences People Love
moore
136
23k
Transcript
20+ Custom mackerel Plugins in 20 minutes mackerel Meetup#7 Masahiro
Nagano @kazeburo
Me • Masahiro Nagano • @kazeburo • Mercari, Inc. Principal
Engineer Site Reliability Engineering (SRE) Team • BASE, Inc. Technical Advisor
We’re Hiring!! ΑΖ͓͘͠ئ͍͠·͢ʂ
Mercari Your Friendly Mobile MarketPlace JP US
Infrastructure ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥυ Amazon Web Service US West (Oregon)
Region JP US
Monitoring (past) ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥυ Amazon Web Service US West
(Oregon) Region JP US Z ZABBIX ҟͳΔzabbixͷόʔδϣϯ.. WebUIΆͪΆͪർΕ.. ࢹՃ·Ͱͷεςοϓͷଟ͞.. ࢹ࿙ΕɺҟͳΔࢹ༰.. Z ZABBIX
Monitoring (now) ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥυ Amazon Web Service US West
(Oregon) Region JP US JP/USͷҰׅࢹ Service & RoleʹΑΔཧ Nagios & Sensu ϕʔεͷPlugin
Migration Items, Graphs Screens, Triggers Auto Discovery, UserParameters Official mackerel
plugins Official check plugins Nagios plugins Custom mackerel Plugins Custom check Pluginss + Simple Monitoring Rules Z ZABBIX ࣜʹΑΔ ࢹ จࣈྻɺɺ ࠩɺܭࢉ݁ՌͳͲͷ σʔλܗࣜ ΧελϜσʔλͷ औಘ
Custom Plugins my-ec2-tag[go], jmx-get[go], diff-detector[go], delay-checker, interval-checker, periodic-checker, check-mysql-uptime[go], check-memcached-uptime[go],
check-conntrack-free, check-crt-expiration, check-dns-rr, check-hydra-pos, check-inode, check-iptables, check-myip, check-solr-replication, check-solr-update, check-uptime, check-mysql-msr[go], mackerel-plugin-accelmail-counter, mackerel-plugin-gaurun-usage, mackerel-plugin-linux-lite, mackerel-plugin-msr[go] mackerel-plugin-ntpq, mackerel-plugin-search, mackerel-plugin-postfix mackerel-plugin-php-and-accesslog, mackerel-plugin-php-version and MORE! and utilitiy commands
my-ec2-tags $ my-ec2-tag --tag Role production EC2ΠϯελϯεͷλάΛऔಘ awsίϚϯυ͍ͷͰGoͰ࡞ ޙड़ͷdiff-detectorͱΈ߹Θͤͯ͏ Utility
delay-checker $ delay-checker -c 3 -- check-tcp -p 80 ࢦఆͨ͠ճΤϥʔ͕࿈ଓ͢ΔͱNG
max_check_attempts Φϓγϣϯ͕Ͱ͖ͨ check ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins
interval-checker $ interval-checker --interval 300 -- \ my-ec2-tag --tag Role
ࢦఆִͨؒ͠ͰࢹίϚϯυ࣮ߦ ࢦఆͨ࣌ؒ͠·ͰલճͷίϚϯυ࣮ߦ݁ՌΛฦ͢ check ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins
periodic-checker $ periodic-checker --range 00:00-06:00,06:30-24:00 \ -- check-tcp -p 80
ࢦఆͨ͠ൣғͷ͚࣌ؒͩࢹ ͦΕҎ֎OKΛฦ͢ Daily ϝϯςφϯεʗॏ͍batch͕͋Δͱ͖ʹ check ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins
diff-detector $ diff-detector -- date NG: detect difference: ```@@ -1
+1 @@ -Tue May 10 08:11:42 UTC 2016 +Tue May 10 08:11:43 UTC 2016``` ίϚϯυ࣮ߦ݁Ռͷ͕ࠩ͋ΔͱΤϥʔ ec2 tagɺ/etc/passwdϑΝΠϧͷࢹ ▶ https://github.com/kazeburo/diff-detector check ࢹ
check-conntrack-free $ check-conntrack-free -w 10000 -c 2000 OK: free:596376 max:655360
count:58984 `nf_conntrack_max - nf_conntrack_count` ͷ ݁Ռͷࢹɻiptables͕༗ޮͳ࣌ʹ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-crt-expiration $ check-crt-expiration -w 7 -c 5 -f /path/cert.pem OK:
Expiration date: 'Jun 1 10:38:57 2016 GMT'. 20 days left ূ໌ॻͷ༗ޮظݶͷ֬ೝ check-cert-file͕ެࣜʹՃ͞Εͨ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-dns-rr $ check-dns-rr -w 3 -c 2 \ --host prod.example.service.consul
OK: 3 dns-rr hosts found DNS-RRͷ݁Ռʹؚ·ΕΔIPͷࢹ Consul/DNSͰࢄ͍ͯ͠Δ࣌ʹ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-inode $ check-inode -w 10 -c 5 OK: /:1%, /dev:1%,
/dev/shm:1%, /run:1%, /sys/fs/cgroup:1%, /boot: 1%, /run/user/1037:1% ͯ͢ͷfilesystemͷinodeΛࢹ Ұ൪গͳ͍ͷ͕ᮢΛԼճΔͱNG mackerel-plugin-inode `inode.percentage.*.used` ʹରͯ͠ࢹ͕Ͱ͖ͳ͍ͷͰ͍ʹ͍͘ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-memcached-uptime check ࢹ $ check-memcached-uptime \ -p 11211 -w 110
-c 110 memcached Uptime OK: up 240 days, 03:08:01 ▶ https://github.com/kazeburo/go-check-memcached-uptime memcachedͷಥવࢮ => ࠶ىಈΛݕ͢Δ check-memcached keyͷऔಘΛߦ͏
check-mysql-uptime check ࢹ $ check-mysql-uptime --host=127.0.0.1 \ -w 110 -c
110 MySQL Uptime OK: up 71 days, 17:47:39 pingͰλΠϛϯάʹΑΓҾ͔͔ͬΒͳ͍࠶ىಈΛݕ MySQL 4.0.xͳͲʹ༗ޮ `check-mysql uptime` ͰՄೳ ▶ https://github.com/kazeburo/go-check-mysql-uptime
check-mysql-msr $ check-mysql-msr --host localhost -w 5 -c 10 MySQL
Multi Source Replication OK: [O]foo-db=io:Yes,sql:Yes,behind:0 bar-db=io:Yes,sql:Yes,behind:0 main-db=io:Yes,sql:Yes,behind:0 baz- db=io:Yes,sql:Yes,behind:0 MySQL 5.7ͷMulti Source Replicationͷ༻ ͲΕ͔ҰͭͰReplicationͷఀࢭ͋Δ͍ɺԆ͕ᮢΛ্ճΔͱNG mackerel-plugin-msr ผʹ࡞ͬͨ inodeಉ༷ `mysql-msr.*.beind` ʹରͯ͠ࢹ͕Ͱ͖ͳ͍ͷͰ͍ʹ͍͘ check ࢹ ▶ https://github.com/kazeburo/go-check-mysql-msr
mackerel-plugin-linux-lite Free Memory, 100%্͕ݶͷCPU Usage CPUίΞ͋ͨΓͷϩʔυΞϕϨʔδ, ϓϩηε Ұ؏ͨ͠ϝτϦΫεͷஅɺࢹͷᮢͷઃఆ͕Ͱ͖Δ Agent Plugin
▶ https://github.com/kazeburo/custom-mackerel-plugins
mackerel-plugin-ntpq ࣌ؒͷͣΕ(ઈର)ɺϦϞʔτͱͷSyncͷ༗ແ check-ntpoffset ͋Δ Agent Plugin ▶ https://github.com/kazeburo/custom-mackerel-plugins
mackerel-plugin-postfix Postfixͷqueue mackerel-plugin-mailq ͋Δ Agent Plugin ▶ https://github.com/kazeburo/custom-mackerel-plugins
mackerel-plugin-mysql-lite ίωΫγϣϯ/max_connectionͷׂ߹ɺϨϓϦέʔγϣϯԆ Slave SQL Thread/IO Threadͷಈ࡞༗ແɺ εϨου KuradoͰϝτϦΫεΛݟ͍ͯΔͷͰɺࢹʹඞཁͳάϥϑʹߜΔ ▶ https://github.com/kazeburo/go-mackerel-plugin-mysql-lite
Agent Plugin
mackerel-plugin-php-version ݹ͍PHPΛݕ^^ Agent Plugin
Plugins & Language • ͍͔ͭ͘ͷίϚϯυͷݺͼग़͠ͷΈ = > Shell Script •
ίϚϯυͷݺͼग़͠ͱ݁Ռͷॲཧ => LL (Perl) • HTTPϦΫΤετͱJSONॲཧ => LL (Perl) • AWS APIɺMySQLͷଓ =>Golang ͜ΕΒΛ͍׳Ε͍ͯΔ͜ͱ͕ॏཁ
ίʔυΛॻ͍ͯΛղܾ͢Δͷָ͍͠!! mackerel͍͕͍ͷ͋Δπʔϧ