Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
20+ custom mackerel plugins in 20 minutes
Search
kazeburo
May 12, 2016
Technology
2
3.6k
20+ custom mackerel plugins in 20 minutes
mackerel meetup #7 Tokyo
mercariでmackerelを導入したときに作ったpluginの紹介です #mackerelio
kazeburo
May 12, 2016
Tweet
Share
More Decks by kazeburo
See All by kazeburo
[SRE kaigi 2025] ガバメントクラウドに向けた開発と変化するSRE組織のあり方 / Development for Government Cloud and the Evolving Role of SRE Teams
kazeburo
4
2.2k
[さくらのTech Day] ガバメントクラウド開発と変化と成長する組織 / sakura techday, Develop govcloud and the team
kazeburo
0
5.5k
ガバメントクラウド開発と変化と成長する組織 / Organizational change and growth in developing a government cloud
kazeburo
4
2.5k
DNS水責め攻撃と監視 / DNS water torture attack Monitoring and SLO
kazeburo
5
4.1k
DBやめてみた / DNS water torture attack and countermeasures
kazeburo
13
13k
IaaSにおけるPlatform Engineeringとこれから / Platform engineering in IaaS
kazeburo
2
1.3k
高信頼IaaSを実現するDevOps / DevOps for Highly Reliable IaaS
kazeburo
1
610
権威DNSサービスへのDDoSと ハイパフォーマンスなベンチマーカ / DNS Pseudo random subdomain attack and High performance Benchmarker
kazeburo
3
5.3k
DNS権威サーバのクラウドサービス向けに行われた攻撃および対策 / DNS Pseudo-Random Subdomain Attack and mitigations
kazeburo
7
13k
Other Decks in Technology
See All in Technology
デスクトップだけじゃないUbuntu
mtyshibata
0
560
PHPカンファレンス名古屋-テックリードの経験から学んだ設計の教訓
hayatokudou
2
510
Helm , Kustomize に代わる !? 次世代 k8s パッケージマネージャー Glasskube 入門 / glasskube-entry
parupappa2929
0
270
Oracle Cloud Infrastructure:2025年2月度サービス・アップデート
oracle4engineer
PRO
1
340
Tech Blogを書きやすい環境づくり
lycorptech_jp
PRO
1
260
LINEギフトにおけるバックエンド開発
lycorptech_jp
PRO
0
110
わたしのOSS活動
kazupon
2
300
N=1から解き明かすAWS ソリューションアーキテクトの魅力
kiiwami
0
140
NFV基盤のOpenStack更新 ~9世代バージョンアップへの挑戦~
vtj
0
230
【詳説】コンテンツ配信 システムの複数機能 基盤への拡張
hatena
0
110
分解して理解する Aspire
nenonaninu
2
490
依存パッケージの更新はコツコツが勝つコツ! / phpcon_nagoya2025
blue_goheimochi
3
180
Featured
See All Featured
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Agile that works and the tools we love
rasmusluckow
328
21k
Site-Speed That Sticks
csswizardry
4
390
Building an army of robots
kneath
303
45k
Writing Fast Ruby
sferik
628
61k
The Cult of Friendly URLs
andyhume
78
6.2k
We Have a Design System, Now What?
morganepeng
51
7.4k
Fashionably flexible responsive web design (full day workshop)
malarkey
406
66k
Designing on Purpose - Digital PM Summit 2013
jponch
117
7.1k
Into the Great Unknown - MozCon
thekraken
35
1.6k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
100
18k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
21
2.5k
Transcript
20+ Custom mackerel Plugins in 20 minutes mackerel Meetup#7 Masahiro
Nagano @kazeburo
Me • Masahiro Nagano • @kazeburo • Mercari, Inc. Principal
Engineer Site Reliability Engineering (SRE) Team • BASE, Inc. Technical Advisor
We’re Hiring!! ΑΖ͓͘͠ئ͍͠·͢ʂ
Mercari Your Friendly Mobile MarketPlace JP US
Infrastructure ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥυ Amazon Web Service US West (Oregon)
Region JP US
Monitoring (past) ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥυ Amazon Web Service US West
(Oregon) Region JP US Z ZABBIX ҟͳΔzabbixͷόʔδϣϯ.. WebUIΆͪΆͪർΕ.. ࢹՃ·Ͱͷεςοϓͷଟ͞.. ࢹ࿙ΕɺҟͳΔࢹ༰.. Z ZABBIX
Monitoring (now) ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥυ Amazon Web Service US West
(Oregon) Region JP US JP/USͷҰׅࢹ Service & RoleʹΑΔཧ Nagios & Sensu ϕʔεͷPlugin
Migration Items, Graphs Screens, Triggers Auto Discovery, UserParameters Official mackerel
plugins Official check plugins Nagios plugins Custom mackerel Plugins Custom check Pluginss + Simple Monitoring Rules Z ZABBIX ࣜʹΑΔ ࢹ จࣈྻɺɺ ࠩɺܭࢉ݁ՌͳͲͷ σʔλܗࣜ ΧελϜσʔλͷ औಘ
Custom Plugins my-ec2-tag[go], jmx-get[go], diff-detector[go], delay-checker, interval-checker, periodic-checker, check-mysql-uptime[go], check-memcached-uptime[go],
check-conntrack-free, check-crt-expiration, check-dns-rr, check-hydra-pos, check-inode, check-iptables, check-myip, check-solr-replication, check-solr-update, check-uptime, check-mysql-msr[go], mackerel-plugin-accelmail-counter, mackerel-plugin-gaurun-usage, mackerel-plugin-linux-lite, mackerel-plugin-msr[go] mackerel-plugin-ntpq, mackerel-plugin-search, mackerel-plugin-postfix mackerel-plugin-php-and-accesslog, mackerel-plugin-php-version and MORE! and utilitiy commands
my-ec2-tags $ my-ec2-tag --tag Role production EC2ΠϯελϯεͷλάΛऔಘ awsίϚϯυ͍ͷͰGoͰ࡞ ޙड़ͷdiff-detectorͱΈ߹Θͤͯ͏ Utility
delay-checker $ delay-checker -c 3 -- check-tcp -p 80 ࢦఆͨ͠ճΤϥʔ͕࿈ଓ͢ΔͱNG
max_check_attempts Φϓγϣϯ͕Ͱ͖ͨ check ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins
interval-checker $ interval-checker --interval 300 -- \ my-ec2-tag --tag Role
ࢦఆִͨؒ͠ͰࢹίϚϯυ࣮ߦ ࢦఆͨ࣌ؒ͠·ͰલճͷίϚϯυ࣮ߦ݁ՌΛฦ͢ check ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins
periodic-checker $ periodic-checker --range 00:00-06:00,06:30-24:00 \ -- check-tcp -p 80
ࢦఆͨ͠ൣғͷ͚࣌ؒͩࢹ ͦΕҎ֎OKΛฦ͢ Daily ϝϯςφϯεʗॏ͍batch͕͋Δͱ͖ʹ check ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins
diff-detector $ diff-detector -- date NG: detect difference: ```@@ -1
+1 @@ -Tue May 10 08:11:42 UTC 2016 +Tue May 10 08:11:43 UTC 2016``` ίϚϯυ࣮ߦ݁Ռͷ͕ࠩ͋ΔͱΤϥʔ ec2 tagɺ/etc/passwdϑΝΠϧͷࢹ ▶ https://github.com/kazeburo/diff-detector check ࢹ
check-conntrack-free $ check-conntrack-free -w 10000 -c 2000 OK: free:596376 max:655360
count:58984 `nf_conntrack_max - nf_conntrack_count` ͷ ݁Ռͷࢹɻiptables͕༗ޮͳ࣌ʹ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-crt-expiration $ check-crt-expiration -w 7 -c 5 -f /path/cert.pem OK:
Expiration date: 'Jun 1 10:38:57 2016 GMT'. 20 days left ূ໌ॻͷ༗ޮظݶͷ֬ೝ check-cert-file͕ެࣜʹՃ͞Εͨ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-dns-rr $ check-dns-rr -w 3 -c 2 \ --host prod.example.service.consul
OK: 3 dns-rr hosts found DNS-RRͷ݁Ռʹؚ·ΕΔIPͷࢹ Consul/DNSͰࢄ͍ͯ͠Δ࣌ʹ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-inode $ check-inode -w 10 -c 5 OK: /:1%, /dev:1%,
/dev/shm:1%, /run:1%, /sys/fs/cgroup:1%, /boot: 1%, /run/user/1037:1% ͯ͢ͷfilesystemͷinodeΛࢹ Ұ൪গͳ͍ͷ͕ᮢΛԼճΔͱNG mackerel-plugin-inode `inode.percentage.*.used` ʹରͯ͠ࢹ͕Ͱ͖ͳ͍ͷͰ͍ʹ͍͘ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-memcached-uptime check ࢹ $ check-memcached-uptime \ -p 11211 -w 110
-c 110 memcached Uptime OK: up 240 days, 03:08:01 ▶ https://github.com/kazeburo/go-check-memcached-uptime memcachedͷಥવࢮ => ࠶ىಈΛݕ͢Δ check-memcached keyͷऔಘΛߦ͏
check-mysql-uptime check ࢹ $ check-mysql-uptime --host=127.0.0.1 \ -w 110 -c
110 MySQL Uptime OK: up 71 days, 17:47:39 pingͰλΠϛϯάʹΑΓҾ͔͔ͬΒͳ͍࠶ىಈΛݕ MySQL 4.0.xͳͲʹ༗ޮ `check-mysql uptime` ͰՄೳ ▶ https://github.com/kazeburo/go-check-mysql-uptime
check-mysql-msr $ check-mysql-msr --host localhost -w 5 -c 10 MySQL
Multi Source Replication OK: [O]foo-db=io:Yes,sql:Yes,behind:0 bar-db=io:Yes,sql:Yes,behind:0 main-db=io:Yes,sql:Yes,behind:0 baz- db=io:Yes,sql:Yes,behind:0 MySQL 5.7ͷMulti Source Replicationͷ༻ ͲΕ͔ҰͭͰReplicationͷఀࢭ͋Δ͍ɺԆ͕ᮢΛ্ճΔͱNG mackerel-plugin-msr ผʹ࡞ͬͨ inodeಉ༷ `mysql-msr.*.beind` ʹରͯ͠ࢹ͕Ͱ͖ͳ͍ͷͰ͍ʹ͍͘ check ࢹ ▶ https://github.com/kazeburo/go-check-mysql-msr
mackerel-plugin-linux-lite Free Memory, 100%্͕ݶͷCPU Usage CPUίΞ͋ͨΓͷϩʔυΞϕϨʔδ, ϓϩηε Ұ؏ͨ͠ϝτϦΫεͷஅɺࢹͷᮢͷઃఆ͕Ͱ͖Δ Agent Plugin
▶ https://github.com/kazeburo/custom-mackerel-plugins
mackerel-plugin-ntpq ࣌ؒͷͣΕ(ઈର)ɺϦϞʔτͱͷSyncͷ༗ແ check-ntpoffset ͋Δ Agent Plugin ▶ https://github.com/kazeburo/custom-mackerel-plugins
mackerel-plugin-postfix Postfixͷqueue mackerel-plugin-mailq ͋Δ Agent Plugin ▶ https://github.com/kazeburo/custom-mackerel-plugins
mackerel-plugin-mysql-lite ίωΫγϣϯ/max_connectionͷׂ߹ɺϨϓϦέʔγϣϯԆ Slave SQL Thread/IO Threadͷಈ࡞༗ແɺ εϨου KuradoͰϝτϦΫεΛݟ͍ͯΔͷͰɺࢹʹඞཁͳάϥϑʹߜΔ ▶ https://github.com/kazeburo/go-mackerel-plugin-mysql-lite
Agent Plugin
mackerel-plugin-php-version ݹ͍PHPΛݕ^^ Agent Plugin
Plugins & Language • ͍͔ͭ͘ͷίϚϯυͷݺͼग़͠ͷΈ = > Shell Script •
ίϚϯυͷݺͼग़͠ͱ݁Ռͷॲཧ => LL (Perl) • HTTPϦΫΤετͱJSONॲཧ => LL (Perl) • AWS APIɺMySQLͷଓ =>Golang ͜ΕΒΛ͍׳Ε͍ͯΔ͜ͱ͕ॏཁ
ίʔυΛॻ͍ͯΛղܾ͢Δͷָ͍͠!! mackerel͍͕͍ͷ͋Δπʔϧ