Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
20+ custom mackerel plugins in 20 minutes
Search
kazeburo
May 12, 2016
Technology
2
3.4k
20+ custom mackerel plugins in 20 minutes
mackerel meetup #7 Tokyo
mercariでmackerelを導入したときに作ったpluginの紹介です #mackerelio
kazeburo
May 12, 2016
Tweet
Share
More Decks by kazeburo
See All by kazeburo
DNS水責め攻撃と監視 / DNS water torture attack Monitoring and SLO
kazeburo
4
3.5k
DBやめてみた / DNS water torture attack and countermeasures
kazeburo
13
11k
IaaSにおけるPlatform Engineeringとこれから / Platform engineering in IaaS
kazeburo
2
1k
高信頼IaaSを実現するDevOps / DevOps for Highly Reliable IaaS
kazeburo
1
430
権威DNSサービスへのDDoSと ハイパフォーマンスなベンチマーカ / DNS Pseudo random subdomain attack and High performance Benchmarker
kazeburo
3
4.4k
DNS権威サーバのクラウドサービス向けに行われた攻撃および対策 / DNS Pseudo-Random Subdomain Attack and mitigations
kazeburo
7
11k
sacloudns
kazeburo
2
270
「orchestratorとGTID運用を支える監視」の勉強 / Monitoring orchestrator and GTID operation
kazeburo
2
1.2k
最近の監視(仮)/Recent system monitoring with mackerel
kazeburo
3
4.4k
Other Decks in Technology
See All in Technology
How to Build a Strong Engineering Culture
alperhankendi
0
120
Combineを中心とした処理をSwift Concurrencyへ (これまでも調べた調査と向き合い)
fumiyasac0921
1
170
庁舎DX推進事例集
senryakuka
0
1.5k
皆がすなるカオスエンジアリングといふものを、ネットワークオペレーションでもしてみむとてするなり
tjmtrhs
0
110
「わたしたちのコード」を安定させるためにフレームワークとの距離を保つ / phperkaigi2024
blue_goheimochi
5
790
GitHub composite actions
hkusu
1
120
Exadata Database Service on Dedicated Infrastructure(ExaDB-D) UI スクリーン・キャプチャ集
oracle4engineer
PRO
0
1.1k
layerx-0-to-1-product-development-in-compound-startups
shnjtk
1
510
技育祭2024春 LT Finatextホールディングス
kevinrobot34
1
110
書を捨てよ、現場へ出よう
nwiizo
11
8.7k
二刀流でWinActorを活用してみた話
tamai_63
0
120
『QAという人』が必要ではなく、『QAという技術』が必要
sadonosake
2
250
Featured
See All Featured
GraphQLとの向き合い方2022年版
quramy
28
12k
The Straight Up "How To Draw Better" Workshop
denniskardys
227
130k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
355
22k
Producing Creativity
orderedlist
PRO
335
39k
Atom: Resistance is Futile
akmur
258
25k
Build The Right Thing And Hit Your Dates
maggiecrowley
23
1.9k
Why You Should Never Use an ORM
jnunemaker
PRO
50
8.5k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
185
15k
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3k
A better future with KSS
kneath
230
16k
Being A Developer After 40
akosma
56
580k
Designing for Performance
lara
601
67k
Transcript
20+ Custom mackerel Plugins in 20 minutes mackerel Meetup#7 Masahiro
Nagano @kazeburo
Me • Masahiro Nagano • @kazeburo • Mercari, Inc. Principal
Engineer Site Reliability Engineering (SRE) Team • BASE, Inc. Technical Advisor
We’re Hiring!! ΑΖ͓͘͠ئ͍͠·͢ʂ
Mercari Your Friendly Mobile MarketPlace JP US
Infrastructure ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥυ Amazon Web Service US West (Oregon)
Region JP US
Monitoring (past) ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥυ Amazon Web Service US West
(Oregon) Region JP US Z ZABBIX ҟͳΔzabbixͷόʔδϣϯ.. WebUIΆͪΆͪർΕ.. ࢹՃ·Ͱͷεςοϓͷଟ͞.. ࢹ࿙ΕɺҟͳΔࢹ༰.. Z ZABBIX
Monitoring (now) ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥυ Amazon Web Service US West
(Oregon) Region JP US JP/USͷҰׅࢹ Service & RoleʹΑΔཧ Nagios & Sensu ϕʔεͷPlugin
Migration Items, Graphs Screens, Triggers Auto Discovery, UserParameters Official mackerel
plugins Official check plugins Nagios plugins Custom mackerel Plugins Custom check Pluginss + Simple Monitoring Rules Z ZABBIX ࣜʹΑΔ ࢹ จࣈྻɺɺ ࠩɺܭࢉ݁ՌͳͲͷ σʔλܗࣜ ΧελϜσʔλͷ औಘ
Custom Plugins my-ec2-tag[go], jmx-get[go], diff-detector[go], delay-checker, interval-checker, periodic-checker, check-mysql-uptime[go], check-memcached-uptime[go],
check-conntrack-free, check-crt-expiration, check-dns-rr, check-hydra-pos, check-inode, check-iptables, check-myip, check-solr-replication, check-solr-update, check-uptime, check-mysql-msr[go], mackerel-plugin-accelmail-counter, mackerel-plugin-gaurun-usage, mackerel-plugin-linux-lite, mackerel-plugin-msr[go] mackerel-plugin-ntpq, mackerel-plugin-search, mackerel-plugin-postfix mackerel-plugin-php-and-accesslog, mackerel-plugin-php-version and MORE! and utilitiy commands
my-ec2-tags $ my-ec2-tag --tag Role production EC2ΠϯελϯεͷλάΛऔಘ awsίϚϯυ͍ͷͰGoͰ࡞ ޙड़ͷdiff-detectorͱΈ߹Θͤͯ͏ Utility
delay-checker $ delay-checker -c 3 -- check-tcp -p 80 ࢦఆͨ͠ճΤϥʔ͕࿈ଓ͢ΔͱNG
max_check_attempts Φϓγϣϯ͕Ͱ͖ͨ check ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins
interval-checker $ interval-checker --interval 300 -- \ my-ec2-tag --tag Role
ࢦఆִͨؒ͠ͰࢹίϚϯυ࣮ߦ ࢦఆͨ࣌ؒ͠·ͰલճͷίϚϯυ࣮ߦ݁ՌΛฦ͢ check ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins
periodic-checker $ periodic-checker --range 00:00-06:00,06:30-24:00 \ -- check-tcp -p 80
ࢦఆͨ͠ൣғͷ͚࣌ؒͩࢹ ͦΕҎ֎OKΛฦ͢ Daily ϝϯςφϯεʗॏ͍batch͕͋Δͱ͖ʹ check ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins
diff-detector $ diff-detector -- date NG: detect difference: ```@@ -1
+1 @@ -Tue May 10 08:11:42 UTC 2016 +Tue May 10 08:11:43 UTC 2016``` ίϚϯυ࣮ߦ݁Ռͷ͕ࠩ͋ΔͱΤϥʔ ec2 tagɺ/etc/passwdϑΝΠϧͷࢹ ▶ https://github.com/kazeburo/diff-detector check ࢹ
check-conntrack-free $ check-conntrack-free -w 10000 -c 2000 OK: free:596376 max:655360
count:58984 `nf_conntrack_max - nf_conntrack_count` ͷ ݁Ռͷࢹɻiptables͕༗ޮͳ࣌ʹ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-crt-expiration $ check-crt-expiration -w 7 -c 5 -f /path/cert.pem OK:
Expiration date: 'Jun 1 10:38:57 2016 GMT'. 20 days left ূ໌ॻͷ༗ޮظݶͷ֬ೝ check-cert-file͕ެࣜʹՃ͞Εͨ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-dns-rr $ check-dns-rr -w 3 -c 2 \ --host prod.example.service.consul
OK: 3 dns-rr hosts found DNS-RRͷ݁Ռʹؚ·ΕΔIPͷࢹ Consul/DNSͰࢄ͍ͯ͠Δ࣌ʹ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-inode $ check-inode -w 10 -c 5 OK: /:1%, /dev:1%,
/dev/shm:1%, /run:1%, /sys/fs/cgroup:1%, /boot: 1%, /run/user/1037:1% ͯ͢ͷfilesystemͷinodeΛࢹ Ұ൪গͳ͍ͷ͕ᮢΛԼճΔͱNG mackerel-plugin-inode `inode.percentage.*.used` ʹରͯ͠ࢹ͕Ͱ͖ͳ͍ͷͰ͍ʹ͍͘ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-memcached-uptime check ࢹ $ check-memcached-uptime \ -p 11211 -w 110
-c 110 memcached Uptime OK: up 240 days, 03:08:01 ▶ https://github.com/kazeburo/go-check-memcached-uptime memcachedͷಥવࢮ => ࠶ىಈΛݕ͢Δ check-memcached keyͷऔಘΛߦ͏
check-mysql-uptime check ࢹ $ check-mysql-uptime --host=127.0.0.1 \ -w 110 -c
110 MySQL Uptime OK: up 71 days, 17:47:39 pingͰλΠϛϯάʹΑΓҾ͔͔ͬΒͳ͍࠶ىಈΛݕ MySQL 4.0.xͳͲʹ༗ޮ `check-mysql uptime` ͰՄೳ ▶ https://github.com/kazeburo/go-check-mysql-uptime
check-mysql-msr $ check-mysql-msr --host localhost -w 5 -c 10 MySQL
Multi Source Replication OK: [O]foo-db=io:Yes,sql:Yes,behind:0 bar-db=io:Yes,sql:Yes,behind:0 main-db=io:Yes,sql:Yes,behind:0 baz- db=io:Yes,sql:Yes,behind:0 MySQL 5.7ͷMulti Source Replicationͷ༻ ͲΕ͔ҰͭͰReplicationͷఀࢭ͋Δ͍ɺԆ͕ᮢΛ্ճΔͱNG mackerel-plugin-msr ผʹ࡞ͬͨ inodeಉ༷ `mysql-msr.*.beind` ʹରͯ͠ࢹ͕Ͱ͖ͳ͍ͷͰ͍ʹ͍͘ check ࢹ ▶ https://github.com/kazeburo/go-check-mysql-msr
mackerel-plugin-linux-lite Free Memory, 100%্͕ݶͷCPU Usage CPUίΞ͋ͨΓͷϩʔυΞϕϨʔδ, ϓϩηε Ұ؏ͨ͠ϝτϦΫεͷஅɺࢹͷᮢͷઃఆ͕Ͱ͖Δ Agent Plugin
▶ https://github.com/kazeburo/custom-mackerel-plugins
mackerel-plugin-ntpq ࣌ؒͷͣΕ(ઈର)ɺϦϞʔτͱͷSyncͷ༗ແ check-ntpoffset ͋Δ Agent Plugin ▶ https://github.com/kazeburo/custom-mackerel-plugins
mackerel-plugin-postfix Postfixͷqueue mackerel-plugin-mailq ͋Δ Agent Plugin ▶ https://github.com/kazeburo/custom-mackerel-plugins
mackerel-plugin-mysql-lite ίωΫγϣϯ/max_connectionͷׂ߹ɺϨϓϦέʔγϣϯԆ Slave SQL Thread/IO Threadͷಈ࡞༗ແɺ εϨου KuradoͰϝτϦΫεΛݟ͍ͯΔͷͰɺࢹʹඞཁͳάϥϑʹߜΔ ▶ https://github.com/kazeburo/go-mackerel-plugin-mysql-lite
Agent Plugin
mackerel-plugin-php-version ݹ͍PHPΛݕ^^ Agent Plugin
Plugins & Language • ͍͔ͭ͘ͷίϚϯυͷݺͼग़͠ͷΈ = > Shell Script •
ίϚϯυͷݺͼग़͠ͱ݁Ռͷॲཧ => LL (Perl) • HTTPϦΫΤετͱJSONॲཧ => LL (Perl) • AWS APIɺMySQLͷଓ =>Golang ͜ΕΒΛ͍׳Ε͍ͯΔ͜ͱ͕ॏཁ
ίʔυΛॻ͍ͯΛղܾ͢Δͷָ͍͠!! mackerel͍͕͍ͷ͋Δπʔϧ