Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
20+ custom mackerel plugins in 20 minutes
Search
kazeburo
May 12, 2016
Technology
2
3.6k
20+ custom mackerel plugins in 20 minutes
mackerel meetup #7 Tokyo
mercariでmackerelを導入したときに作ったpluginの紹介です #mackerelio
kazeburo
May 12, 2016
Tweet
Share
More Decks by kazeburo
See All by kazeburo
[SRE kaigi 2025] ガバメントクラウドに向けた開発と変化するSRE組織のあり方 / Development for Government Cloud and the Evolving Role of SRE Teams
kazeburo
4
2.2k
[さくらのTech Day] ガバメントクラウド開発と変化と成長する組織 / sakura techday, Develop govcloud and the team
kazeburo
0
5.5k
ガバメントクラウド開発と変化と成長する組織 / Organizational change and growth in developing a government cloud
kazeburo
4
2.5k
DNS水責め攻撃と監視 / DNS water torture attack Monitoring and SLO
kazeburo
5
4.1k
DBやめてみた / DNS water torture attack and countermeasures
kazeburo
13
13k
IaaSにおけるPlatform Engineeringとこれから / Platform engineering in IaaS
kazeburo
2
1.3k
高信頼IaaSを実現するDevOps / DevOps for Highly Reliable IaaS
kazeburo
1
610
権威DNSサービスへのDDoSと ハイパフォーマンスなベンチマーカ / DNS Pseudo random subdomain attack and High performance Benchmarker
kazeburo
3
5.3k
DNS権威サーバのクラウドサービス向けに行われた攻撃および対策 / DNS Pseudo-Random Subdomain Attack and mitigations
kazeburo
7
13k
Other Decks in Technology
See All in Technology
Building Products in the LLM Era
ymatsuwitter
10
5.7k
オブザーバビリティの観点でみるAWS / AWS from observability perspective
ymotongpoo
8
1.5k
データマネジメントのトレードオフに立ち向かう
ikkimiyazaki
6
1.1k
インフラをつくるとはどういうことなのか、 あるいはPlatform Engineeringについて
nwiizo
5
2.6k
速くて安いWebサイトを作る
nishiharatsubasa
13
15k
君も受託系GISエンジニアにならないか
sudataka
2
450
クラウドサービス事業者におけるOSS
tagomoris
3
910
明日からできる!技術的負債の返済を加速するための実践ガイド~『ホットペッパービューティー』の事例をもとに~
recruitengineers
PRO
3
440
リーダブルテストコード 〜メンテナンスしやすい テストコードを作成する方法を考える〜 #DevSumi #DevSumiB / Readable test code
nihonbuson
11
7.4k
抽象化をするということ - 具体と抽象の往復を身につける / Abstraction and concretization
soudai
23
11k
2024.02.19 W&B AIエージェントLT会 / AIエージェントが業務を代行するための計画と実行 / Algomatic 宮脇
smiyawaki0820
14
3.8k
全文検索+セマンティックランカー+LLMの自然文検索サ−ビスで得られた知見
segavvy
2
120
Featured
See All Featured
Bootstrapping a Software Product
garrettdimon
PRO
306
110k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
129
19k
Automating Front-end Workflow
addyosmani
1368
200k
Building a Modern Day E-commerce SEO Strategy
aleyda
38
7.1k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
133
33k
Rails Girls Zürich Keynote
gr2m
94
13k
A Tale of Four Properties
chriscoyier
158
23k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
21
2.5k
Making the Leap to Tech Lead
cromwellryan
133
9.1k
How to Ace a Technical Interview
jacobian
276
23k
Fontdeck: Realign not Redesign
paulrobertlloyd
83
5.4k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.1k
Transcript
20+ Custom mackerel Plugins in 20 minutes mackerel Meetup#7 Masahiro
Nagano @kazeburo
Me • Masahiro Nagano • @kazeburo • Mercari, Inc. Principal
Engineer Site Reliability Engineering (SRE) Team • BASE, Inc. Technical Advisor
We’re Hiring!! ΑΖ͓͘͠ئ͍͠·͢ʂ
Mercari Your Friendly Mobile MarketPlace JP US
Infrastructure ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥυ Amazon Web Service US West (Oregon)
Region JP US
Monitoring (past) ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥυ Amazon Web Service US West
(Oregon) Region JP US Z ZABBIX ҟͳΔzabbixͷόʔδϣϯ.. WebUIΆͪΆͪർΕ.. ࢹՃ·Ͱͷεςοϓͷଟ͞.. ࢹ࿙ΕɺҟͳΔࢹ༰.. Z ZABBIX
Monitoring (now) ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥυ Amazon Web Service US West
(Oregon) Region JP US JP/USͷҰׅࢹ Service & RoleʹΑΔཧ Nagios & Sensu ϕʔεͷPlugin
Migration Items, Graphs Screens, Triggers Auto Discovery, UserParameters Official mackerel
plugins Official check plugins Nagios plugins Custom mackerel Plugins Custom check Pluginss + Simple Monitoring Rules Z ZABBIX ࣜʹΑΔ ࢹ จࣈྻɺɺ ࠩɺܭࢉ݁ՌͳͲͷ σʔλܗࣜ ΧελϜσʔλͷ औಘ
Custom Plugins my-ec2-tag[go], jmx-get[go], diff-detector[go], delay-checker, interval-checker, periodic-checker, check-mysql-uptime[go], check-memcached-uptime[go],
check-conntrack-free, check-crt-expiration, check-dns-rr, check-hydra-pos, check-inode, check-iptables, check-myip, check-solr-replication, check-solr-update, check-uptime, check-mysql-msr[go], mackerel-plugin-accelmail-counter, mackerel-plugin-gaurun-usage, mackerel-plugin-linux-lite, mackerel-plugin-msr[go] mackerel-plugin-ntpq, mackerel-plugin-search, mackerel-plugin-postfix mackerel-plugin-php-and-accesslog, mackerel-plugin-php-version and MORE! and utilitiy commands
my-ec2-tags $ my-ec2-tag --tag Role production EC2ΠϯελϯεͷλάΛऔಘ awsίϚϯυ͍ͷͰGoͰ࡞ ޙड़ͷdiff-detectorͱΈ߹Θͤͯ͏ Utility
delay-checker $ delay-checker -c 3 -- check-tcp -p 80 ࢦఆͨ͠ճΤϥʔ͕࿈ଓ͢ΔͱNG
max_check_attempts Φϓγϣϯ͕Ͱ͖ͨ check ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins
interval-checker $ interval-checker --interval 300 -- \ my-ec2-tag --tag Role
ࢦఆִͨؒ͠ͰࢹίϚϯυ࣮ߦ ࢦఆͨ࣌ؒ͠·ͰલճͷίϚϯυ࣮ߦ݁ՌΛฦ͢ check ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins
periodic-checker $ periodic-checker --range 00:00-06:00,06:30-24:00 \ -- check-tcp -p 80
ࢦఆͨ͠ൣғͷ͚࣌ؒͩࢹ ͦΕҎ֎OKΛฦ͢ Daily ϝϯςφϯεʗॏ͍batch͕͋Δͱ͖ʹ check ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins
diff-detector $ diff-detector -- date NG: detect difference: ```@@ -1
+1 @@ -Tue May 10 08:11:42 UTC 2016 +Tue May 10 08:11:43 UTC 2016``` ίϚϯυ࣮ߦ݁Ռͷ͕ࠩ͋ΔͱΤϥʔ ec2 tagɺ/etc/passwdϑΝΠϧͷࢹ ▶ https://github.com/kazeburo/diff-detector check ࢹ
check-conntrack-free $ check-conntrack-free -w 10000 -c 2000 OK: free:596376 max:655360
count:58984 `nf_conntrack_max - nf_conntrack_count` ͷ ݁Ռͷࢹɻiptables͕༗ޮͳ࣌ʹ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-crt-expiration $ check-crt-expiration -w 7 -c 5 -f /path/cert.pem OK:
Expiration date: 'Jun 1 10:38:57 2016 GMT'. 20 days left ূ໌ॻͷ༗ޮظݶͷ֬ೝ check-cert-file͕ެࣜʹՃ͞Εͨ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-dns-rr $ check-dns-rr -w 3 -c 2 \ --host prod.example.service.consul
OK: 3 dns-rr hosts found DNS-RRͷ݁Ռʹؚ·ΕΔIPͷࢹ Consul/DNSͰࢄ͍ͯ͠Δ࣌ʹ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-inode $ check-inode -w 10 -c 5 OK: /:1%, /dev:1%,
/dev/shm:1%, /run:1%, /sys/fs/cgroup:1%, /boot: 1%, /run/user/1037:1% ͯ͢ͷfilesystemͷinodeΛࢹ Ұ൪গͳ͍ͷ͕ᮢΛԼճΔͱNG mackerel-plugin-inode `inode.percentage.*.used` ʹରͯ͠ࢹ͕Ͱ͖ͳ͍ͷͰ͍ʹ͍͘ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-memcached-uptime check ࢹ $ check-memcached-uptime \ -p 11211 -w 110
-c 110 memcached Uptime OK: up 240 days, 03:08:01 ▶ https://github.com/kazeburo/go-check-memcached-uptime memcachedͷಥવࢮ => ࠶ىಈΛݕ͢Δ check-memcached keyͷऔಘΛߦ͏
check-mysql-uptime check ࢹ $ check-mysql-uptime --host=127.0.0.1 \ -w 110 -c
110 MySQL Uptime OK: up 71 days, 17:47:39 pingͰλΠϛϯάʹΑΓҾ͔͔ͬΒͳ͍࠶ىಈΛݕ MySQL 4.0.xͳͲʹ༗ޮ `check-mysql uptime` ͰՄೳ ▶ https://github.com/kazeburo/go-check-mysql-uptime
check-mysql-msr $ check-mysql-msr --host localhost -w 5 -c 10 MySQL
Multi Source Replication OK: [O]foo-db=io:Yes,sql:Yes,behind:0 bar-db=io:Yes,sql:Yes,behind:0 main-db=io:Yes,sql:Yes,behind:0 baz- db=io:Yes,sql:Yes,behind:0 MySQL 5.7ͷMulti Source Replicationͷ༻ ͲΕ͔ҰͭͰReplicationͷఀࢭ͋Δ͍ɺԆ͕ᮢΛ্ճΔͱNG mackerel-plugin-msr ผʹ࡞ͬͨ inodeಉ༷ `mysql-msr.*.beind` ʹରͯ͠ࢹ͕Ͱ͖ͳ͍ͷͰ͍ʹ͍͘ check ࢹ ▶ https://github.com/kazeburo/go-check-mysql-msr
mackerel-plugin-linux-lite Free Memory, 100%্͕ݶͷCPU Usage CPUίΞ͋ͨΓͷϩʔυΞϕϨʔδ, ϓϩηε Ұ؏ͨ͠ϝτϦΫεͷஅɺࢹͷᮢͷઃఆ͕Ͱ͖Δ Agent Plugin
▶ https://github.com/kazeburo/custom-mackerel-plugins
mackerel-plugin-ntpq ࣌ؒͷͣΕ(ઈର)ɺϦϞʔτͱͷSyncͷ༗ແ check-ntpoffset ͋Δ Agent Plugin ▶ https://github.com/kazeburo/custom-mackerel-plugins
mackerel-plugin-postfix Postfixͷqueue mackerel-plugin-mailq ͋Δ Agent Plugin ▶ https://github.com/kazeburo/custom-mackerel-plugins
mackerel-plugin-mysql-lite ίωΫγϣϯ/max_connectionͷׂ߹ɺϨϓϦέʔγϣϯԆ Slave SQL Thread/IO Threadͷಈ࡞༗ແɺ εϨου KuradoͰϝτϦΫεΛݟ͍ͯΔͷͰɺࢹʹඞཁͳάϥϑʹߜΔ ▶ https://github.com/kazeburo/go-mackerel-plugin-mysql-lite
Agent Plugin
mackerel-plugin-php-version ݹ͍PHPΛݕ^^ Agent Plugin
Plugins & Language • ͍͔ͭ͘ͷίϚϯυͷݺͼग़͠ͷΈ = > Shell Script •
ίϚϯυͷݺͼग़͠ͱ݁Ռͷॲཧ => LL (Perl) • HTTPϦΫΤετͱJSONॲཧ => LL (Perl) • AWS APIɺMySQLͷଓ =>Golang ͜ΕΒΛ͍׳Ε͍ͯΔ͜ͱ͕ॏཁ
ίʔυΛॻ͍ͯΛղܾ͢Δͷָ͍͠!! mackerel͍͕͍ͷ͋Δπʔϧ