Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
20+ custom mackerel plugins in 20 minutes
Search
kazeburo
May 12, 2016
Technology
2
3.6k
20+ custom mackerel plugins in 20 minutes
mackerel meetup #7 Tokyo
mercariでmackerelを導入したときに作ったpluginの紹介です #mackerelio
kazeburo
May 12, 2016
Tweet
Share
More Decks by kazeburo
See All by kazeburo
[さくらのTech Day] ガバメントクラウド開発と変化と成長する組織 / sakura techday, Develop govcloud and the team
kazeburo
0
1.3k
ガバメントクラウド開発と変化と成長する組織 / Organizational change and growth in developing a government cloud
kazeburo
4
2k
DNS水責め攻撃と監視 / DNS water torture attack Monitoring and SLO
kazeburo
5
4.1k
DBやめてみた / DNS water torture attack and countermeasures
kazeburo
13
13k
IaaSにおけるPlatform Engineeringとこれから / Platform engineering in IaaS
kazeburo
2
1.3k
高信頼IaaSを実現するDevOps / DevOps for Highly Reliable IaaS
kazeburo
1
570
権威DNSサービスへのDDoSと ハイパフォーマンスなベンチマーカ / DNS Pseudo random subdomain attack and High performance Benchmarker
kazeburo
3
5.2k
DNS権威サーバのクラウドサービス向けに行われた攻撃および対策 / DNS Pseudo-Random Subdomain Attack and mitigations
kazeburo
7
13k
sacloudns
kazeburo
2
320
Other Decks in Technology
See All in Technology
プロダクト開発を加速させるためのQA文化の築き方 / How to build QA culture to accelerate product development
mii3king
1
290
小学3年生夏休みの自由研究「夏休みに Copilot で遊んでみた」
taichinakamura
0
180
サイボウズフロントエンドエキスパートチームについて / FrontendExpert Team
cybozuinsideout
PRO
5
38k
Opcodeを読んでいたら何故かphp-srcを読んでいた話
murashotaro
0
320
Microsoft Azure全冠になってみた ~アレを使い倒した者が試験を制す!?~/Obtained all Microsoft Azure certifications Those who use "that" to the full will win the exam! ?
yuj1osm
2
120
1等無人航空機操縦士一発試験 合格までの道のり ドローンミートアップ@大阪 2024/12/18
excdinc
0
180
LINE Developersプロダクト(LIFF/LINE Login)におけるフロントエンド開発
lycorptech_jp
PRO
0
150
マイクロサービスにおける容易なトランザクション管理に向けて
scalar
0
190
DevFest 2024 Incheon / Songdo - Compose UI 조합 심화
wisemuji
0
160
組み込みアプリパフォーマンス格闘記 検索画面編
wataruhigasi
1
140
Storage Browser for Amazon S3
miu_crescent
1
290
AWS re:Invent 2024 ふりかえり勉強会
yhana
0
460
Featured
See All Featured
It's Worth the Effort
3n
183
28k
GraphQLとの向き合い方2022年版
quramy
44
13k
Done Done
chrislema
182
16k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Reflections from 52 weeks, 52 projects
jeffersonlam
347
20k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
26
1.5k
Making the Leap to Tech Lead
cromwellryan
133
9k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
44
9.3k
Gamification - CAS2011
davidbonilla
80
5.1k
VelocityConf: Rendering Performance Case Studies
addyosmani
326
24k
Building Better People: How to give real-time feedback that sticks.
wjessup
366
19k
Building a Scalable Design System with Sketch
lauravandoore
460
33k
Transcript
20+ Custom mackerel Plugins in 20 minutes mackerel Meetup#7 Masahiro
Nagano @kazeburo
Me • Masahiro Nagano • @kazeburo • Mercari, Inc. Principal
Engineer Site Reliability Engineering (SRE) Team • BASE, Inc. Technical Advisor
We’re Hiring!! ΑΖ͓͘͠ئ͍͠·͢ʂ
Mercari Your Friendly Mobile MarketPlace JP US
Infrastructure ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥυ Amazon Web Service US West (Oregon)
Region JP US
Monitoring (past) ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥυ Amazon Web Service US West
(Oregon) Region JP US Z ZABBIX ҟͳΔzabbixͷόʔδϣϯ.. WebUIΆͪΆͪർΕ.. ࢹՃ·Ͱͷεςοϓͷଟ͞.. ࢹ࿙ΕɺҟͳΔࢹ༰.. Z ZABBIX
Monitoring (now) ͘͞ΒΠϯλʔωοτ ੴङDC ઐ༻αʔόͱΫϥυ Amazon Web Service US West
(Oregon) Region JP US JP/USͷҰׅࢹ Service & RoleʹΑΔཧ Nagios & Sensu ϕʔεͷPlugin
Migration Items, Graphs Screens, Triggers Auto Discovery, UserParameters Official mackerel
plugins Official check plugins Nagios plugins Custom mackerel Plugins Custom check Pluginss + Simple Monitoring Rules Z ZABBIX ࣜʹΑΔ ࢹ จࣈྻɺɺ ࠩɺܭࢉ݁ՌͳͲͷ σʔλܗࣜ ΧελϜσʔλͷ औಘ
Custom Plugins my-ec2-tag[go], jmx-get[go], diff-detector[go], delay-checker, interval-checker, periodic-checker, check-mysql-uptime[go], check-memcached-uptime[go],
check-conntrack-free, check-crt-expiration, check-dns-rr, check-hydra-pos, check-inode, check-iptables, check-myip, check-solr-replication, check-solr-update, check-uptime, check-mysql-msr[go], mackerel-plugin-accelmail-counter, mackerel-plugin-gaurun-usage, mackerel-plugin-linux-lite, mackerel-plugin-msr[go] mackerel-plugin-ntpq, mackerel-plugin-search, mackerel-plugin-postfix mackerel-plugin-php-and-accesslog, mackerel-plugin-php-version and MORE! and utilitiy commands
my-ec2-tags $ my-ec2-tag --tag Role production EC2ΠϯελϯεͷλάΛऔಘ awsίϚϯυ͍ͷͰGoͰ࡞ ޙड़ͷdiff-detectorͱΈ߹Θͤͯ͏ Utility
delay-checker $ delay-checker -c 3 -- check-tcp -p 80 ࢦఆͨ͠ճΤϥʔ͕࿈ଓ͢ΔͱNG
max_check_attempts Φϓγϣϯ͕Ͱ͖ͨ check ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins
interval-checker $ interval-checker --interval 300 -- \ my-ec2-tag --tag Role
ࢦఆִͨؒ͠ͰࢹίϚϯυ࣮ߦ ࢦఆͨ࣌ؒ͠·ͰલճͷίϚϯυ࣮ߦ݁ՌΛฦ͢ check ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins
periodic-checker $ periodic-checker --range 00:00-06:00,06:30-24:00 \ -- check-tcp -p 80
ࢦఆͨ͠ൣғͷ͚࣌ؒͩࢹ ͦΕҎ֎OKΛฦ͢ Daily ϝϯςφϯεʗॏ͍batch͕͋Δͱ͖ʹ check ࢹ Utility ▶ https://github.com/kazeburo/custom-mackerel-plugins
diff-detector $ diff-detector -- date NG: detect difference: ```@@ -1
+1 @@ -Tue May 10 08:11:42 UTC 2016 +Tue May 10 08:11:43 UTC 2016``` ίϚϯυ࣮ߦ݁Ռͷ͕ࠩ͋ΔͱΤϥʔ ec2 tagɺ/etc/passwdϑΝΠϧͷࢹ ▶ https://github.com/kazeburo/diff-detector check ࢹ
check-conntrack-free $ check-conntrack-free -w 10000 -c 2000 OK: free:596376 max:655360
count:58984 `nf_conntrack_max - nf_conntrack_count` ͷ ݁Ռͷࢹɻiptables͕༗ޮͳ࣌ʹ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-crt-expiration $ check-crt-expiration -w 7 -c 5 -f /path/cert.pem OK:
Expiration date: 'Jun 1 10:38:57 2016 GMT'. 20 days left ূ໌ॻͷ༗ޮظݶͷ֬ೝ check-cert-file͕ެࣜʹՃ͞Εͨ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-dns-rr $ check-dns-rr -w 3 -c 2 \ --host prod.example.service.consul
OK: 3 dns-rr hosts found DNS-RRͷ݁Ռʹؚ·ΕΔIPͷࢹ Consul/DNSͰࢄ͍ͯ͠Δ࣌ʹ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-inode $ check-inode -w 10 -c 5 OK: /:1%, /dev:1%,
/dev/shm:1%, /run:1%, /sys/fs/cgroup:1%, /boot: 1%, /run/user/1037:1% ͯ͢ͷfilesystemͷinodeΛࢹ Ұ൪গͳ͍ͷ͕ᮢΛԼճΔͱNG mackerel-plugin-inode `inode.percentage.*.used` ʹରͯ͠ࢹ͕Ͱ͖ͳ͍ͷͰ͍ʹ͍͘ check ࢹ ▶ https://github.com/kazeburo/custom-mackerel-plugins
check-memcached-uptime check ࢹ $ check-memcached-uptime \ -p 11211 -w 110
-c 110 memcached Uptime OK: up 240 days, 03:08:01 ▶ https://github.com/kazeburo/go-check-memcached-uptime memcachedͷಥવࢮ => ࠶ىಈΛݕ͢Δ check-memcached keyͷऔಘΛߦ͏
check-mysql-uptime check ࢹ $ check-mysql-uptime --host=127.0.0.1 \ -w 110 -c
110 MySQL Uptime OK: up 71 days, 17:47:39 pingͰλΠϛϯάʹΑΓҾ͔͔ͬΒͳ͍࠶ىಈΛݕ MySQL 4.0.xͳͲʹ༗ޮ `check-mysql uptime` ͰՄೳ ▶ https://github.com/kazeburo/go-check-mysql-uptime
check-mysql-msr $ check-mysql-msr --host localhost -w 5 -c 10 MySQL
Multi Source Replication OK: [O]foo-db=io:Yes,sql:Yes,behind:0 bar-db=io:Yes,sql:Yes,behind:0 main-db=io:Yes,sql:Yes,behind:0 baz- db=io:Yes,sql:Yes,behind:0 MySQL 5.7ͷMulti Source Replicationͷ༻ ͲΕ͔ҰͭͰReplicationͷఀࢭ͋Δ͍ɺԆ͕ᮢΛ্ճΔͱNG mackerel-plugin-msr ผʹ࡞ͬͨ inodeಉ༷ `mysql-msr.*.beind` ʹରͯ͠ࢹ͕Ͱ͖ͳ͍ͷͰ͍ʹ͍͘ check ࢹ ▶ https://github.com/kazeburo/go-check-mysql-msr
mackerel-plugin-linux-lite Free Memory, 100%্͕ݶͷCPU Usage CPUίΞ͋ͨΓͷϩʔυΞϕϨʔδ, ϓϩηε Ұ؏ͨ͠ϝτϦΫεͷஅɺࢹͷᮢͷઃఆ͕Ͱ͖Δ Agent Plugin
▶ https://github.com/kazeburo/custom-mackerel-plugins
mackerel-plugin-ntpq ࣌ؒͷͣΕ(ઈର)ɺϦϞʔτͱͷSyncͷ༗ແ check-ntpoffset ͋Δ Agent Plugin ▶ https://github.com/kazeburo/custom-mackerel-plugins
mackerel-plugin-postfix Postfixͷqueue mackerel-plugin-mailq ͋Δ Agent Plugin ▶ https://github.com/kazeburo/custom-mackerel-plugins
mackerel-plugin-mysql-lite ίωΫγϣϯ/max_connectionͷׂ߹ɺϨϓϦέʔγϣϯԆ Slave SQL Thread/IO Threadͷಈ࡞༗ແɺ εϨου KuradoͰϝτϦΫεΛݟ͍ͯΔͷͰɺࢹʹඞཁͳάϥϑʹߜΔ ▶ https://github.com/kazeburo/go-mackerel-plugin-mysql-lite
Agent Plugin
mackerel-plugin-php-version ݹ͍PHPΛݕ^^ Agent Plugin
Plugins & Language • ͍͔ͭ͘ͷίϚϯυͷݺͼग़͠ͷΈ = > Shell Script •
ίϚϯυͷݺͼग़͠ͱ݁Ռͷॲཧ => LL (Perl) • HTTPϦΫΤετͱJSONॲཧ => LL (Perl) • AWS APIɺMySQLͷଓ =>Golang ͜ΕΒΛ͍׳Ε͍ͯΔ͜ͱ͕ॏཁ
ίʔυΛॻ͍ͯΛղܾ͢Δͷָ͍͠!! mackerel͍͕͍ͷ͋Δπʔϧ