Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
最近の監視(仮)/Recent system monitoring with mackerel
kazeburo
December 23, 2019
Technology
3
3.6k
最近の監視(仮)/Recent system monitoring with mackerel
mackerel day #2
kazeburo
December 23, 2019
Tweet
Share
More Decks by kazeburo
See All by kazeburo
sacloudns
kazeburo
2
200
「orchestratorとGTID運用を支える監視」の勉強 / Monitoring orchestrator and GTID operation
kazeburo
2
950
Mercari Item Search: Behind The Scenes (20min)
kazeburo
3
2.4k
成長し続けるインフラストラクチャとメルカリの挑戦/mercari infrastructure and software
kazeburo
23
4.5k
Microservices on Multi-Cloud
kazeburo
21
5.3k
logrotate殺プロセス事件 YAPC::Okinawa 2018 前夜祭 LT/Mystery of logrotate's death
kazeburo
9
10k
Perl in Mercari YAPC::Okinawa 2018 ONNASON
kazeburo
8
4.8k
インフラチームからSREへ / SRE in Mercari Developers Summit 2018
kazeburo
26
8.8k
メルカリのシステム・サービス監視について/Monitoring Mercari service and servers
kazeburo
5
6.1k
Other Decks in Technology
See All in Technology
CAMのサービス開発の歴史と共通基盤を使った 開発スタイルへの変遷について
ishikawa_pro
0
110
OSS ことはじめ
hsbt
3
600
Dagu | オンプレ向けワークフローエンジン(WebUI 同梱)
yohamta
1
210
KubeCon Recap -Platform migration at Scale-
inductor
0
130
Declarative Clients in Spring
olgamaciaszek
0
120
LINE API Update 20220525
line_developers_tw
PRO
0
100
アルプの 認証/認可分離戦略と手法
ma2k8
PRO
2
370
我々はなぜテストをするのか?
kawaguti
PRO
0
640
Microsoft Build 2022 - Azure のデータ & 分析サービス 最新アップデート / Microsoft Build 2022 Updates on Azure Data and Analytics Services
nakazax
1
220
LINEのData Platform室が実践する大規模分散環境のCapacity Planning
line_developers
PRO
0
930
CTOのためのQAのつくりかた #scrumniigata / SigSQA How to create QA for CTOs and VPoEs
caori_t
0
360
Kubernetesの上に作る、統一されたマイクロサービス運用体験
tkuchiki
1
1.4k
Featured
See All Featured
jQuery: Nuts, Bolts and Bling
dougneiner
56
6.4k
The MySQL Ecosystem @ GitHub 2015
samlambert
238
11k
What's in a price? How to price your products and services
michaelherold
229
9.3k
Building Better People: How to give real-time feedback that sticks.
wjessup
343
17k
For a Future-Friendly Web
brad_frost
164
7.4k
The Illustrated Children's Guide to Kubernetes
chrisshort
14
35k
Why Our Code Smells
bkeepers
PRO
324
54k
How GitHub (no longer) Works
holman
296
140k
Building Flexible Design Systems
yeseniaperezcruz
310
33k
Three Pipe Problems
jasonvnalue
89
8.6k
Optimizing for Happiness
mojombo
365
63k
Teambox: Starting and Learning
jrom
121
7.6k
Transcript
࠷ۙͷࢹ(Ծ) Mackerel Day #2 kazeburo
Me • Masahiro Nagano • @kazeburo • Mackerel Ξϯόαμʔ •
Ұ൪͖ͳ mackerelϓϥάΠϯ mackerel-plugin-axslog (ࣗ࡞)ͩͳɻ ʹཱͭͱ͜ΖͰ diff-detector(ࣗ࡞)
https://ihara2525.tumblr.com/ “Ұߦͷϩάͷ͜͏ʹɺҰਓͷϢʔβ͕͍Δ”
• ΤϥʔόδΣοτͱ͍͏ߟ͑ • ೦ʹࢥ͏ϢʔβΛগͳ͍͖͍ͯͨ͘͠ • ࢹϢʔβͷظʹԠ͑ΔͨΊͷπʔϧ
• ʮࢹʯʹΑΓ • োΛະવʹ͙ • োʹૉૣ͘ؾͮ͘ • োͷݪҼಛఆͷώϯτ/MTTRͷॖ
mackerel-plugin-axslog ͱੜ
None
• mackerel-plugin-axslog • https://github.com/kazeburo/mackerel-plugin-axslog • mackerel-plugin-accesslog ͱಉ͘͡accesslogͷՄࢹԽ • status͝ͱͷͱׂ߹ɺresponse timeͷpercentileΛܭଌ
• ूܭ͢ΔlabelΛมߋՄೳɾJSONαϙʔτ • ߴ
0 2 4 6 8 accesslog(ltsv) axslog(ltsv) axslog(json) 1.17s 0.584s
7.976s 120ສߦ(20000req/sec)ͷϩάΛॲཧ͢Δ࣌ؒ
• ߴԽͷར • γεςϜෛՙΛؾʹͤͣɺʮΨϯΨϯʯϝτϦΫεΛ૿ͤΔ • ϑΟϧλػೳͷՃ
[plugin.metrics.axslog-web-googlebot] command = "/usr/local/bin/mackerel-plugin-axslog --key-prefix=googlebot \ --logfile /var/log/nginx/access.log --filter Googlebot"
[plugin.metrics.axslog-web-adsbot] command = "/usr/local/bin/mackerel-plugin-axslog --key-prefix=adsbot \ --logfile /var/log/nginx/access.log --filter AdsBot-Google"
• mackerel-plugin-postfix-log • https://github.com/kazeburo/mackerel-plugin-postfix-log • SMTP • AxlogͰlogrotateΛै͢Δίʔυॻ͍ͨͷͰ͙͢Ͱ͖ͨ
None
postfixlog.total_delay.average 0.240476 1555681849 postfixlog.total_delay.99_percentile 0.250000 1555681849 postfixlog.total_delay.95_percentile 0.240000 1555681849 postfixlog.total_delay.90_percentile
0.240000 1555681849 postfixlog.recving_delay.average 0.040000 1555681849 postfixlog.recving_delay.99_percentile 0.040000 1555681849 postfixlog.recving_delay.95_percentile 0.040000 1555681849 postfixlog.recving_delay.90_percentile 0.040000 1555681849 postfixlog.queuing_delay.average 0.000476 1555681849 postfixlog.queuing_delay.99_percentile 0.010000 1555681849 postfixlog.queuing_delay.95_percentile 0.000000 1555681849 postfixlog.queuing_delay.90_percentile 0.000000 1555681849 postfixlog.connection_delay.average 0.090000 1555681849 postfixlog.connection_delay.99_percentile 0.090000 1555681849 postfixlog.connection_delay.95_percentile 0.090000 1555681849 postfixlog.connection_delay.90_percentile 0.090000 1555681849 postfixlog.transmission_delay.average 0.090000 1555681849 postfixlog.transmission_delay.99_percentile 0.090000 1555681849 postfixlog.transmission_delay.95_percentile 0.090000 1555681849 postfixlog.transmission_delay.90_percentile 0.090000 1555681849 postfixlog.transfer_num.2xx_count 1.615385 1555681849 postfixlog.transfer_num.4xx_count 0.000000 1555681849 postfixlog.transfer_num.5xx_count 0.000000 1555681849 postfixlog.transfer_total.count 1.615385 1555681849 postfixlog.transfer_ratio.2xx_percentage 100.000000 1555681849 postfixlog.transfer_ratio.4xx_percentage 0.000000 1555681849 postfixlog.transfer_ratio.5xx_percentage 0.000000 1555681849 ड৴࣌ͷԆ Ωϡʔʹೖ͍ͬͯΔ࣌ؒ ૹ৴ઌͷଓʹ͔͔Δ࣌ؒ ૹ৴ͷԆ
• mackerel-plugin-log-incr-rate • https://github.com/kazeburo/mackerel-plugin-log-incr-rate • ͋Δϩάʹର͢Δผͷϩάͷߦͷ૿Ճׂ߹Λࢹ͢Δ • accesslogʹର͢Δapplication_log, errorlogͷ݅ͳͲ •
application_logͷॻ͖ࠐΈ͕രൃͯ͠ϩάܭଌʹӨڹͨ͠োʹର͢Δࢹ
None
• mackerel-plugin-gzip-response • Not OSS • ϨεϙϯεͷGzipѹॖ͕ߦΘΕ͍ͯΔ͔ࢹ • Ұ෦ΫϥΠΞϯτͷ࣮ͰGzip͕ແޮʹͳͬͨ··ɺେ͖ͳ৴ͱͳΔϚε λʔߋ৽͕ߦΘΕɺωοτϫʔΫʹෛՙ͕͔͔Δࣄଶ͕͋ͬͨ
• ϩάʹϨεϙϯεͷ Conent-Encoding ϔομΛه͠ूܭ
None
• Mackerel Plugin ࡞ύλʔϯ • ো/োʹܨ͕Δͷൃݟ • Metrics Plugin ͷ࡞ͱαʔόͷө
• MetricsΛΈͯͷɺᮢઃఆɾਵ࣌ݟ͠
DB ӡ༻ͱࢹ
• ϗετͷεςʔλεΛࢹ͢Δ • ϗετεςʔλε Standby ͷDB(slave)αʔόΛ production ʹೖ • ϝϯςφϯε࡞ۀͷͨΊϨϓϦέʔγϣϯ͕ࢭ·͍͕ͬͯͨɺ௨དྷͳ͍
App App consul consul DNS consul consul 1) look up
DNS 2) connect DB check mysql uptime before slave slave slave
App App consul consul DNS consul consul 1) look up
DNS 2) connect DB A) check mysql uptime After Mackerel batch batch batch B) check host status file B’ ) update host status file by cron slave slave slave
• Consul ͷαʔϏε͕༗ޮʹͳΔͨΊʹɺmackerelͷεςʔλεͷมߋ͕ ඞཁɻΞϥʔτʹؾ͕ͭ͘Մೳੑˢ • ϗετεςʔλεΛϩʔΧϧͷͲ͔͜ʹॻ͖ࠐΉػೳɺ mackerel-agent ʹ͋Δͱศར?
• Slave Ͱͷ Trigger ࣮ߦΛकΔ • ProductionͷҰ෦ͷslaveΛɺௐࠪɾूܭ༻ͱͯ͑͠ΔΑ͏ʹ͍ͯ͠Δ • ηϯγςΟϒͳใʹ͍ͭͯSlave্ͷTriggerͰআ
• MySQLͱϨϓϦέʔγϣϯͱTrigger • MySQLͷϨϓϦέʔγϣϯͷํࣜʹSQL͕సૹ͞Εͯ͘ΔStatementํࣜͱɺ σʔλ͕ૹΒΕͯ͘ΔRow Basedํࣜɺ·ͨͲͪΒ͔͕ࣗಈͰબ͞ΕΔMixed ํ͕ࣜ͋ΔɻMixedํࣜͰجຊStatementํ͕ࣜͩɺඇܾఆੑΫΤϦ͕Θ ΕΔͱɺRow BasedͱͳΔ •
ϨϓϦέʔγϣϯΛड͚Δslaveαʔό্ͰͷTriggerɺStatementํࣜͷ߹ͷ Έ࣮ߦ͞ΕΔ • SlaveͰtriggerΛ҆શʹར༻͢ΔͨΊʹҙਤ͠ͳ͍ Row basedͳϨϓϦέʔ γϣϯΛൃݟ͢Δඞཁ͕͋Δ
• check-mysqlbinlog-rbr-fallbacks • No OSS • mysqlbinlog ίϚϯυ͍ɺRow BasedʹͳͬͨςʔϒϧΛݕग़
Master Backup Anon Slave Trigger binlog binlog check-mysqlbinlog-rbr-fallbacks Detect the
SQL and fix data and applications if necessary App
• ઓ͍ͯ͠ΔݶΓɺࢹ૿͑Δ • Application Server: 175 Metrics • DB Server:
245 Metrics
• Ҏ্