Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
最近の監視(仮)/Recent system monitoring with mackerel
kazeburo
December 23, 2019
Technology
3
3.8k
最近の監視(仮)/Recent system monitoring with mackerel
mackerel day #2
kazeburo
December 23, 2019
Tweet
Share
More Decks by kazeburo
See All by kazeburo
DNS権威サーバのクラウドサービス向けに行われた攻撃および対策 / DNS Pseudo-Random Subdomain Attack and mitigations
kazeburo
4
850
sacloudns
kazeburo
2
220
「orchestratorとGTID運用を支える監視」の勉強 / Monitoring orchestrator and GTID operation
kazeburo
2
1k
Mercari Item Search: Behind The Scenes (20min)
kazeburo
3
2.6k
成長し続けるインフラストラクチャとメルカリの挑戦/mercari infrastructure and software
kazeburo
23
4.6k
Microservices on Multi-Cloud
kazeburo
21
5.4k
logrotate殺プロセス事件 YAPC::Okinawa 2018 前夜祭 LT/Mystery of logrotate's death
kazeburo
9
11k
Perl in Mercari YAPC::Okinawa 2018 ONNASON
kazeburo
8
5k
インフラチームからSREへ / SRE in Mercari Developers Summit 2018
kazeburo
26
9k
Other Decks in Technology
See All in Technology
データエンジニアを助けてくれるFivetranとSnowflakeの仕様&機能のご紹介
sagara
0
420
20230121_BuriKaigi
oyakata2438
0
120
MoT/コネヒト/Kanmu が語るプロダクト開発xデータ分析 - 分析から機械学習システムの開発まで一人で複数ロールを担う大変さ
masatakashiwagi
2
550
Optimizing your Swift code
kateinoigakukun
0
1.3k
CEXやDEXに依存しないブロックチェーン取引について考える
sbtechnight
0
310
JAWS-UG朝会_41_NakagawaAkihiro.pptx.pdf
anakagawa
2
620
plotlyで動くグラフを作る
kosshi
0
690
re:Inventで発表があったIoT事例の紹介と考察
kizawa2020
0
110
エンタープライズ領域でのブロックチェーン・インターオペラビリティの発展 / Enterprise Blockchain Interoperability
gakumura
0
100
Google Cloud Updates 2022/12/01-12/15
no24oka
1
150
NGINXENG JP#2 - 3-NGINX Plus・プロダクトのアップデート
hiropo20
0
140
経営統合をきっかけに会社をエンジニアリングした話 / btconjp-2023
carta_engineering
0
100
Featured
See All Featured
Documentation Writing (for coders)
carmenintech
51
2.9k
Unsuck your backbone
ammeep
659
56k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
22
1.7k
What's in a price? How to price your products and services
michaelherold
233
9.7k
How GitHub Uses GitHub to Build GitHub
holman
465
280k
We Have a Design System, Now What?
morganepeng
37
5.9k
WebSockets: Embracing the real-time Web
robhawkes
58
6k
Code Review Best Practice
trishagee
50
11k
Building Applications with DynamoDB
mza
85
4.9k
Building Adaptive Systems
keathley
27
1.3k
Building Flexible Design Systems
yeseniaperezcruz
314
35k
Web Components: a chance to create the future
zenorocha
304
40k
Transcript
࠷ۙͷࢹ(Ծ) Mackerel Day #2 kazeburo
Me • Masahiro Nagano • @kazeburo • Mackerel Ξϯόαμʔ •
Ұ൪͖ͳ mackerelϓϥάΠϯ mackerel-plugin-axslog (ࣗ࡞)ͩͳɻ ʹཱͭͱ͜ΖͰ diff-detector(ࣗ࡞)
https://ihara2525.tumblr.com/ “Ұߦͷϩάͷ͜͏ʹɺҰਓͷϢʔβ͕͍Δ”
• ΤϥʔόδΣοτͱ͍͏ߟ͑ • ೦ʹࢥ͏ϢʔβΛগͳ͍͖͍ͯͨ͘͠ • ࢹϢʔβͷظʹԠ͑ΔͨΊͷπʔϧ
• ʮࢹʯʹΑΓ • োΛະવʹ͙ • োʹૉૣ͘ؾͮ͘ • োͷݪҼಛఆͷώϯτ/MTTRͷॖ
mackerel-plugin-axslog ͱੜ
None
• mackerel-plugin-axslog • https://github.com/kazeburo/mackerel-plugin-axslog • mackerel-plugin-accesslog ͱಉ͘͡accesslogͷՄࢹԽ • status͝ͱͷͱׂ߹ɺresponse timeͷpercentileΛܭଌ
• ूܭ͢ΔlabelΛมߋՄೳɾJSONαϙʔτ • ߴ
0 2 4 6 8 accesslog(ltsv) axslog(ltsv) axslog(json) 1.17s 0.584s
7.976s 120ສߦ(20000req/sec)ͷϩάΛॲཧ͢Δ࣌ؒ
• ߴԽͷར • γεςϜෛՙΛؾʹͤͣɺʮΨϯΨϯʯϝτϦΫεΛ૿ͤΔ • ϑΟϧλػೳͷՃ
[plugin.metrics.axslog-web-googlebot] command = "/usr/local/bin/mackerel-plugin-axslog --key-prefix=googlebot \ --logfile /var/log/nginx/access.log --filter Googlebot"
[plugin.metrics.axslog-web-adsbot] command = "/usr/local/bin/mackerel-plugin-axslog --key-prefix=adsbot \ --logfile /var/log/nginx/access.log --filter AdsBot-Google"
• mackerel-plugin-postfix-log • https://github.com/kazeburo/mackerel-plugin-postfix-log • SMTP • AxlogͰlogrotateΛै͢Δίʔυॻ͍ͨͷͰ͙͢Ͱ͖ͨ
None
postfixlog.total_delay.average 0.240476 1555681849 postfixlog.total_delay.99_percentile 0.250000 1555681849 postfixlog.total_delay.95_percentile 0.240000 1555681849 postfixlog.total_delay.90_percentile
0.240000 1555681849 postfixlog.recving_delay.average 0.040000 1555681849 postfixlog.recving_delay.99_percentile 0.040000 1555681849 postfixlog.recving_delay.95_percentile 0.040000 1555681849 postfixlog.recving_delay.90_percentile 0.040000 1555681849 postfixlog.queuing_delay.average 0.000476 1555681849 postfixlog.queuing_delay.99_percentile 0.010000 1555681849 postfixlog.queuing_delay.95_percentile 0.000000 1555681849 postfixlog.queuing_delay.90_percentile 0.000000 1555681849 postfixlog.connection_delay.average 0.090000 1555681849 postfixlog.connection_delay.99_percentile 0.090000 1555681849 postfixlog.connection_delay.95_percentile 0.090000 1555681849 postfixlog.connection_delay.90_percentile 0.090000 1555681849 postfixlog.transmission_delay.average 0.090000 1555681849 postfixlog.transmission_delay.99_percentile 0.090000 1555681849 postfixlog.transmission_delay.95_percentile 0.090000 1555681849 postfixlog.transmission_delay.90_percentile 0.090000 1555681849 postfixlog.transfer_num.2xx_count 1.615385 1555681849 postfixlog.transfer_num.4xx_count 0.000000 1555681849 postfixlog.transfer_num.5xx_count 0.000000 1555681849 postfixlog.transfer_total.count 1.615385 1555681849 postfixlog.transfer_ratio.2xx_percentage 100.000000 1555681849 postfixlog.transfer_ratio.4xx_percentage 0.000000 1555681849 postfixlog.transfer_ratio.5xx_percentage 0.000000 1555681849 ड৴࣌ͷԆ Ωϡʔʹೖ͍ͬͯΔ࣌ؒ ૹ৴ઌͷଓʹ͔͔Δ࣌ؒ ૹ৴ͷԆ
• mackerel-plugin-log-incr-rate • https://github.com/kazeburo/mackerel-plugin-log-incr-rate • ͋Δϩάʹର͢Δผͷϩάͷߦͷ૿Ճׂ߹Λࢹ͢Δ • accesslogʹର͢Δapplication_log, errorlogͷ݅ͳͲ •
application_logͷॻ͖ࠐΈ͕രൃͯ͠ϩάܭଌʹӨڹͨ͠োʹର͢Δࢹ
None
• mackerel-plugin-gzip-response • Not OSS • ϨεϙϯεͷGzipѹॖ͕ߦΘΕ͍ͯΔ͔ࢹ • Ұ෦ΫϥΠΞϯτͷ࣮ͰGzip͕ແޮʹͳͬͨ··ɺେ͖ͳ৴ͱͳΔϚε λʔߋ৽͕ߦΘΕɺωοτϫʔΫʹෛՙ͕͔͔Δࣄଶ͕͋ͬͨ
• ϩάʹϨεϙϯεͷ Conent-Encoding ϔομΛه͠ूܭ
None
• Mackerel Plugin ࡞ύλʔϯ • ো/োʹܨ͕Δͷൃݟ • Metrics Plugin ͷ࡞ͱαʔόͷө
• MetricsΛΈͯͷɺᮢઃఆɾਵ࣌ݟ͠
DB ӡ༻ͱࢹ
• ϗετͷεςʔλεΛࢹ͢Δ • ϗετεςʔλε Standby ͷDB(slave)αʔόΛ production ʹೖ • ϝϯςφϯε࡞ۀͷͨΊϨϓϦέʔγϣϯ͕ࢭ·͍͕ͬͯͨɺ௨དྷͳ͍
App App consul consul DNS consul consul 1) look up
DNS 2) connect DB check mysql uptime before slave slave slave
App App consul consul DNS consul consul 1) look up
DNS 2) connect DB A) check mysql uptime After Mackerel batch batch batch B) check host status file B’ ) update host status file by cron slave slave slave
• Consul ͷαʔϏε͕༗ޮʹͳΔͨΊʹɺmackerelͷεςʔλεͷมߋ͕ ඞཁɻΞϥʔτʹؾ͕ͭ͘Մೳੑˢ • ϗετεςʔλεΛϩʔΧϧͷͲ͔͜ʹॻ͖ࠐΉػೳɺ mackerel-agent ʹ͋Δͱศར?
• Slave Ͱͷ Trigger ࣮ߦΛकΔ • ProductionͷҰ෦ͷslaveΛɺௐࠪɾूܭ༻ͱͯ͑͠ΔΑ͏ʹ͍ͯ͠Δ • ηϯγςΟϒͳใʹ͍ͭͯSlave্ͷTriggerͰআ
• MySQLͱϨϓϦέʔγϣϯͱTrigger • MySQLͷϨϓϦέʔγϣϯͷํࣜʹSQL͕సૹ͞Εͯ͘ΔStatementํࣜͱɺ σʔλ͕ૹΒΕͯ͘ΔRow Basedํࣜɺ·ͨͲͪΒ͔͕ࣗಈͰબ͞ΕΔMixed ํ͕ࣜ͋ΔɻMixedํࣜͰجຊStatementํ͕ࣜͩɺඇܾఆੑΫΤϦ͕Θ ΕΔͱɺRow BasedͱͳΔ •
ϨϓϦέʔγϣϯΛड͚Δslaveαʔό্ͰͷTriggerɺStatementํࣜͷ߹ͷ Έ࣮ߦ͞ΕΔ • SlaveͰtriggerΛ҆શʹར༻͢ΔͨΊʹҙਤ͠ͳ͍ Row basedͳϨϓϦέʔ γϣϯΛൃݟ͢Δඞཁ͕͋Δ
• check-mysqlbinlog-rbr-fallbacks • No OSS • mysqlbinlog ίϚϯυ͍ɺRow BasedʹͳͬͨςʔϒϧΛݕग़
Master Backup Anon Slave Trigger binlog binlog check-mysqlbinlog-rbr-fallbacks Detect the
SQL and fix data and applications if necessary App
• ઓ͍ͯ͠ΔݶΓɺࢹ૿͑Δ • Application Server: 175 Metrics • DB Server:
245 Metrics
• Ҏ্