Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
最近の監視(仮)/Recent system monitoring with mackerel
Search
kazeburo
December 23, 2019
Technology
5.6k
3
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
最近の監視(仮)/Recent system monitoring with mackerel
mackerel day #2
kazeburo
December 23, 2019
Transcript
30 slides
࠷ۙͷࢹ(Ծ) Mackerel Day #2 kazeburo
Me • Masahiro Nagano • @kazeburo • Mackerel Ξϯόαμʔ •
Ұ൪͖ͳ mackerelϓϥάΠϯ mackerel-plugin-axslog (ࣗ࡞)ͩͳɻ ʹཱͭͱ͜ΖͰ diff-detector(ࣗ࡞)
https://ihara2525.tumblr.com/ “Ұߦͷϩάͷ͜͏ʹɺҰਓͷϢʔβ͕͍Δ”
• ΤϥʔόδΣοτͱ͍͏ߟ͑ • ೦ʹࢥ͏ϢʔβΛগͳ͍͖͍ͯͨ͘͠ • ࢹϢʔβͷظʹԠ͑ΔͨΊͷπʔϧ
• ʮࢹʯʹΑΓ • োΛະવʹ͙ • োʹૉૣ͘ؾͮ͘ • োͷݪҼಛఆͷώϯτ/MTTRͷॖ
mackerel-plugin-axslog ͱੜ
None
• mackerel-plugin-axslog • https://github.com/kazeburo/mackerel-plugin-axslog • mackerel-plugin-accesslog ͱಉ͘͡accesslogͷՄࢹԽ • status͝ͱͷͱׂ߹ɺresponse timeͷpercentileΛܭଌ
• ूܭ͢ΔlabelΛมߋՄೳɾJSONαϙʔτ • ߴ
0 2 4 6 8 accesslog(ltsv) axslog(ltsv) axslog(json) 1.17s 0.584s
7.976s 120ສߦ(20000req/sec)ͷϩάΛॲཧ͢Δ࣌ؒ
• ߴԽͷར • γεςϜෛՙΛؾʹͤͣɺʮΨϯΨϯʯϝτϦΫεΛ૿ͤΔ • ϑΟϧλػೳͷՃ
[plugin.metrics.axslog-web-googlebot] command = "/usr/local/bin/mackerel-plugin-axslog --key-prefix=googlebot \ --logfile /var/log/nginx/access.log --filter Googlebot"
[plugin.metrics.axslog-web-adsbot] command = "/usr/local/bin/mackerel-plugin-axslog --key-prefix=adsbot \ --logfile /var/log/nginx/access.log --filter AdsBot-Google"
• mackerel-plugin-postfix-log • https://github.com/kazeburo/mackerel-plugin-postfix-log • SMTP • AxlogͰlogrotateΛै͢Δίʔυॻ͍ͨͷͰ͙͢Ͱ͖ͨ
None
postfixlog.total_delay.average 0.240476 1555681849 postfixlog.total_delay.99_percentile 0.250000 1555681849 postfixlog.total_delay.95_percentile 0.240000 1555681849 postfixlog.total_delay.90_percentile
0.240000 1555681849 postfixlog.recving_delay.average 0.040000 1555681849 postfixlog.recving_delay.99_percentile 0.040000 1555681849 postfixlog.recving_delay.95_percentile 0.040000 1555681849 postfixlog.recving_delay.90_percentile 0.040000 1555681849 postfixlog.queuing_delay.average 0.000476 1555681849 postfixlog.queuing_delay.99_percentile 0.010000 1555681849 postfixlog.queuing_delay.95_percentile 0.000000 1555681849 postfixlog.queuing_delay.90_percentile 0.000000 1555681849 postfixlog.connection_delay.average 0.090000 1555681849 postfixlog.connection_delay.99_percentile 0.090000 1555681849 postfixlog.connection_delay.95_percentile 0.090000 1555681849 postfixlog.connection_delay.90_percentile 0.090000 1555681849 postfixlog.transmission_delay.average 0.090000 1555681849 postfixlog.transmission_delay.99_percentile 0.090000 1555681849 postfixlog.transmission_delay.95_percentile 0.090000 1555681849 postfixlog.transmission_delay.90_percentile 0.090000 1555681849 postfixlog.transfer_num.2xx_count 1.615385 1555681849 postfixlog.transfer_num.4xx_count 0.000000 1555681849 postfixlog.transfer_num.5xx_count 0.000000 1555681849 postfixlog.transfer_total.count 1.615385 1555681849 postfixlog.transfer_ratio.2xx_percentage 100.000000 1555681849 postfixlog.transfer_ratio.4xx_percentage 0.000000 1555681849 postfixlog.transfer_ratio.5xx_percentage 0.000000 1555681849 ड৴࣌ͷԆ Ωϡʔʹೖ͍ͬͯΔ࣌ؒ ૹ৴ઌͷଓʹ͔͔Δ࣌ؒ ૹ৴ͷԆ
• mackerel-plugin-log-incr-rate • https://github.com/kazeburo/mackerel-plugin-log-incr-rate • ͋Δϩάʹର͢Δผͷϩάͷߦͷ૿Ճׂ߹Λࢹ͢Δ • accesslogʹର͢Δapplication_log, errorlogͷ݅ͳͲ •
application_logͷॻ͖ࠐΈ͕രൃͯ͠ϩάܭଌʹӨڹͨ͠োʹର͢Δࢹ
None
• mackerel-plugin-gzip-response • Not OSS • ϨεϙϯεͷGzipѹॖ͕ߦΘΕ͍ͯΔ͔ࢹ • Ұ෦ΫϥΠΞϯτͷ࣮ͰGzip͕ແޮʹͳͬͨ··ɺେ͖ͳ৴ͱͳΔϚε λʔߋ৽͕ߦΘΕɺωοτϫʔΫʹෛՙ͕͔͔Δࣄଶ͕͋ͬͨ
• ϩάʹϨεϙϯεͷ Conent-Encoding ϔομΛه͠ूܭ
None
• Mackerel Plugin ࡞ύλʔϯ • ো/োʹܨ͕Δͷൃݟ • Metrics Plugin ͷ࡞ͱαʔόͷө
• MetricsΛΈͯͷɺᮢઃఆɾਵ࣌ݟ͠
DB ӡ༻ͱࢹ
• ϗετͷεςʔλεΛࢹ͢Δ • ϗετεςʔλε Standby ͷDB(slave)αʔόΛ production ʹೖ • ϝϯςφϯε࡞ۀͷͨΊϨϓϦέʔγϣϯ͕ࢭ·͍͕ͬͯͨɺ௨དྷͳ͍
App App consul consul DNS consul consul 1) look up
DNS 2) connect DB check mysql uptime before slave slave slave
App App consul consul DNS consul consul 1) look up
DNS 2) connect DB A) check mysql uptime After Mackerel batch batch batch B) check host status file B’ ) update host status file by cron slave slave slave
• Consul ͷαʔϏε͕༗ޮʹͳΔͨΊʹɺmackerelͷεςʔλεͷมߋ͕ ඞཁɻΞϥʔτʹؾ͕ͭ͘Մೳੑˢ • ϗετεςʔλεΛϩʔΧϧͷͲ͔͜ʹॻ͖ࠐΉػೳɺ mackerel-agent ʹ͋Δͱศར?
• Slave Ͱͷ Trigger ࣮ߦΛकΔ • ProductionͷҰ෦ͷslaveΛɺௐࠪɾूܭ༻ͱͯ͑͠ΔΑ͏ʹ͍ͯ͠Δ • ηϯγςΟϒͳใʹ͍ͭͯSlave্ͷTriggerͰআ
• MySQLͱϨϓϦέʔγϣϯͱTrigger • MySQLͷϨϓϦέʔγϣϯͷํࣜʹSQL͕సૹ͞Εͯ͘ΔStatementํࣜͱɺ σʔλ͕ૹΒΕͯ͘ΔRow Basedํࣜɺ·ͨͲͪΒ͔͕ࣗಈͰબ͞ΕΔMixed ํ͕ࣜ͋ΔɻMixedํࣜͰجຊStatementํ͕ࣜͩɺඇܾఆੑΫΤϦ͕Θ ΕΔͱɺRow BasedͱͳΔ •
ϨϓϦέʔγϣϯΛड͚Δslaveαʔό্ͰͷTriggerɺStatementํࣜͷ߹ͷ Έ࣮ߦ͞ΕΔ • SlaveͰtriggerΛ҆શʹར༻͢ΔͨΊʹҙਤ͠ͳ͍ Row basedͳϨϓϦέʔ γϣϯΛൃݟ͢Δඞཁ͕͋Δ
• check-mysqlbinlog-rbr-fallbacks • No OSS • mysqlbinlog ίϚϯυ͍ɺRow BasedʹͳͬͨςʔϒϧΛݕग़
Master Backup Anon Slave Trigger binlog binlog check-mysqlbinlog-rbr-fallbacks Detect the
SQL and fix data and applications if necessary App
• ઓ͍ͯ͠ΔݶΓɺࢹ૿͑Δ • Application Server: 175 Metrics • DB Server:
245 Metrics
• Ҏ্
More Decks by kazeburo
See All by kazeburo
さくらのクラウド開発ふりかえり2025
kazeburo
2
5.2k
国産クラウドを支える設計とチームの変遷 “技術・組織・ミッション”
kazeburo
7
18k
クラウド開発の舞台裏とSRE文化の醸成 / SRE NEXT 2025 Lunch Session
kazeburo
1
2.9k
さくらのクラウド 開発の挑戦とその舞台裏
kazeburo
1
1.6k
[SRE kaigi 2025] ガバメントクラウドに向けた開発と変化するSRE組織のあり方 / Development for Government Cloud and the Evolving Role of SRE Teams
kazeburo
4
4.7k
[さくらのTech Day] ガバメントクラウド開発と変化と成長する組織 / sakura techday, Develop govcloud and the team
kazeburo
0
10k
ガバメントクラウド開発と変化と成長する組織 / Organizational change and growth in developing a government cloud
kazeburo
4
4k
DNS水責め攻撃と監視 / DNS water torture attack Monitoring and SLO
kazeburo
5
4.7k
DBやめてみた / DNS water torture attack and countermeasures
kazeburo
13
14k
Other Decks in Technology
See All in Technology
AWS Client VPNの設定でハマったポイントとその過程での気づき
amarelo_n24
0
210
「AIチャットからもできないの?」と言われたら ~業務アプリをMCP対応するときの認証・認可の勘所
oidfj
0
350
Kansai WordPress Meetup@大阪『ブロックテーマを作る上で知っておくべきこと
torounit
0
140
パスワードのない鍵で、 データを手元に置く
ginjihayashi
0
120
DevOps Agentにあらゆるセキュリティインシデントのトリアージを任せた件
jirtosterone
1
170
Agent Harnessはどう作る?事例から学ぶAI Agentの設計思想
sergicalsix
0
330
Kiro アップデート 2026 年 11 月号 (11 月 8 日時点) | 目指せ Kiro マスター アップデート編
inariku
PRO
2
130
Kiro を使うコツ (頼み方、覚えさせ方、任せ方) | 目指せ Kiro マスター 09
inariku
PRO
3
170
81天的 AI Agent 軟體開發團隊治理實務分享
regionbbs
0
180
さようならSSH鍵、こんにちはゼロトラスト ― AIフレンドリーで快適な自宅鯖SSH
atalie2m
12
5.3k
Shared Signals Framework 仕様解説
oidfj
0
450
IP重複は"ランディングVPC"で受け止める ― PrivateLink と Transit Gatewayによるマルチテナント接続基盤の設計
hatahata021
2
190
Featured
See All Featured
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
659
62k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
28
3.8k
The State of eCommerce SEO: How to Win in Today's Products SERPs - #SEOweek
aleyda
2
12k
AI: The stuff that nobody shows you
jnunemaker
PRO
12
1.3k
Leveraging Curiosity to Care for An Aging Population
cassininazir
1
570
The Cult of Friendly URLs
andyhume
79
7.1k
Fantastic passwords and where to find them - at NoRuKo
philnash
53
4k
How to train your dragon (web standard)
notwaldorf
96
6.9k
How to Get Subject Matter Experts Bought In and Actively Contributing to SEO & PR Initiatives.
livdayseo
1
240
Reality Check: Gamification 10 Years Later
codingconduct
0
2.4k
Navigating Algorithm Shifts & AI Overviews - #SMXNext
aleyda
1
1.7k
Agile Leadership in an Agile Organization
kimpetersen
PRO
0
310