Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ヘビーなGraphite運用
Search
Yuuki Tsubouchi (yuuk1)
June 12, 2014
Technology
5
290k
ヘビーなGraphite運用
Monitoring Casual Talks #6
Yuuki Tsubouchi (yuuk1)
June 12, 2014
Tweet
Share
More Decks by Yuuki Tsubouchi (yuuk1)
See All by Yuuki Tsubouchi (yuuk1)
Cloudless Computingの論文紹介
yuukit
1
300
#SRE論文紹介 Detection is Better Than Cure: A Cloud Incidents Perspective V. Ganatra et. al., ESEC/FSE’23
yuukit
3
950
エンジニアのためのSRE論文への招待 / Introduction to SRE Papers for Engineers
yuukit
1
10k
博士課程での研究まとめ 2023年1月版 / Summary of my research in the PhD course
yuukit
1
200
AI時代に向けたクラウドにおける信頼性エンジニアリングの未来構想 / DICOMO2022 6A-1
yuukit
7
2.7k
AIOps研究録―SREのための システム障害の自動原因診断 / SRE NEXT 2022
yuukit
10
12k
Interactive AIOps
yuukit
0
1.9k
Meltria: マイクロサービスにおける 異常検知・原因分析のための データセットの動的生成システム / Meltria in IOTS2021
yuukit
1
1.6k
AIOpsの研究動向と AIOps向けデータセットの動的生成の研究 / Introducing AIOps and A Dynamic Datasets Generating System
yuukit
1
690
Other Decks in Technology
See All in Technology
RAGのサービスをリリースして1年3ヶ月が経ちました
segavvy
4
960
コンテナ・K8s研修 - 後半 Kubernetes 基礎&ハンズオン【MIXI 24新卒技術研修】
mixi_engineers
PRO
1
120
Android研修【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
100
推薦システムを本番導入する上で一番優先すべきだったこと~NewsPicks記事推薦機能の改善事例を元に~
morinota
0
130
テスト・設計研修【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
170
さらに高品質・高速化を目指すAI時代のテスト設計支援と、めざす先 / AI Test Lab vol.1
shift_evolve
0
190
可視化プラットフォームGrafanaの基本と活用方法の全て
hamadakoji
0
230
AI研修【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
130
頼られるのが大好きな 皆さんへ - 支援相手との期待の合わせ方、突き放し方 -/For_people_who_like_to_be_relied_on
naitosatoshi
1
290
エンジニアリングマネージャーはどう学んでいくのか #devsumi / How Do Engineering Managers Continue to Learn and Grow?
expajp
4
1.3k
Azure AI ことはじめ
tsubakimoto_s
0
130
「我々はどこに向かっているのか」を問い続けるための仕組みづくり / Establishing a System for Continuous Inquiry about where we are
daitasu
0
170
Featured
See All Featured
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
353
29k
The World Runs on Bad Software
bkeepers
PRO
63
11k
Gamification - CAS2011
davidbonilla
78
4.9k
Embracing the Ebb and Flow
colly
81
4.3k
What's new in Ruby 2.0
geeforr
338
31k
Music & Morning Musume
bryan
43
5.9k
Debugging Ruby Performance
tmm1
71
11k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
662
120k
Infographics Made Easy
chrislema
238
18k
Design by the Numbers
sachag
277
18k
Docker and Python
trallard
37
2.9k
The MySQL Ecosystem @ GitHub 2015
samlambert
248
12k
Transcript
ϔϏʔͳ Graphite ӡ༻ id:y_uuki Monitoring Casual Talks #6
@y_uuk1 id:y_uuki ӡ༻
Graphite
Graphite Data Flow Graphite Webapp Carbon Whisper Collector TCP HTTP
metrics Graph JSON Store Get Graphite
carbon-cache • 1ϓϩηε/2εϨου • ίωΫγϣϯϋϯυϦϯάεϨου • ॻ͖ࠐΈεϨου • carbon-relay: carbon-cache
ͷલஈ • (carbon-aggregator)
whisper • RRDtool ͱ΄΅ಉ͡σʔλߏ • 1ϑΝΠϧ1ϝτϦοΫ (xxxhost ͷ loadavg) •
ಠࣗܗࣜͷϑΝΠϧΛ࡞Δ python ϥΠϒϥϦ
graphite-web • Web ΞϓϦέʔγϣϯαʔό • ศརάϥϑURLه๏ (ը૾ or json) •
Python Django • ApacheඞਢͰͳ͍ (uwsgi, gunicorn) • ͍ͩ͞μογϡϘʔυ
app ॳظߏ XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 carbon-cache graphite-web whisper
app LVS carbon-relay LVS carbon-relay carbon-cache graphite-web carbon-cache graphite-web SFQMJDBUJPO
XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 whisper whisper XSJUFSFRVFTU QJDLFQSPUPDPM Խ DBSCPOSFMBZ
όοΫΞοϓ app LVS carbon-relay LVS carbon-relay carbon- cache graphite- web
master whisper carbon- cache graphite- web master whisper carbon- cache graphite- web backup whisper
carbon-cache ͕CPUͰ٧·Δ
multiple carbon-cache app LVS LVS carbon-relay carbon-relay carbon- cache graphite-web
tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-backup carbon- cache carbon- cache carbon- cache carbon- relay DPOTJTUFOU IBTIJOH
carbon-relay ͕CPUͰ٧·Δ
! ! tsdb-relay-lb app LVS LVS carbon-relay carbon-relay carbon- relay
LVS LVS carbon- relay carbon- relay carbon- relay carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web backup carbon- cache carbon- cache carbon- cache carbon- relay
10k scale Write IOPS ioDrive CPUϘτϧωοΫ
ύϥϝʔλνϡʔχϯά
carbon-cache MAX_CACHE_SIZE = inf MAX_UPDATES_PER_SECOND = inf MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = inf
MAX_CREATES_PER_MINUTE = inf CACHE_WRITE_STRATEGY = naive
carbon-cache MAX_CACHE_SIZE = MAX_UPDATES_PER_SECOND = MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = MAX_CREATES_PER_MINUTE = CACHE_WRITE_STRATEGY
= $16͞ΔͷͰ ͳʹΒͤͳ͍
σʔλͷҰ؏ੑ
DRBD
app %3#% XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 carbon-cache graphite-web primary whisper
carbon-cache graphite-web secondary whisper
ఆظతͳόοΫΞοϓ • Ұ؏ੑͪΌΜͱ͢ΔͳΒߋ৽ΛͱΊΔΈ͕ ผ్ඞཁ (stop slave IO_THREAD;ͱ͔ͳ͍) • ߋ৽ͱΊͯΔؒʹ rsync
/ nc • ϝοηʔδΩϡʔతͳͷʹஷΊ͜Ή • ΦϯϥΠϯͰόοΫΞοϓͨ͠ͱͯ͠Ұఆظ ؒޙwhisper͕aggregate͢Δ
ͦͷଞ • ҙ֎ͱηοτΞοϓ໘ • தಁ͚ͯΔʢPythonʣ • whisper ͷσʔλߏI/Oޮѱ͍ • MySQL
ͱ͔͑Δ͚ͲϢʔβใͱ͔͍ΕΔ ͚ͩ
ȦȀȀ✣Ǐ
https://mackerel.io “৽͍͠ΞϓϦέʔγϣϯύϑΥʔϚϯε Ϛωδϝϯτ”
None
ϞχλϦϯά͕͖ͳΤϯδχΞΛ ืू͓ͯ͠Γ·͢