Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ヘビーなGraphite運用
Search
Yuuki Tsubouchi (yuuk1)
June 12, 2014
Technology
5
290k
ヘビーなGraphite運用
Monitoring Casual Talks #6
Yuuki Tsubouchi (yuuk1)
June 12, 2014
Tweet
Share
More Decks by Yuuki Tsubouchi (yuuk1)
See All by Yuuki Tsubouchi (yuuk1)
eBPFを用いたAIネットワーク監視システム論文の実装 / eBPF Japan Meetup #4
yuukit
3
830
クラウドのテレメトリーシステム研究動向2025年
yuukit
3
950
博士論文公聴会: Scaling Telemetry Workloads in Cloud Applications: Techniques for Instrumentation, Storage, and Mining / PhD Defence
yuukit
1
170
博士学位論文予備審査 / Scaling Telemetry Workloads in Cloud Applications: Techniques for Instrumentation, Storage, and Mining
yuukit
1
1.9k
MetricSifter:クラウドアプリケーションにおける故障箇所特定の効率化のための多変量時系列データの特徴量削減 / FIT 2024
yuukit
2
260
工学としてのSRE再訪 / Revisiting SRE as Engineering
yuukit
19
14k
Cloudless Computingの論文紹介
yuukit
2
560
#SRE論文紹介 Detection is Better Than Cure: A Cloud Incidents Perspective V. Ganatra et. al., ESEC/FSE’23
yuukit
3
2.1k
エンジニアのためのSRE論文への招待 / Introduction to SRE Papers for Engineers
yuukit
2
11k
Other Decks in Technology
See All in Technology
MySQL5.6から8.4へ 戦いの記録
kyoshidaxx
1
200
AWS テクニカルサポートとエンドカスタマーの中間地点から見えるより良いサポートの活用方法
kazzpapa3
2
540
CI/CD/IaC 久々に0から環境を作ったらこうなりました
kaz29
1
170
HiMoR: Monocular Deformable Gaussian Reconstruction with Hierarchical Motion Representation
spatial_ai_network
0
110
Fabric + Databricks 2025.6 の最新情報ピックアップ
ryomaru0825
1
130
より良いプロダクトの開発を目指して - 情報を中心としたプロダクト開発 #phpcon #phpcon2025
bengo4com
1
3.1k
SalesforceArchitectGroupOsaka#20_CNX'25_Report
atomica7sei
0
160
Postman AI エージェントビルダー最新情報
nagix
0
110
PHPでWebブラウザのレンダリングエンジンを実装する
dip_tech
PRO
0
200
米国国防総省のDevSecOpsライフサイクルをAWSのセキュリティサービスとOSSで実現
syoshie
2
1.1k
PHP開発者のためのSOLID原則再入門 #phpcon / PHP Conference Japan 2025
shogogg
4
730
_第3回__AIxIoTビジネス共創ラボ紹介資料_20250617.pdf
iotcomjpadmin
0
150
Featured
See All Featured
Optimizing for Happiness
mojombo
379
70k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
700
Designing Experiences People Love
moore
142
24k
Speed Design
sergeychernyshev
32
1k
Visualization
eitanlees
146
16k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.7k
A Tale of Four Properties
chriscoyier
160
23k
Thoughts on Productivity
jonyablonski
69
4.7k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.2k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
Transcript
ϔϏʔͳ Graphite ӡ༻ id:y_uuki Monitoring Casual Talks #6
@y_uuk1 id:y_uuki ӡ༻
Graphite
Graphite Data Flow Graphite Webapp Carbon Whisper Collector TCP HTTP
metrics Graph JSON Store Get Graphite
carbon-cache • 1ϓϩηε/2εϨου • ίωΫγϣϯϋϯυϦϯάεϨου • ॻ͖ࠐΈεϨου • carbon-relay: carbon-cache
ͷલஈ • (carbon-aggregator)
whisper • RRDtool ͱ΄΅ಉ͡σʔλߏ • 1ϑΝΠϧ1ϝτϦοΫ (xxxhost ͷ loadavg) •
ಠࣗܗࣜͷϑΝΠϧΛ࡞Δ python ϥΠϒϥϦ
graphite-web • Web ΞϓϦέʔγϣϯαʔό • ศརάϥϑURLه๏ (ը૾ or json) •
Python Django • ApacheඞਢͰͳ͍ (uwsgi, gunicorn) • ͍ͩ͞μογϡϘʔυ
app ॳظߏ XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 carbon-cache graphite-web whisper
app LVS carbon-relay LVS carbon-relay carbon-cache graphite-web carbon-cache graphite-web SFQMJDBUJPO
XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 whisper whisper XSJUFSFRVFTU QJDLFQSPUPDPM Խ DBSCPOSFMBZ
όοΫΞοϓ app LVS carbon-relay LVS carbon-relay carbon- cache graphite- web
master whisper carbon- cache graphite- web master whisper carbon- cache graphite- web backup whisper
carbon-cache ͕CPUͰ٧·Δ
multiple carbon-cache app LVS LVS carbon-relay carbon-relay carbon- cache graphite-web
tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-backup carbon- cache carbon- cache carbon- cache carbon- relay DPOTJTUFOU IBTIJOH
carbon-relay ͕CPUͰ٧·Δ
! ! tsdb-relay-lb app LVS LVS carbon-relay carbon-relay carbon- relay
LVS LVS carbon- relay carbon- relay carbon- relay carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web backup carbon- cache carbon- cache carbon- cache carbon- relay
10k scale Write IOPS ioDrive CPUϘτϧωοΫ
ύϥϝʔλνϡʔχϯά
carbon-cache MAX_CACHE_SIZE = inf MAX_UPDATES_PER_SECOND = inf MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = inf
MAX_CREATES_PER_MINUTE = inf CACHE_WRITE_STRATEGY = naive
carbon-cache MAX_CACHE_SIZE = MAX_UPDATES_PER_SECOND = MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = MAX_CREATES_PER_MINUTE = CACHE_WRITE_STRATEGY
= $16͞ΔͷͰ ͳʹΒͤͳ͍
σʔλͷҰ؏ੑ
DRBD
app %3#% XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 carbon-cache graphite-web primary whisper
carbon-cache graphite-web secondary whisper
ఆظతͳόοΫΞοϓ • Ұ؏ੑͪΌΜͱ͢ΔͳΒߋ৽ΛͱΊΔΈ͕ ผ్ඞཁ (stop slave IO_THREAD;ͱ͔ͳ͍) • ߋ৽ͱΊͯΔؒʹ rsync
/ nc • ϝοηʔδΩϡʔతͳͷʹஷΊ͜Ή • ΦϯϥΠϯͰόοΫΞοϓͨ͠ͱͯ͠Ұఆظ ؒޙwhisper͕aggregate͢Δ
ͦͷଞ • ҙ֎ͱηοτΞοϓ໘ • தಁ͚ͯΔʢPythonʣ • whisper ͷσʔλߏI/Oޮѱ͍ • MySQL
ͱ͔͑Δ͚ͲϢʔβใͱ͔͍ΕΔ ͚ͩ
ȦȀȀ✣Ǐ
https://mackerel.io “৽͍͠ΞϓϦέʔγϣϯύϑΥʔϚϯε Ϛωδϝϯτ”
None
ϞχλϦϯά͕͖ͳΤϯδχΞΛ ืू͓ͯ͠Γ·͢