Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ヘビーなGraphite運用
Search
Yuuki Tsubouchi (yuuk1)
June 12, 2014
Technology
5
290k
ヘビーなGraphite運用
Monitoring Casual Talks #6
Yuuki Tsubouchi (yuuk1)
June 12, 2014
Tweet
Share
More Decks by Yuuki Tsubouchi (yuuk1)
See All by Yuuki Tsubouchi (yuuk1)
eBPFを用いたAIネットワーク監視システム論文の実装 / eBPF Japan Meetup #4
yuukit
3
700
クラウドのテレメトリーシステム研究動向2025年
yuukit
3
930
博士論文公聴会: Scaling Telemetry Workloads in Cloud Applications: Techniques for Instrumentation, Storage, and Mining / PhD Defence
yuukit
1
150
博士学位論文予備審査 / Scaling Telemetry Workloads in Cloud Applications: Techniques for Instrumentation, Storage, and Mining
yuukit
1
1.9k
MetricSifter:クラウドアプリケーションにおける故障箇所特定の効率化のための多変量時系列データの特徴量削減 / FIT 2024
yuukit
2
250
工学としてのSRE再訪 / Revisiting SRE as Engineering
yuukit
19
14k
Cloudless Computingの論文紹介
yuukit
2
550
#SRE論文紹介 Detection is Better Than Cure: A Cloud Incidents Perspective V. Ganatra et. al., ESEC/FSE’23
yuukit
3
2k
エンジニアのためのSRE論文への招待 / Introduction to SRE Papers for Engineers
yuukit
2
11k
Other Decks in Technology
See All in Technology
障害を回避するHttpClient再入門 / Avoiding Failures HttpClient Reintroduction
uskey512
1
440
Two-Tower モデルで実現する 検索リランキング / Shibuya_AI_2
visional_engineering_and_design
2
120
[zh-TW] DevOpsDays Taipei 2025 -- Creating Awesome Change in SmartNews!(machine translation)
martin_lover
1
690
現場で役立つAPIデザイン
nagix
1
170
Kafka vs. Pulsar: Performance Evaluation by Petabyte-Scale Streaming Platform Providers
lycorptech_jp
PRO
1
300
セキュリティSaaS企業が実践するCursor運用ルールと知見 / How a Security SaaS Company Runs Cursor: Rules & Insights
tetsuzawa
1
2.8k
データベースの引越しを Ora2Pg でスマートにやろう
jri_narita
0
180
Drawing with LLMs
rist
0
200
Tenstorrent HW/SW 概要説明
tenstorrent_japan
0
150
Devin&Cursor、それぞれの「本質」から導く最適ユースケース戦略
empitsu
8
3k
Oracle Cloud Infrastructure IaaS 新機能アップデート 2025/03 - 2025/05
oracle4engineer
PRO
1
130
JavaのMCPサーバーで体験するAIエージェントの世界
tatsuya1bm
1
210
Featured
See All Featured
Build The Right Thing And Hit Your Dates
maggiecrowley
35
2.7k
Statistics for Hackers
jakevdp
799
220k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
Speed Design
sergeychernyshev
30
980
Writing Fast Ruby
sferik
628
61k
YesSQL, Process and Tooling at Scale
rocio
172
14k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Site-Speed That Sticks
csswizardry
9
610
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
45
9.6k
The Pragmatic Product Professional
lauravandoore
35
6.7k
Making the Leap to Tech Lead
cromwellryan
134
9.3k
Automating Front-end Workflow
addyosmani
1370
200k
Transcript
ϔϏʔͳ Graphite ӡ༻ id:y_uuki Monitoring Casual Talks #6
@y_uuk1 id:y_uuki ӡ༻
Graphite
Graphite Data Flow Graphite Webapp Carbon Whisper Collector TCP HTTP
metrics Graph JSON Store Get Graphite
carbon-cache • 1ϓϩηε/2εϨου • ίωΫγϣϯϋϯυϦϯάεϨου • ॻ͖ࠐΈεϨου • carbon-relay: carbon-cache
ͷલஈ • (carbon-aggregator)
whisper • RRDtool ͱ΄΅ಉ͡σʔλߏ • 1ϑΝΠϧ1ϝτϦοΫ (xxxhost ͷ loadavg) •
ಠࣗܗࣜͷϑΝΠϧΛ࡞Δ python ϥΠϒϥϦ
graphite-web • Web ΞϓϦέʔγϣϯαʔό • ศརάϥϑURLه๏ (ը૾ or json) •
Python Django • ApacheඞਢͰͳ͍ (uwsgi, gunicorn) • ͍ͩ͞μογϡϘʔυ
app ॳظߏ XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 carbon-cache graphite-web whisper
app LVS carbon-relay LVS carbon-relay carbon-cache graphite-web carbon-cache graphite-web SFQMJDBUJPO
XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 whisper whisper XSJUFSFRVFTU QJDLFQSPUPDPM Խ DBSCPOSFMBZ
όοΫΞοϓ app LVS carbon-relay LVS carbon-relay carbon- cache graphite- web
master whisper carbon- cache graphite- web master whisper carbon- cache graphite- web backup whisper
carbon-cache ͕CPUͰ٧·Δ
multiple carbon-cache app LVS LVS carbon-relay carbon-relay carbon- cache graphite-web
tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-backup carbon- cache carbon- cache carbon- cache carbon- relay DPOTJTUFOU IBTIJOH
carbon-relay ͕CPUͰ٧·Δ
! ! tsdb-relay-lb app LVS LVS carbon-relay carbon-relay carbon- relay
LVS LVS carbon- relay carbon- relay carbon- relay carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web backup carbon- cache carbon- cache carbon- cache carbon- relay
10k scale Write IOPS ioDrive CPUϘτϧωοΫ
ύϥϝʔλνϡʔχϯά
carbon-cache MAX_CACHE_SIZE = inf MAX_UPDATES_PER_SECOND = inf MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = inf
MAX_CREATES_PER_MINUTE = inf CACHE_WRITE_STRATEGY = naive
carbon-cache MAX_CACHE_SIZE = MAX_UPDATES_PER_SECOND = MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = MAX_CREATES_PER_MINUTE = CACHE_WRITE_STRATEGY
= $16͞ΔͷͰ ͳʹΒͤͳ͍
σʔλͷҰ؏ੑ
DRBD
app %3#% XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 carbon-cache graphite-web primary whisper
carbon-cache graphite-web secondary whisper
ఆظతͳόοΫΞοϓ • Ұ؏ੑͪΌΜͱ͢ΔͳΒߋ৽ΛͱΊΔΈ͕ ผ్ඞཁ (stop slave IO_THREAD;ͱ͔ͳ͍) • ߋ৽ͱΊͯΔؒʹ rsync
/ nc • ϝοηʔδΩϡʔతͳͷʹஷΊ͜Ή • ΦϯϥΠϯͰόοΫΞοϓͨ͠ͱͯ͠Ұఆظ ؒޙwhisper͕aggregate͢Δ
ͦͷଞ • ҙ֎ͱηοτΞοϓ໘ • தಁ͚ͯΔʢPythonʣ • whisper ͷσʔλߏI/Oޮѱ͍ • MySQL
ͱ͔͑Δ͚ͲϢʔβใͱ͔͍ΕΔ ͚ͩ
ȦȀȀ✣Ǐ
https://mackerel.io “৽͍͠ΞϓϦέʔγϣϯύϑΥʔϚϯε Ϛωδϝϯτ”
None
ϞχλϦϯά͕͖ͳΤϯδχΞΛ ืू͓ͯ͠Γ·͢