Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ヘビーなGraphite運用
Search
Yuuki Tsubouchi (yuuk1)
June 12, 2014
Technology
5
290k
ヘビーなGraphite運用
Monitoring Casual Talks #6
Yuuki Tsubouchi (yuuk1)
June 12, 2014
Tweet
Share
More Decks by Yuuki Tsubouchi (yuuk1)
See All by Yuuki Tsubouchi (yuuk1)
エンジニアのためのSRE論文への招待 / Introduction to SRE Papers for Engineers
yuukit
1
5.5k
博士課程での研究まとめ 2023年1月版 / Summary of my research in the PhD course
yuukit
1
150
AI時代に向けたクラウドにおける信頼性エンジニアリングの未来構想 / DICOMO2022 6A-1
yuukit
7
2.5k
AIOps研究録―SREのための システム障害の自動原因診断 / SRE NEXT 2022
yuukit
9
11k
Interactive AIOps
yuukit
0
1.8k
Meltria: マイクロサービスにおける 異常検知・原因分析のための データセットの動的生成システム / Meltria in IOTS2021
yuukit
1
1.4k
AIOpsの研究動向と AIOps向けデータセットの動的生成の研究 / Introducing AIOps and A Dynamic Datasets Generating System
yuukit
1
640
分散アプリケーションの高信頼化のための 運用技術に関する研究 / A Study on Operation Technology for High Reliability of Distributed Applications
yuukit
1
700
ネットワークサービスの依存発見に向いた TCP/UDP通信の低負荷なトレース手法 / Low Overhead TCP-UDP Tracing in Kernel
yuukit
5
3.9k
Other Decks in Technology
See All in Technology
LLMプロダクト事業の立ち上げにおける挑戦
layerx
PRO
7
1.4k
業務で使えるかもしれない…!?GitHub Actions の Tips 集 / CI/CD Test Night #7
ponkio_o
PRO
24
6.9k
統計的学習理論読み Chapter 2
kmatsui
1
200
生成AIの不確実性と向き合うためのオブジェクト指向設計
tkikuchi1002
2
660
長文から長文を生成するLLMツールをオープンソースで作ってみた。
tomohisa
2
140
エンジニア候補者向け資料2024.03.28.pdf
macloud
0
2.9k
どう買う?Azure
kuniteru
1
190
家族アルバム みてねで直面してきた技術的負債 / MIXI KAG 2024
isaoshimizu
17
7.7k
任意コード実行の原理
ffri
0
170
技術イベントはなんとかひねり出す 日経の技術広報の取り組み/techpr3
nishiuma
0
220
技育祭2024春 LT Finatextホールディングス
kevinrobot34
1
160
AMLD 2024 - Build Your Own GPT
donlelef
1
260
Featured
See All Featured
Building Effective Engineering Teams - LeadDev
addyosmani
25
1.8k
Why You Should Never Use an ORM
jnunemaker
PRO
50
8.6k
Testing 201, or: Great Expectations
jmmastey
27
6.3k
Building Better People: How to give real-time feedback that sticks.
wjessup
350
18k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
242
20k
A Tale of Four Properties
chriscoyier
150
22k
StorybookのUI Testing Handbookを読んだ
zakiyama
10
4.5k
Music & Morning Musume
bryan
39
5.5k
Documentation Writing (for coders)
carmenintech
59
3.8k
The Cult of Friendly URLs
andyhume
73
5.6k
A Philosophy of Restraint
colly
195
15k
Debugging Ruby Performance
tmm1
68
11k
Transcript
ϔϏʔͳ Graphite ӡ༻ id:y_uuki Monitoring Casual Talks #6
@y_uuk1 id:y_uuki ӡ༻
Graphite
Graphite Data Flow Graphite Webapp Carbon Whisper Collector TCP HTTP
metrics Graph JSON Store Get Graphite
carbon-cache • 1ϓϩηε/2εϨου • ίωΫγϣϯϋϯυϦϯάεϨου • ॻ͖ࠐΈεϨου • carbon-relay: carbon-cache
ͷલஈ • (carbon-aggregator)
whisper • RRDtool ͱ΄΅ಉ͡σʔλߏ • 1ϑΝΠϧ1ϝτϦοΫ (xxxhost ͷ loadavg) •
ಠࣗܗࣜͷϑΝΠϧΛ࡞Δ python ϥΠϒϥϦ
graphite-web • Web ΞϓϦέʔγϣϯαʔό • ศརάϥϑURLه๏ (ը૾ or json) •
Python Django • ApacheඞਢͰͳ͍ (uwsgi, gunicorn) • ͍ͩ͞μογϡϘʔυ
app ॳظߏ XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 carbon-cache graphite-web whisper
app LVS carbon-relay LVS carbon-relay carbon-cache graphite-web carbon-cache graphite-web SFQMJDBUJPO
XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 whisper whisper XSJUFSFRVFTU QJDLFQSPUPDPM Խ DBSCPOSFMBZ
όοΫΞοϓ app LVS carbon-relay LVS carbon-relay carbon- cache graphite- web
master whisper carbon- cache graphite- web master whisper carbon- cache graphite- web backup whisper
carbon-cache ͕CPUͰ٧·Δ
multiple carbon-cache app LVS LVS carbon-relay carbon-relay carbon- cache graphite-web
tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-backup carbon- cache carbon- cache carbon- cache carbon- relay DPOTJTUFOU IBTIJOH
carbon-relay ͕CPUͰ٧·Δ
! ! tsdb-relay-lb app LVS LVS carbon-relay carbon-relay carbon- relay
LVS LVS carbon- relay carbon- relay carbon- relay carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web backup carbon- cache carbon- cache carbon- cache carbon- relay
10k scale Write IOPS ioDrive CPUϘτϧωοΫ
ύϥϝʔλνϡʔχϯά
carbon-cache MAX_CACHE_SIZE = inf MAX_UPDATES_PER_SECOND = inf MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = inf
MAX_CREATES_PER_MINUTE = inf CACHE_WRITE_STRATEGY = naive
carbon-cache MAX_CACHE_SIZE = MAX_UPDATES_PER_SECOND = MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = MAX_CREATES_PER_MINUTE = CACHE_WRITE_STRATEGY
= $16͞ΔͷͰ ͳʹΒͤͳ͍
σʔλͷҰ؏ੑ
DRBD
app %3#% XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 carbon-cache graphite-web primary whisper
carbon-cache graphite-web secondary whisper
ఆظతͳόοΫΞοϓ • Ұ؏ੑͪΌΜͱ͢ΔͳΒߋ৽ΛͱΊΔΈ͕ ผ్ඞཁ (stop slave IO_THREAD;ͱ͔ͳ͍) • ߋ৽ͱΊͯΔؒʹ rsync
/ nc • ϝοηʔδΩϡʔతͳͷʹஷΊ͜Ή • ΦϯϥΠϯͰόοΫΞοϓͨ͠ͱͯ͠Ұఆظ ؒޙwhisper͕aggregate͢Δ
ͦͷଞ • ҙ֎ͱηοτΞοϓ໘ • தಁ͚ͯΔʢPythonʣ • whisper ͷσʔλߏI/Oޮѱ͍ • MySQL
ͱ͔͑Δ͚ͲϢʔβใͱ͔͍ΕΔ ͚ͩ
ȦȀȀ✣Ǐ
https://mackerel.io “৽͍͠ΞϓϦέʔγϣϯύϑΥʔϚϯε Ϛωδϝϯτ”
None
ϞχλϦϯά͕͖ͳΤϯδχΞΛ ืू͓ͯ͠Γ·͢