Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
ヘビーなGraphite運用
Yuuki Tsubouchi (yuuk1)
June 12, 2014
Technology
5
290k
ヘビーなGraphite運用
Monitoring Casual Talks #6
Yuuki Tsubouchi (yuuk1)
June 12, 2014
Tweet
Share
More Decks by Yuuki Tsubouchi (yuuk1)
See All by Yuuki Tsubouchi (yuuk1)
yuukit
4
1.6k
yuukit
1
770
yuukit
0
77
yuukit
0
1.2k
yuukit
0
890
yuukit
0
310
yuukit
5
810
yuukit
0
120
yuukit
0
50
Other Decks in Technology
See All in Technology
korodroid
0
220
kanaugust
PRO
0
120
sadayoshitada0919
0
360
ippey
2
210
kanaugust
PRO
0
160
udonyuya
1
560
tosh2230
3
320
yshr1200
0
170
hagyyyy
1
200
hololab
0
170
koukyo1994
5
1.6k
piazza
0
100
Featured
See All Featured
thoeni
3
610
iamctodd
19
2k
nonsquared
81
3.4k
mthomps
38
2.3k
jonyablonski
19
1.2k
lemiorhan
627
43k
pauljervisheath
195
15k
dougneiner
118
7.9k
bkeepers
408
58k
ammeep
656
54k
chriscoyier
683
180k
orderedlist
PRO
328
36k
Transcript
ϔϏʔͳ Graphite ӡ༻ id:y_uuki Monitoring Casual Talks #6
@y_uuk1 id:y_uuki ӡ༻
Graphite
Graphite Data Flow Graphite Webapp Carbon Whisper Collector TCP HTTP
metrics Graph JSON Store Get Graphite
carbon-cache • 1ϓϩηε/2εϨου • ίωΫγϣϯϋϯυϦϯάεϨου • ॻ͖ࠐΈεϨου • carbon-relay: carbon-cache
ͷલஈ • (carbon-aggregator)
whisper • RRDtool ͱ΄΅ಉ͡σʔλߏ • 1ϑΝΠϧ1ϝτϦοΫ (xxxhost ͷ loadavg) •
ಠࣗܗࣜͷϑΝΠϧΛ࡞Δ python ϥΠϒϥϦ
graphite-web • Web ΞϓϦέʔγϣϯαʔό • ศརάϥϑURLه๏ (ը૾ or json) •
Python Django • ApacheඞਢͰͳ͍ (uwsgi, gunicorn) • ͍ͩ͞μογϡϘʔυ
app ॳظߏ XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 carbon-cache graphite-web whisper
app LVS carbon-relay LVS carbon-relay carbon-cache graphite-web carbon-cache graphite-web SFQMJDBUJPO
XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 whisper whisper XSJUFSFRVFTU QJDLFQSPUPDPM Խ DBSCPOSFMBZ
όοΫΞοϓ app LVS carbon-relay LVS carbon-relay carbon- cache graphite- web
master whisper carbon- cache graphite- web master whisper carbon- cache graphite- web backup whisper
carbon-cache ͕CPUͰ٧·Δ
multiple carbon-cache app LVS LVS carbon-relay carbon-relay carbon- cache graphite-web
tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-backup carbon- cache carbon- cache carbon- cache carbon- relay DPOTJTUFOU IBTIJOH
carbon-relay ͕CPUͰ٧·Δ
! ! tsdb-relay-lb app LVS LVS carbon-relay carbon-relay carbon- relay
LVS LVS carbon- relay carbon- relay carbon- relay carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web backup carbon- cache carbon- cache carbon- cache carbon- relay
10k scale Write IOPS ioDrive CPUϘτϧωοΫ
ύϥϝʔλνϡʔχϯά
carbon-cache MAX_CACHE_SIZE = inf MAX_UPDATES_PER_SECOND = inf MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = inf
MAX_CREATES_PER_MINUTE = inf CACHE_WRITE_STRATEGY = naive
carbon-cache MAX_CACHE_SIZE = MAX_UPDATES_PER_SECOND = MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = MAX_CREATES_PER_MINUTE = CACHE_WRITE_STRATEGY
= $16͞ΔͷͰ ͳʹΒͤͳ͍
σʔλͷҰ؏ੑ
DRBD
app %3#% XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 carbon-cache graphite-web primary whisper
carbon-cache graphite-web secondary whisper
ఆظతͳόοΫΞοϓ • Ұ؏ੑͪΌΜͱ͢ΔͳΒߋ৽ΛͱΊΔΈ͕ ผ్ඞཁ (stop slave IO_THREAD;ͱ͔ͳ͍) • ߋ৽ͱΊͯΔؒʹ rsync
/ nc • ϝοηʔδΩϡʔతͳͷʹஷΊ͜Ή • ΦϯϥΠϯͰόοΫΞοϓͨ͠ͱͯ͠Ұఆظ ؒޙwhisper͕aggregate͢Δ
ͦͷଞ • ҙ֎ͱηοτΞοϓ໘ • தಁ͚ͯΔʢPythonʣ • whisper ͷσʔλߏI/Oޮѱ͍ • MySQL
ͱ͔͑Δ͚ͲϢʔβใͱ͔͍ΕΔ ͚ͩ
ȦȀȀ✣Ǐ
https://mackerel.io “৽͍͠ΞϓϦέʔγϣϯύϑΥʔϚϯε Ϛωδϝϯτ”
None
ϞχλϦϯά͕͖ͳΤϯδχΞΛ ืू͓ͯ͠Γ·͢