ヘビーなGraphite運用

Slide 1

Slide 1 text

ϔϏʔͳ Graphite ӡ༻ id:y_uuki Monitoring Casual Talks #6

Slide 2

Slide 2 text

@y_uuk1 id:y_uuki ӡ༻

Slide 3

Slide 3 text

Graphite

Slide 4

Slide 4 text

Graphite Data Flow Graphite Webapp Carbon Whisper Collector TCP HTTP metrics Graph JSON Store Get Graphite

Slide 5

Slide 5 text

carbon-cache • 1ϓϩηε/2εϨου • ίωΫγϣϯϋϯυϦϯάεϨου • ॻ͖ࠐΈεϨου • carbon-relay: carbon-cache ͷલஈ • (carbon-aggregator)

Slide 6

Slide 6 text

whisper • RRDtool ͱ΄΅ಉ͡σʔλߏ଄ • 1ϑΝΠϧ1ϝτϦοΫ (xxxhost ͷ loadavg) • ಠࣗܗࣜͷϑΝΠϧΛ࡞Δ python ϥΠϒϥϦ

Slide 7

Slide 7 text

graphite-web • Web ΞϓϦέʔγϣϯαʔό • ศརάϥϑURLه๏ (ը૾ or json) • Python Django • ApacheඞਢͰ͸ͳ͍ (uwsgi, gunicorn) • ͍ͩ͞μογϡϘʔυ

Slide 8

Slide 8 text

app ॳظߏ੒ XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 carbon-cache graphite-web whisper

Slide 9

Slide 9 text

app LVS carbon-relay LVS carbon-relay carbon-cache graphite-web carbon-cache graphite-web SFQMJDBUJPO XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 whisper whisper XSJUFSFRVFTU QJDLFQSPUPDPM ৑௕Խ DBSCPOSFMBZ

Slide 10

Slide 10 text

όοΫΞοϓ app LVS carbon-relay LVS carbon-relay carbon- cache graphite- web master whisper carbon- cache graphite- web master whisper carbon- cache graphite- web backup whisper

Slide 11

Slide 11 text

carbon-cache ͕CPUͰ٧·Δ

Slide 12

Slide 12 text

multiple carbon-cache app LVS LVS carbon-relay carbon-relay carbon- cache graphite-web tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-backup carbon- cache carbon- cache carbon- cache carbon- relay DPOTJTUFOU IBTIJOH

Slide 13

Slide 13 text

carbon-relay ͕CPUͰ٧·Δ

Slide 14

Slide 14 text

! ! tsdb-relay-lb app LVS LVS carbon-relay carbon-relay carbon- relay LVS LVS carbon- relay carbon- relay carbon- relay carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web backup carbon- cache carbon- cache carbon- cache carbon- relay

Slide 15

Slide 15 text

10k scale Write IOPS ioDrive CPUϘτϧωοΫ

Slide 16

Slide 16 text

ύϥϝʔλνϡʔχϯά

Slide 17

Slide 17 text

carbon-cache MAX_CACHE_SIZE = inf MAX_UPDATES_PER_SECOND = inf MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = inf MAX_CREATES_PER_MINUTE = inf CACHE_WRITE_STRATEGY = naive

Slide 18

Slide 18 text

carbon-cache MAX_CACHE_SIZE = MAX_UPDATES_PER_SECOND = MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = MAX_CREATES_PER_MINUTE = CACHE_WRITE_STRATEGY = $16ࢗ͞ΔͷͰ ͳʹ΋΍Βͤͳ͍

Slide 19

Slide 19 text

σʔλͷҰ؏ੑ

Slide 20

Slide 20 text

DRBD

Slide 21

Slide 21 text

app %3#% XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 carbon-cache graphite-web primary whisper carbon-cache graphite-web secondary whisper

Slide 22

Slide 22 text

ఆظతͳόοΫΞοϓ • Ұ؏ੑͪΌΜͱ͢ΔͳΒߋ৽ΛͱΊΔ࢓૊Έ͕ ผ్ඞཁ (stop slave IO_THREAD;ͱ͔ͳ͍) • ߋ৽ͱΊͯΔؒʹ rsync / nc • ϝοηʔδΩϡʔతͳ΋ͷʹஷΊ͜Ή • ΦϯϥΠϯͰόοΫΞοϓͨ͠ͱͯ͠΋Ұఆظ ؒޙwhisper͕aggregate͢Δ

Slide 23

Slide 23 text

ͦͷଞ • ҙ֎ͱηοτΞοϓ໘౗ • த਎͸ಁ͚ͯΔʢPythonʣ • whisper ͷσʔλߏ଄I/Oޮ཰ѱ͍ • MySQL ͱ͔࢖͑Δ͚ͲϢʔβ৘ใͱ͔͍ΕΔ ͚ͩ