ヘビーなGraphite運用

Monitoring Casual Talks #6

Transcript

1. ϔϏʔͳ Graphite ӡ༻ id:y_uuki Monitoring Casual Talks #6
2. @y_uuk1 id:y_uuki ӡ༻
3. Graphite
4. Graphite Data Flow Graphite Webapp Carbon Whisper Collector TCP HTTP metrics Graph JSON Store Get Graphite
5. carbon-cache • 1ϓϩηε/2εϨου • ίωΫγϣϯϋϯυϦϯάεϨου • ॻ͖ࠐΈεϨου • carbon-relay: carbon-cache ͷલஈ • (carbon-aggregator)
6. whisper • RRDtool ͱ΄΅ಉ͡σʔλߏ଄ • 1ϑΝΠϧ1ϝτϦοΫ (xxxhost ͷ loadavg) • ಠࣗܗࣜͷϑΝΠϧΛ࡞Δ python ϥΠϒϥϦ
7. graphite-web • Web ΞϓϦέʔγϣϯαʔό • ศརάϥϑURLه๏ (ը૾ or json) • Python Django • ApacheඞਢͰ͸ͳ͍ (uwsgi, gunicorn) • ͍ͩ͞μογϡϘʔυ
8. app ॳظߏ੒ XSJUF
   SFRVFTU
   QMBJO
   UFYU
   QSPUPDPM SFBE
   SFRVFTU
   )551 carbon-cache graphite-web whisper
9. app LVS carbon-relay LVS carbon-relay carbon-cache graphite-web carbon-cache graphite-web SFQMJDBUJPO XSJUF
   SFRVFTU
   QMBJO
   UFYU
   QSPUPDPM SFBE
   SFRVFTU
   )551 whisper whisper XSJUF
   SFRVFTU
   QJDLF
   QSPUPDPM ৑௕Խ
   DBSCPOSFMBZ
10. όοΫΞοϓ app LVS carbon-relay LVS carbon-relay carbon- cache graphite- web master whisper carbon- cache graphite- web master whisper carbon- cache graphite- web backup whisper
11. carbon-cache ͕CPUͰ٧·Δ
12. multiple carbon-cache app LVS LVS carbon-relay carbon-relay carbon- cache graphite-web tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-backup carbon- cache carbon- cache carbon- cache carbon- relay DPOTJTUFOU
    IBTIJOH
13. carbon-relay ͕CPUͰ٧·Δ
14. ! ! tsdb-relay-lb app LVS LVS carbon-relay carbon-relay carbon- relay LVS LVS carbon- relay carbon- relay carbon- relay carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web backup carbon- cache carbon- cache carbon- cache carbon- relay
15. 10k scale Write IOPS ioDrive CPUϘτϧωοΫ
16. ύϥϝʔλνϡʔχϯά
17. carbon-cache MAX_CACHE_SIZE = inf MAX_UPDATES_PER_SECOND = inf MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = inf MAX_CREATES_PER_MINUTE = inf CACHE_WRITE_STRATEGY = naive
18. carbon-cache MAX_CACHE_SIZE = MAX_UPDATES_PER_SECOND = MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = MAX_CREATES_PER_MINUTE = CACHE_WRITE_STRATEGY = $16ࢗ͞ΔͷͰ
    ͳʹ΋΍Βͤͳ͍
19. σʔλͷҰ؏ੑ
20. DRBD
21. app %3#% XSJUF
    SFRVFTU
    QMBJO
    UFYU
    QSPUPDPM SFBE
    SFRVFTU
    )551 carbon-cache graphite-web primary whisper carbon-cache graphite-web secondary whisper
22. ఆظతͳόοΫΞοϓ • Ұ؏ੑͪΌΜͱ͢ΔͳΒߋ৽ΛͱΊΔ࢓૊Έ͕ ผ్ඞཁ (stop slave IO_THREAD;ͱ͔ͳ͍) • ߋ৽ͱΊͯΔؒʹ rsync / nc • ϝοηʔδΩϡʔతͳ΋ͷʹஷΊ͜Ή • ΦϯϥΠϯͰόοΫΞοϓͨ͠ͱͯ͠΋Ұఆظ ؒޙwhisper͕aggregate͢Δ
23. ͦͷଞ • ҙ֎ͱηοτΞοϓ໘౗ • த਎͸ಁ͚ͯΔʢPythonʣ • whisper ͷσʔλߏ଄I/Oޮ཰ѱ͍ • MySQL ͱ͔࢖͑Δ͚ͲϢʔβ৘ใͱ͔͍ΕΔ ͚ͩ
24. 
    ȦȀȀ✣Ǐ
25. https://mackerel.io “৽͍͠ΞϓϦέʔγϣϯύϑΥʔϚϯε Ϛωδϝϯτ”
27. ϞχλϦϯά͕޷͖ͳΤϯδχΞΛ
    ืू͓ͯ͠Γ·͢