ヘビーなGraphite運用

 ヘビーなGraphite運用

Monitoring Casual Talks #6

A658ec7f1badf73819dfa501165016c1?s=128

Yuuki Tsubouchi (yuuk1)

June 12, 2014
Tweet

Transcript

  1. ϔϏʔͳ Graphite ӡ༻ id:y_uuki Monitoring Casual Talks #6

  2. @y_uuk1 id:y_uuki ӡ༻

  3. Graphite

  4. Graphite Data Flow Graphite Webapp Carbon Whisper Collector TCP HTTP

    metrics Graph JSON Store Get Graphite
  5. carbon-cache • 1ϓϩηε/2εϨου • ίωΫγϣϯϋϯυϦϯάεϨου • ॻ͖ࠐΈεϨου • carbon-relay: carbon-cache

    ͷલஈ • (carbon-aggregator)
  6. whisper • RRDtool ͱ΄΅ಉ͡σʔλߏ଄ • 1ϑΝΠϧ1ϝτϦοΫ (xxxhost ͷ loadavg) •

    ಠࣗܗࣜͷϑΝΠϧΛ࡞Δ python ϥΠϒϥϦ
  7. graphite-web • Web ΞϓϦέʔγϣϯαʔό • ศརάϥϑURLه๏ (ը૾ or json) •

    Python Django • ApacheඞਢͰ͸ͳ͍ (uwsgi, gunicorn) • ͍ͩ͞μογϡϘʔυ
  8. app ॳظߏ੒ XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 carbon-cache graphite-web whisper

  9. app LVS carbon-relay LVS carbon-relay carbon-cache graphite-web carbon-cache graphite-web SFQMJDBUJPO

    XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 whisper whisper XSJUFSFRVFTU QJDLFQSPUPDPM ৑௕Խ DBSCPOSFMBZ
  10. όοΫΞοϓ app LVS carbon-relay LVS carbon-relay carbon- cache graphite- web

    master whisper carbon- cache graphite- web master whisper carbon- cache graphite- web backup whisper
  11. carbon-cache ͕CPUͰ٧·Δ

  12. multiple carbon-cache app LVS LVS carbon-relay carbon-relay carbon- cache graphite-web

    tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-backup carbon- cache carbon- cache carbon- cache carbon- relay DPOTJTUFOU IBTIJOH
  13. carbon-relay ͕CPUͰ٧·Δ

  14. ! ! tsdb-relay-lb app LVS LVS carbon-relay carbon-relay carbon- relay

    LVS LVS carbon- relay carbon- relay carbon- relay carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web backup carbon- cache carbon- cache carbon- cache carbon- relay
  15. 10k scale Write IOPS ioDrive CPUϘτϧωοΫ

  16. ύϥϝʔλνϡʔχϯά

  17. carbon-cache MAX_CACHE_SIZE = inf MAX_UPDATES_PER_SECOND = inf MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = inf

    MAX_CREATES_PER_MINUTE = inf CACHE_WRITE_STRATEGY = naive
  18. carbon-cache MAX_CACHE_SIZE = MAX_UPDATES_PER_SECOND = MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = MAX_CREATES_PER_MINUTE = CACHE_WRITE_STRATEGY

    = $16ࢗ͞ΔͷͰ ͳʹ΋΍Βͤͳ͍
  19. σʔλͷҰ؏ੑ

  20. DRBD

  21. app %3#% XSJUFSFRVFTU QMBJOUFYUQSPUPDPM SFBESFRVFTU )551 carbon-cache graphite-web primary whisper

    carbon-cache graphite-web secondary whisper
  22. ఆظతͳόοΫΞοϓ • Ұ؏ੑͪΌΜͱ͢ΔͳΒߋ৽ΛͱΊΔ࢓૊Έ͕ ผ్ඞཁ (stop slave IO_THREAD;ͱ͔ͳ͍) • ߋ৽ͱΊͯΔؒʹ rsync

    / nc • ϝοηʔδΩϡʔతͳ΋ͷʹஷΊ͜Ή • ΦϯϥΠϯͰόοΫΞοϓͨ͠ͱͯ͠΋Ұఆظ ؒޙwhisper͕aggregate͢Δ
  23. ͦͷଞ • ҙ֎ͱηοτΞοϓ໘౗ • த਎͸ಁ͚ͯΔʢPythonʣ • whisper ͷσʔλߏ଄I/Oޮ཰ѱ͍ • MySQL

    ͱ͔࢖͑Δ͚ͲϢʔβ৘ใͱ͔͍ΕΔ ͚ͩ
  24. ȦȀȀ✣Ǐ

  25. https://mackerel.io “৽͍͠ΞϓϦέʔγϣϯύϑΥʔϚϯε Ϛωδϝϯτ”

  26. None
  27. ϞχλϦϯά͕޷͖ͳΤϯδχΞΛ ืू͓ͯ͠Γ·͢