ヘビーなGraphite運用

Transcript

1. 1.

   ϔϏʔͳ Graphite ӡ༻ id:y_uuki Monitoring Casual Talks #6

2. 2.

   @y_uuk1 id:y_uuki ӡ༻

3. 3.

   Graphite

4. 4.

   Graphite Data Flow Graphite Webapp Carbon Whisper Collector TCP HTTP

   metrics Graph JSON Store Get Graphite
5. 5.

   carbon-cache • 1ϓϩηε/2εϨου • ίωΫγϣϯϋϯυϦϯάεϨου • ॻ͖ࠐΈεϨου • carbon-relay: carbon-cache

   ͷલஈ • (carbon-aggregator)
6. 6.

   whisper • RRDtool ͱ΄΅ಉ͡σʔλߏ଄ • 1ϑΝΠϧ1ϝτϦοΫ (xxxhost ͷ loadavg) •

   ಠࣗܗࣜͷϑΝΠϧΛ࡞Δ python ϥΠϒϥϦ
7. 7.

   graphite-web • Web ΞϓϦέʔγϣϯαʔό • ศརάϥϑURLه๏ (ը૾ or json) •

   Python Django • ApacheඞਢͰ͸ͳ͍ (uwsgi, gunicorn) • ͍ͩ͞μογϡϘʔυ
8. 8.

   app ॳظߏ੒ XSJUF
   SFRVFTU
   QMBJO
   UFYU
   QSPUPDPM SFBE
   SFRVFTU
   )551 carbon-cache graphite-web whisper

9. 9.

   app LVS carbon-relay LVS carbon-relay carbon-cache graphite-web carbon-cache graphite-web SFQMJDBUJPO

   XSJUF
   SFRVFTU
   QMBJO
   UFYU
   QSPUPDPM SFBE
   SFRVFTU
   )551 whisper whisper XSJUF
   SFRVFTU
   QJDLF
   QSPUPDPM ৑௕Խ
   DBSCPOSFMBZ
10. 10.

    όοΫΞοϓ app LVS carbon-relay LVS carbon-relay carbon- cache graphite- web

    master whisper carbon- cache graphite- web master whisper carbon- cache graphite- web backup whisper
11. 11.

    carbon-cache ͕CPUͰ٧·Δ

12. 12.

    multiple carbon-cache app LVS LVS carbon-relay carbon-relay carbon- cache graphite-web

    tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-master carbon- cache carbon- cache carbon- cache carbon- relay carbon- cache graphite-web tsdb-backup carbon- cache carbon- cache carbon- cache carbon- relay DPOTJTUFOU
    IBTIJOH
13. 13.

    carbon-relay ͕CPUͰ٧·Δ

14. 14.

    ! ! tsdb-relay-lb app LVS LVS carbon-relay carbon-relay carbon- relay

    LVS LVS carbon- relay carbon- relay carbon- relay carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web master carbon- cache carbon- cache carbon- cache carbon- cache graphite-web backup carbon- cache carbon- cache carbon- cache carbon- relay
15. 15.

    10k scale Write IOPS ioDrive CPUϘτϧωοΫ

16. 16.

    ύϥϝʔλνϡʔχϯά

17. 17.

    carbon-cache MAX_CACHE_SIZE = inf MAX_UPDATES_PER_SECOND = inf MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = inf

    MAX_CREATES_PER_MINUTE = inf CACHE_WRITE_STRATEGY = naive
18. 18.

    carbon-cache MAX_CACHE_SIZE = MAX_UPDATES_PER_SECOND = MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = MAX_CREATES_PER_MINUTE = CACHE_WRITE_STRATEGY

    = $16ࢗ͞ΔͷͰ
    ͳʹ΋΍Βͤͳ͍
19. 19.

    σʔλͷҰ؏ੑ

20. 20.

    DRBD

21. 21.

    app %3#% XSJUF
    SFRVFTU
    QMBJO
    UFYU
    QSPUPDPM SFBE
    SFRVFTU
    )551 carbon-cache graphite-web primary whisper

    carbon-cache graphite-web secondary whisper
22. 22.

    ఆظతͳόοΫΞοϓ • Ұ؏ੑͪΌΜͱ͢ΔͳΒߋ৽ΛͱΊΔ࢓૊Έ͕ ผ్ඞཁ (stop slave IO_THREAD;ͱ͔ͳ͍) • ߋ৽ͱΊͯΔؒʹ rsync

    / nc • ϝοηʔδΩϡʔతͳ΋ͷʹஷΊ͜Ή • ΦϯϥΠϯͰόοΫΞοϓͨ͠ͱͯ͠΋Ұఆظ ؒޙwhisper͕aggregate͢Δ
23. 23.

    ͦͷଞ • ҙ֎ͱηοτΞοϓ໘౗ • த਎͸ಁ͚ͯΔʢPythonʣ • whisper ͷσʔλߏ଄I/Oޮ཰ѱ͍ • MySQL

    ͱ͔࢖͑Δ͚ͲϢʔβ৘ใͱ͔͍ΕΔ ͚ͩ
24. 24.

    
    ȦȀȀ✣Ǐ

25. 25.

    https://mackerel.io “৽͍͠ΞϓϦέʔγϣϯύϑΥʔϚϯε Ϛωδϝϯτ”

26. 26.

    None
27. 27.

    ϞχλϦϯά͕޷͖ͳΤϯδχΞΛ
    ืू͓ͯ͠Γ·͢