ヘビーなGraphite運用

Monitoring Casual Talks #6

Transcript

1. ϔϏʔͳ Graphite ӡ༻
   id:y_uuki
   Monitoring Casual Talks #6
   

   View Slide

2. @y_uuk1
   id:y_uuki
   ӡ༻
   

   View Slide

3. Graphite
   

   View Slide

4. Graphite Data Flow
   Graphite
   Webapp
   Carbon
   Whisper
   Collector
   TCP
   HTTP
   metrics
   Graph
   JSON
   Store
   Get
   Graphite
   

   View Slide

5. carbon-cache
   • 1ϓϩηε/2εϨου
   • ίωΫγϣϯϋϯυϦϯάεϨου
   • ॻ͖ࠐΈεϨου
   • carbon-relay: carbon-cache ͷલஈ
   • (carbon-aggregator)
   

   View Slide

6. whisper
   • RRDtool ͱ΄΅ಉ͡σʔλߏ଄
   • 1ϑΝΠϧ1ϝτϦοΫ (xxxhost ͷ loadavg)
   • ಠࣗܗࣜͷϑΝΠϧΛ࡞Δ python ϥΠϒϥϦ
   

   View Slide

7. graphite-web
   • Web ΞϓϦέʔγϣϯαʔό
   • ศརάϥϑURLه๏ (ը૾ or json)
   • Python Django
   • ApacheඞਢͰ͸ͳ͍ (uwsgi, gunicorn)
   • ͍ͩ͞μογϡϘʔυ
   

   View Slide

8. app
   ॳظߏ੒
   XSJUFSFRVFTU
   QMBJOUFYUQSPUPDPM
   SFBESFRVFTU
   )551
   
   carbon-cache graphite-web
   whisper
   

   View Slide

9. app
   LVS
   carbon-relay
   LVS
   carbon-relay
   carbon-cache graphite-web carbon-cache graphite-web
   SFQMJDBUJPO
   XSJUFSFRVFTU
   QMBJOUFYUQSPUPDPM
   SFBESFRVFTU
   )551
   
   whisper whisper
   XSJUFSFRVFTU
   QJDLFQSPUPDPM
   
   ৑௕Խ DBSCPOSFMBZ
   
   

   View Slide

10. όοΫΞοϓ
    app
    LVS
    carbon-relay
    LVS
    carbon-relay
    carbon-
    cache
    graphite-
    web
    master
    whisper
    carbon-
    cache
    graphite-
    web
    master
    whisper
    carbon-
    cache
    graphite-
    web
    backup
    whisper
    

    View Slide

11. carbon-cache ͕CPUͰ٧·Δ
    

    View Slide

12. multiple carbon-cache
    app
    LVS LVS
    carbon-relay carbon-relay
    carbon-
    cache
    graphite-web
    tsdb-master
    carbon-
    cache
    carbon-
    cache
    carbon-
    cache
    carbon-
    relay
    carbon-
    cache
    graphite-web
    tsdb-master
    carbon-
    cache
    carbon-
    cache
    carbon-
    cache
    carbon-
    relay
    carbon-
    cache
    graphite-web
    tsdb-backup
    carbon-
    cache
    carbon-
    cache
    carbon-
    cache
    carbon-
    relay
    DPOTJTUFOU
    IBTIJOH
    

    View Slide

13. carbon-relay ͕CPUͰ٧·Δ
    

    View Slide

14. !
    !
    tsdb-relay-lb
    app
    LVS LVS
    carbon-relay carbon-relay
    carbon-
    relay
    LVS LVS
    carbon-
    relay
    carbon-
    relay
    carbon-
    relay
    carbon-
    cache
    graphite-web
    master
    carbon-
    cache
    carbon-
    cache
    carbon-
    cache
    carbon-
    cache
    graphite-web
    master
    carbon-
    cache
    carbon-
    cache
    carbon-
    cache
    carbon-
    cache
    graphite-web
    backup
    carbon-
    cache
    carbon-
    cache
    carbon-
    cache
    carbon-
    relay
    

    View Slide

15. 10k scale Write IOPS
    ioDrive
    CPUϘτϧωοΫ
    

    View Slide

16. ύϥϝʔλνϡʔχϯά
    

    View Slide

17. carbon-cache
    MAX_CACHE_SIZE = inf
    MAX_UPDATES_PER_SECOND = inf
    MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = inf
    MAX_CREATES_PER_MINUTE = inf
    CACHE_WRITE_STRATEGY = naive
    

    View Slide

18. carbon-cache
    MAX_CACHE_SIZE =
    MAX_UPDATES_PER_SECOND =
    MAX_UPDATES_PER_SECOND_ON_SHUTDOWN =
    MAX_CREATES_PER_MINUTE =
    CACHE_WRITE_STRATEGY =
    $16ࢗ͞ΔͷͰ
    ͳʹ΋΍Βͤͳ͍
    

    View Slide

19. σʔλͷҰ؏ੑ
    

    View Slide

20. DRBD
    

    View Slide

21. app
    %3#%
    XSJUFSFRVFTU
    QMBJOUFYUQSPUPDPM
    SFBESFRVFTU
    )551
    
    carbon-cache graphite-web
    primary
    whisper
    carbon-cache graphite-web
    secondary
    whisper
    

    View Slide

22. ఆظతͳόοΫΞοϓ
    • Ұ؏ੑͪΌΜͱ͢ΔͳΒߋ৽ΛͱΊΔ࢓૊Έ͕
    ผ్ඞཁ (stop slave IO_THREAD;ͱ͔ͳ͍)
    • ߋ৽ͱΊͯΔؒʹ rsync / nc
    • ϝοηʔδΩϡʔతͳ΋ͷʹஷΊ͜Ή
    • ΦϯϥΠϯͰόοΫΞοϓͨ͠ͱͯ͠΋Ұఆظ
    ؒޙwhisper͕aggregate͢Δ
    

    View Slide

23. ͦͷଞ
    • ҙ֎ͱηοτΞοϓ໘౗
    • த਎͸ಁ͚ͯΔʢPythonʣ
    • whisper ͷσʔλߏ଄I/Oޮ཰ѱ͍
    • MySQL ͱ͔࢖͑Δ͚ͲϢʔβ৘ใͱ͔͍ΕΔ
    ͚ͩ
    

    View Slide

24. ȦȀȀ✣Ǐ
    

    View Slide

25. https://mackerel.io
    “৽͍͠ΞϓϦέʔγϣϯύϑΥʔϚϯε
    Ϛωδϝϯτ”
    

    View Slide

26. View Slide

27. ϞχλϦϯά͕޷͖ͳΤϯδχΞΛ
    ืू͓ͯ͠Γ·͢
    

    View Slide