Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ヘビーなGraphite運用

 ヘビーなGraphite運用

Monitoring Casual Talks #6

Yuuki Tsubouchi (yuuk1)

June 12, 2014
Tweet

More Decks by Yuuki Tsubouchi (yuuk1)

Other Decks in Technology

Transcript

  1. ϔϏʔͳ Graphite ӡ༻
    id:y_uuki
    Monitoring Casual Talks #6

    View Slide

  2. @y_uuk1
    id:y_uuki
    ӡ༻

    View Slide

  3. Graphite

    View Slide

  4. Graphite Data Flow
    Graphite
    Webapp
    Carbon
    Whisper
    Collector
    TCP
    HTTP
    metrics
    Graph
    JSON
    Store
    Get
    Graphite

    View Slide

  5. carbon-cache
    • 1ϓϩηε/2εϨου
    • ίωΫγϣϯϋϯυϦϯάεϨου
    • ॻ͖ࠐΈεϨου
    • carbon-relay: carbon-cache ͷલஈ
    • (carbon-aggregator)

    View Slide

  6. whisper
    • RRDtool ͱ΄΅ಉ͡σʔλߏ଄
    • 1ϑΝΠϧ1ϝτϦοΫ (xxxhost ͷ loadavg)
    • ಠࣗܗࣜͷϑΝΠϧΛ࡞Δ python ϥΠϒϥϦ

    View Slide

  7. graphite-web
    • Web ΞϓϦέʔγϣϯαʔό
    • ศརάϥϑURLه๏ (ը૾ or json)
    • Python Django
    • ApacheඞਢͰ͸ͳ͍ (uwsgi, gunicorn)
    • ͍ͩ͞μογϡϘʔυ

    View Slide

  8. app
    ॳظߏ੒
    XSJUFSFRVFTU
    QMBJOUFYUQSPUPDPM
    SFBESFRVFTU
    )551

    carbon-cache graphite-web
    whisper

    View Slide

  9. app
    LVS
    carbon-relay
    LVS
    carbon-relay
    carbon-cache graphite-web carbon-cache graphite-web
    SFQMJDBUJPO
    XSJUFSFRVFTU
    QMBJOUFYUQSPUPDPM
    SFBESFRVFTU
    )551

    whisper whisper
    XSJUFSFRVFTU
    QJDLFQSPUPDPM

    ৑௕Խ DBSCPOSFMBZ

    View Slide

  10. όοΫΞοϓ
    app
    LVS
    carbon-relay
    LVS
    carbon-relay
    carbon-
    cache
    graphite-
    web
    master
    whisper
    carbon-
    cache
    graphite-
    web
    master
    whisper
    carbon-
    cache
    graphite-
    web
    backup
    whisper

    View Slide

  11. carbon-cache ͕CPUͰ٧·Δ

    View Slide

  12. multiple carbon-cache
    app
    LVS LVS
    carbon-relay carbon-relay
    carbon-
    cache
    graphite-web
    tsdb-master
    carbon-
    cache
    carbon-
    cache
    carbon-
    cache
    carbon-
    relay
    carbon-
    cache
    graphite-web
    tsdb-master
    carbon-
    cache
    carbon-
    cache
    carbon-
    cache
    carbon-
    relay
    carbon-
    cache
    graphite-web
    tsdb-backup
    carbon-
    cache
    carbon-
    cache
    carbon-
    cache
    carbon-
    relay
    DPOTJTUFOU
    IBTIJOH

    View Slide

  13. carbon-relay ͕CPUͰ٧·Δ

    View Slide

  14. !
    !
    tsdb-relay-lb
    app
    LVS LVS
    carbon-relay carbon-relay
    carbon-
    relay
    LVS LVS
    carbon-
    relay
    carbon-
    relay
    carbon-
    relay
    carbon-
    cache
    graphite-web
    master
    carbon-
    cache
    carbon-
    cache
    carbon-
    cache
    carbon-
    cache
    graphite-web
    master
    carbon-
    cache
    carbon-
    cache
    carbon-
    cache
    carbon-
    cache
    graphite-web
    backup
    carbon-
    cache
    carbon-
    cache
    carbon-
    cache
    carbon-
    relay

    View Slide

  15. 10k scale Write IOPS
    ioDrive
    CPUϘτϧωοΫ

    View Slide

  16. ύϥϝʔλνϡʔχϯά

    View Slide

  17. carbon-cache
    MAX_CACHE_SIZE = inf
    MAX_UPDATES_PER_SECOND = inf
    MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = inf
    MAX_CREATES_PER_MINUTE = inf
    CACHE_WRITE_STRATEGY = naive

    View Slide

  18. carbon-cache
    MAX_CACHE_SIZE =
    MAX_UPDATES_PER_SECOND =
    MAX_UPDATES_PER_SECOND_ON_SHUTDOWN =
    MAX_CREATES_PER_MINUTE =
    CACHE_WRITE_STRATEGY =
    $16ࢗ͞ΔͷͰ
    ͳʹ΋΍Βͤͳ͍

    View Slide

  19. σʔλͷҰ؏ੑ

    View Slide

  20. DRBD

    View Slide

  21. app
    %3#%
    XSJUFSFRVFTU
    QMBJOUFYUQSPUPDPM
    SFBESFRVFTU
    )551

    carbon-cache graphite-web
    primary
    whisper
    carbon-cache graphite-web
    secondary
    whisper

    View Slide

  22. ఆظతͳόοΫΞοϓ
    • Ұ؏ੑͪΌΜͱ͢ΔͳΒߋ৽ΛͱΊΔ࢓૊Έ͕
    ผ్ඞཁ (stop slave IO_THREAD;ͱ͔ͳ͍)
    • ߋ৽ͱΊͯΔؒʹ rsync / nc
    • ϝοηʔδΩϡʔతͳ΋ͷʹஷΊ͜Ή
    • ΦϯϥΠϯͰόοΫΞοϓͨ͠ͱͯ͠΋Ұఆظ
    ؒޙwhisper͕aggregate͢Δ

    View Slide

  23. ͦͷଞ
    • ҙ֎ͱηοτΞοϓ໘౗
    • த਎͸ಁ͚ͯΔʢPythonʣ
    • whisper ͷσʔλߏ଄I/Oޮ཰ѱ͍
    • MySQL ͱ͔࢖͑Δ͚ͲϢʔβ৘ใͱ͔͍ΕΔ
    ͚ͩ

    View Slide

  24. ȦȀȀ✣Ǐ

    View Slide

  25. https://mackerel.io
    “৽͍͠ΞϓϦέʔγϣϯύϑΥʔϚϯε
    Ϛωδϝϯτ”

    View Slide

  26. View Slide

  27. ϞχλϦϯά͕޷͖ͳΤϯδχΞΛ
    ืू͓ͯ͠Γ·͢

    View Slide