Monitoring Casual Talks #6
ϔϏʔͳ Graphite ӡ༻id:y_uukiMonitoring Casual Talks #6
View Slide
@y_uuk1id:y_uukiӡ༻
Graphite
Graphite Data FlowGraphiteWebappCarbonWhisperCollectorTCPHTTPmetricsGraphJSONStoreGetGraphite
carbon-cache• 1ϓϩηε/2εϨου• ίωΫγϣϯϋϯυϦϯάεϨου• ॻ͖ࠐΈεϨου• carbon-relay: carbon-cache ͷલஈ• (carbon-aggregator)
whisper• RRDtool ͱ΄΅ಉ͡σʔλߏ• 1ϑΝΠϧ1ϝτϦοΫ (xxxhost ͷ loadavg)• ಠࣗܗࣜͷϑΝΠϧΛ࡞Δ python ϥΠϒϥϦ
graphite-web• Web ΞϓϦέʔγϣϯαʔό• ศརάϥϑURLه๏ (ը૾ or json)• Python Django• ApacheඞਢͰͳ͍ (uwsgi, gunicorn)• ͍ͩ͞μογϡϘʔυ
appॳظߏXSJUFSFRVFTUQMBJOUFYUQSPUPDPM SFBESFRVFTU)551carbon-cache graphite-webwhisper
appLVScarbon-relayLVScarbon-relaycarbon-cache graphite-web carbon-cache graphite-webSFQMJDBUJPOXSJUFSFRVFTUQMBJOUFYUQSPUPDPM SFBESFRVFTU)551whisper whisperXSJUFSFRVFTUQJDLFQSPUPDPMԽ DBSCPOSFMBZ
όοΫΞοϓappLVScarbon-relayLVScarbon-relaycarbon-cachegraphite-webmasterwhispercarbon-cachegraphite-webmasterwhispercarbon-cachegraphite-webbackupwhisper
carbon-cache ͕CPUͰ٧·Δ
multiple carbon-cacheappLVS LVScarbon-relay carbon-relaycarbon-cachegraphite-webtsdb-mastercarbon-cachecarbon-cachecarbon-cachecarbon-relaycarbon-cachegraphite-webtsdb-mastercarbon-cachecarbon-cachecarbon-cachecarbon-relaycarbon-cachegraphite-webtsdb-backupcarbon-cachecarbon-cachecarbon-cachecarbon-relayDPOTJTUFOUIBTIJOH
carbon-relay ͕CPUͰ٧·Δ
!!tsdb-relay-lbappLVS LVScarbon-relay carbon-relaycarbon-relayLVS LVScarbon-relaycarbon-relaycarbon-relaycarbon-cachegraphite-webmastercarbon-cachecarbon-cachecarbon-cachecarbon-cachegraphite-webmastercarbon-cachecarbon-cachecarbon-cachecarbon-cachegraphite-webbackupcarbon-cachecarbon-cachecarbon-cachecarbon-relay
10k scale Write IOPSioDriveCPUϘτϧωοΫ
ύϥϝʔλνϡʔχϯά
carbon-cacheMAX_CACHE_SIZE = infMAX_UPDATES_PER_SECOND = infMAX_UPDATES_PER_SECOND_ON_SHUTDOWN = infMAX_CREATES_PER_MINUTE = infCACHE_WRITE_STRATEGY = naive
carbon-cacheMAX_CACHE_SIZE =MAX_UPDATES_PER_SECOND =MAX_UPDATES_PER_SECOND_ON_SHUTDOWN =MAX_CREATES_PER_MINUTE =CACHE_WRITE_STRATEGY =$16͞ΔͷͰͳʹΒͤͳ͍
σʔλͷҰ؏ੑ
DRBD
app%3#%XSJUFSFRVFTUQMBJOUFYUQSPUPDPM SFBESFRVFTU)551carbon-cache graphite-webprimarywhispercarbon-cache graphite-websecondarywhisper
ఆظతͳόοΫΞοϓ• Ұ؏ੑͪΌΜͱ͢ΔͳΒߋ৽ΛͱΊΔΈ͕ผ్ඞཁ (stop slave IO_THREAD;ͱ͔ͳ͍)• ߋ৽ͱΊͯΔؒʹ rsync / nc• ϝοηʔδΩϡʔతͳͷʹஷΊ͜Ή• ΦϯϥΠϯͰόοΫΞοϓͨ͠ͱͯ͠Ұఆظؒޙwhisper͕aggregate͢Δ
ͦͷଞ• ҙ֎ͱηοτΞοϓ໘• தಁ͚ͯΔʢPythonʣ• whisper ͷσʔλߏI/Oޮѱ͍• MySQL ͱ͔͑Δ͚ͲϢʔβใͱ͔͍ΕΔ͚ͩ
ȦȀȀ✣Ǐ
https://mackerel.io“৽͍͠ΞϓϦέʔγϣϯύϑΥʔϚϯεϚωδϝϯτ”
ϞχλϦϯά͕͖ͳΤϯδχΞΛืू͓ͯ͠Γ·͢