Share
Embed
Transcript
22 slides
-
Using Thanos as a long-term storage for your Prometheus metrics
Moto Ishizawa Z Lab Corporation -
Prometheus ಠࣗΞμϓλʔ ཧऀ ΦϒδΣΫτετϨʔδ Remote Read/Write API ϝτϦΫεΛࢀর ϝτϦΫεΛసૹ ࣮ҊಠࣗΞμϓλʔͷ։ൃ
w1SPNFUIFVTͱΦϒδΣΫτετϨʔδΛܨ͙͚ͩͷ୯७ͳΞμϓλʔΛ։ൃ w1SPNFUIFVTͷ3FNPUF3FBE8SJUF"1*ͱ࣮ͯ͠ w1SPNFUIFVTͷ54%#ΛGPSLͯ͠ɺΦϒδΣΫτετϨʔδʹDIVOLΛอଘ͢ΔΑ͏मਖ਼ wཧऀ։ൃऀ͔Β1SPNFUIFVTͷετϨʔδ͚͕ͩେ͖͘ͳͬͨΑ͏ʹݟ͑Δ wϝτϦΫεͷࢀরʹඞཁͳͷͰ1SPNFUIFVTʹेͳ3".͕·ͩඞཁ -
3FNPUF3FBE8SJUF"1* w1SPNFUIFVT͕֎෦ͷετϨʔδͱ࿈ܞͯ͠ϝτϦΫεΛಡΈॻ͖͢ΔΈ w1SPNFUIFVTͷઃఆϑΝΠϧʹ"1*Λ࣮ͨ͠ΤϯυϙΠϯτΛࢦఆ͢Δ͚ͩͰ͑Δ w$PSUFY*OqVY%#ͳͲɺଟ͘ͷ࣮͕͋Δ # Remote Write API ͷΤϯυϙΠϯτΛࢦఆ remote_write:
- url: http://127.0.0.1:9091/write # Remote Read API ͷΤϯυϙΠϯτΛࢦఆ remote_read: - url: http://127.0.0.1:9091/read Prometheus ͷઃఆϑΝΠϧ -
3FNPUF3FBE8SJUF"1*ͷ࣮ w3FBEͱ8SJUFͷΤϯυϙΠϯτΛͭͣͭ༻ҙ͢Δ͚ͩͷγϯϓϧͳ࣮ w1SPUPDPM#V⒎FSTͰΤϯίʔυ͞ΕͨϝτϦΫεΛ4OBQQZͰѹॖ͠ɺ)551Ͱసૹ͢Δ w1SPNFUIFVTϦϙδτϦʹ͋Δαϯϓϧ࣮<>͕ࢀߟʹͳΔ wυΩϡϝϯτʹʮࠓޙϓϩτίϧ͕มΘΔՄೳੑ͕͋Δʯͱॻ͔Ε͍ͯΔ H31$ <> [1]: https://github.com/prometheus/prometheus/tree/master/documentation/examples/remote_storage [2]:
https://prometheus.io/docs/prometheus/latest/storage/ -
3FNPUF8SJUF"1*ར༻࣌ͷҙ w1SPNFUIFVTʹ3FNPUF8SJUF"1*ͷΤϯυϙΠϯτΛෳࢦఆ͢Δͱʜ wશͯͷΤϯυϙΠϯτʹಉ͡ϝτϦΫε͕సૹ͞ΕΔ 'BO0VUʹͳΔ <> wϥϯυϩϏϯͰͳ͍ͷͰɺෛՙࢄʹ͑ͳ͍ w1SPNFUIFVTWҎલΛར༻࣌ʹ3FNPUF8SJUF"1*ͷΤϯυϙΠϯτ͕μϯ͢Δͱʜ w1SPNFUIFVT͕సૹͷϦτϥΠΛ܁Γฦ͢ͷͰΤϯυϙΠϯτ෮ؼ࣌ͷෛՙ͕ʹ wసૹϦτϥΠதͷϝτϦΫεϝϞϦʔʹอ࣋͞ΕΔͷͰɺࣦΘΕΔϦεΫ͕͋Δ w1SPNFUIFVTW͔Βసૹࣦഊͨ͠ϝτϦΫε8"-ʹอଘ͞ΕΔΑ͏ʹͳͬͨ<>
wݱࡏ3FNPUF8SJUF"1*ར༻ऀ͚ͷνϡʔχϯάυΩϡϝϯτެ։͞Ε͍ͯΔ<> [1]: https://github.com/prometheus/prometheus/blob/master/documentation/internal_architecture.md [2]: https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/ [3]: https://prometheus.io/docs/practices/remote_write/ -
5IBOPTͷίϯϙʔωϯτ 4JEFDBS 1SPNFUIFVTͷσʔλσΟϨΫτϦΛࢹͯ͠ɺΦϒδΣΫτετϨʔδʹϝτϦΫεΛΞοϓϩʔυ͢ Δɻ1SPNFUIFVTͱͷؔͰσϓϩΠ͢Δɻ 4UPSF ΦϒδΣΫτετϨʔδʹอଘ͞ΕͨϝτϦΫεʹΞΫηε͢ΔͨΊͷ"1*ɻ௨ৗ2VFSZͱΈ߹Θͤ ͯ͏ɻ 2VFSZ 1SPN2-Λ༻ͯ͠ϝτϦΫεΛूܭ͢Δɻ1SPNFUIFVTޓͷ6*ͱ"1*Λඋ͍͑ͯΔͨΊɺ 1SPNFUIFVTͱಉ͡Α͏ʹ͑Δɻ
3VMF 1SPNFUIFVTͷϧʔϧϑΝΠϧΛಡΈࠐΈɺΞϥʔτΛੜ͢ΔɻΦϒδΣΫτετϨʔδʹอଘ͞Εͨ ϝτϦΫεશମʹϧʔϧΛద༻͍ͨ͠߹ͳͲʹ༻ɻΞϥʔτͷૹ৴"MFSUNBOBHFSΛ͏ɻ $PNQBDU ΦϒδΣΫτετϨʔδʹอଘ͞ΕͨϝτϦΫεΛఆظతʹूͨ͠ΓμϯαϯϓϦϯά͢Δɻϲ݄ܦ աͨ͠ϝτϦΫεΛ·ͨ࣌ؒ͝ͱʹαϯϓϦϯά͠ͳ͓͢ɺͱ͍ͬͨ͜ͱ͕Ͱ͖Δɻ 3FDFJWF 3FNPUF8SJUF"1*Λඋ͓͑ͯΓɺ֎෦͔ΒϝτϦΫεΛड͚͚ΔɻૹΒΕ͖ͯͨϝτϦΫε 4JEFDBSͱಉ༷ʹΦϒδΣΫτετϨʔδʹอଘ͢Δɻ·࣮ͩݧతͳѻ͍ɻ -
5IBOPT3FDFJWFΛར༻ͨ͠ߏ w5IBOPT3FDFJWFΛར༻ͯ͠ɺಠࣗΞμϓλʔΛஔ͖͑ wཧऀ։ൃऀ5IBOPT2VFSZΛ௨ͯ͡ϝτϦΫεΛࢀর͢Δ Prometheus Thanos Receive ཧऀ ΦϒδΣΫτετϨʔδ Remote Write
API ϝτϦΫεΛࢀর ϝτϦΫεΛอଘ Thanos Store Thanos Query ϝτϦΫεͷऔಘ ϝτϦΫεͷूܭ -
5IBOPT3FDFJWFͷԽ wෳͷ5IBOPT3FDFJWFΛฒͯϩʔυόϥϯαʔΛஔ͢Εෛՙࢄ͕Ͱ͖Δ wૹΒΕ͖ͯͨϝτϦΫεͷϥϕϧηοτͳͲ͔Β୲͢Δ5IBOPT3FDFJWF͕ܾ·Δ w୲Ͱͳ͍ϝτϦΫεσʔλΛड৴ͨ͠߹୲͢Δ5IBOPT3FDFJWFʹసૹ͢Δ Prometheus Thanos Receive #3 ϩʔυόϥϯαʔ Remote
Write API ϦΫΤετૹ৴ ϦΫΤετసૹ Thanos Receive #2 Thanos Receive #1 Thanos Receive #2 ͕୲͢ΔϝτϦΫε Thanos Receive #1 ͕୲͢ΔϝτϦΫε -
5IBOPT3FDFJWFʹΑΔϝτϦΫεͷϨϓϦέʔγϣϯ wෳͷ5IBOPT3FDFJWFͰಉ͡ϝτϦΫεΛอଘ͢Δ w3FQMJDBUJPO'BDUPSͷ͔ΒɺԿͷ5IBOPT3FDFJWFͷॻ͖ࠐΈ͕ඞཁ͔Λܾఆ w 3FQMJDBUJPO'BDUPS ͷ5IBOPT3FDFJWFͷॻ͖ࠐΈʹࣦഊ͢ΔͱΤϥʔ Prometheus Thanos Receive
#3 ϩʔυόϥϯαʔ Remote Write API ϦΫΤετૹ৴ ϨϓϦέʔγϣϯ Thanos Receive #2 Thanos Receive #1 -
5IBOPT3FDFJWFͷ࣮ߦ $ thanos receive \ --tsdb.path /var/lib/thanos \ # TSDB
ϑΝΠϧͷύε --tsdb.retention 3h \ # TSDB ϑΝΠϧͷอ࣋ظݶ --objstore.config-file /etc/thanos/bucket_config.yaml \ # ΦϒδΣΫτετϨʔδͷઃఆ --receive.local-endpoint http://thanos1.local:19291/api/v1/receive \ # ࣗͷΤϯυϙΠϯτ໊ --receive.hashrings-file /etc/thanos/hashring.json \ # ԽͷͨΊͷϋογϡϦϯάͷઃఆ --receive.replication-factor 3 # Replication Factor ͷઃఆ --label "thanos='1'" ɹɹɹɹɹɹ# Thanos ͷ External Label [ { "hashring": "cluster", "endpoints": [ "http://thanos1.local:19291/api/v1/receive", "http://thanos2.local:19291/api/v1/receive", "http://thanos3.local:19291/api/v1/receive" ] } ] /etc/thanos/hashring.json -
5IBOPT3FDFJWFར༻࣌ͷҙ w࣮ݧతͳѻ͍ͷίϯϙʔωϯτͰ͋Γɺ͍ํʹؔ͢ΔυΩϡϝϯτ͕ͳ͍ wػೳ༷1SPQPTBMυΩϡϝϯτͰ֬ೝͰ͖Δ<> wڍಈͷࡉ͔͍෦ιʔείʔυΛ͍͔͚͍ͯ͘ඞཁ͕͋Δ wthanos receiveίϚϯυͷιʔείʔυ͔Β͍͔͚͍ͯ͘ͱޮ͕Α͍ w--label ΦϓγϣϯͷࢦఆΛ͠ͳ͍ͱϝτϦΫεͷΞοϓϩʔυʹࣦഊ͢Δ w4JEFDBSͷ࣮Λྲྀ༻ͯ͠3FDFJWF͕࣮͞Εͨ͜ͱʹΑΔෆ۩߹ͱࢥΘΕΔ w4JEFDBSͰ1SPNFUIFVTͷ&YUFSOBM-BCFMͷઃఆΛ༻͢Δ͕ɺ3FDFJWFʹͰ͖ͳ͍
wదͳϥϕϧΛࢦఆ͢Δ͜ͱͰͻͱ·ͣճආՄೳ [1]: https://thanos.io/proposals/201812_thanos-remote-receive.md/