Using Thanos as a long-term storage for your Prometheus metrics Moto Ishizawa Z Lab Corporation

Moto Ishizawa @summerwind Infrastructure Engineer, Z Lab

1SPNFUIFVT

എܠ wҎ্ͷ,VCFSOFUFTΫϥελʔ͕Քಇத w֤,VCFSOFUFTΫϥελʔͰ͸ઐ༻ͷ1SPNFUIFVT͕ىಈ͍ͯ͠Δ w֤,VCFSOFUFTΫϥελʔ͸ӬଓετϨʔδΛ࣋ͨͳ͍ w֤,VCFSOFUFTΫϥελʔͷϊʔυ͸ϝϯςφϯεͷͨΊʹ਺िؒ͝ͱʹ࠶࡞੒͞ΕΔ wύϒϦοΫΫϥ΢υ΍4BB4͸࢖༻͍ͯ͠ͳ͍

՝୊ w֤Ϋϥελʔͷ1SPNFUIFVTͷϝτϦΫε͕ϊʔυͷ࠶࡞੒࣌ʹࣦΘΕͯ͠·͏ w؅ཧऀ΍։ൃऀ͸୹ظؒͷϝτϦΫε͔͠ࢀরͰ͖ͳ͍ w֤ΫϥελʔͰ1SPNFUIFVTΛ࣮ߦ͢Δͷʹඞཁे෼ͳ3".Λ֬อ͠ͳ͚Ε͹ͳΒͳ͍

w؅ཧऀ΍։ൃऀ͕೔Λ௒͑Δ௕ظؒͷϝτϦΫεΛࢀরͰ͖Δ wશͯͷϝτϦΫεΛ"NB[PO4ͷΑ͏ͳ֎෦ͷΦϒδΣΫτετϨʔδʹอଘ͢Δ w֤Ϋϥελʔʹ͍Δ1SPNFUIFVTͷ3".ͷফඅྔΛ࠷খݶʹ͢Δ ΰʔϧ

Prometheus ΦϒδΣΫτετϨʔδ ؅ཧऀ ϝτϦΫεͷอଘ ϝτϦΫεͷࢀর wΫϥελʔͷ1SPNFUIFVT͸୹ظؒͷϝτϦΫε͚ͩΛอ࣋͢Δ w1SPNFUIFVT͕ऩूͨ͠ϝτϦΫε͸શͯΦϒδΣΫτετϨʔδʹอଘ͢Δ w؅ཧऀ΍։ൃऀ͸ΦϒδΣΫτετϨʔδʹอଘ͞ΕͨϝτϦΫεΛࢀরͰ͖Δ ͜Μͳ࢓૊Έ͕͋ΔͱΑͦ͞͏

࣮૷Ҋ$PSUFYͷར༻ wεέʔϥϒϧͳ1SPNFUIFVTޓ׵γεςϜͷઌۦ͚ wΦϒδΣΫτετϨʔδͰ͸ͳ͘σʔλετΞʹϝτϦΫεΛอଘ͢Δ %ZOBNP%#ͳͲ w8FBWF$MPVE΍(SBGBOB$MPVEʹ͓͍ͯӡ༻࣮੷͕͋Δ

Prometheus ಠࣗΞμϓλʔ ؅ཧऀ ΦϒδΣΫτετϨʔδ Remote Read/Write API ϝτϦΫεΛࢀর ϝτϦΫεΛసૹ ࣮૷ҊಠࣗΞμϓλʔͷ։ൃ w1SPNFUIFVTͱΦϒδΣΫτετϨʔδΛܨ͙͚ͩͷ୯७ͳΞμϓλʔΛ։ൃ w1SPNFUIFVTͷ3FNPUF3FBE8SJUF"1*ͱ࣮ͯ͠૷ w1SPNFUIFVTͷ54%#ΛGPSLͯ͠ɺΦϒδΣΫτετϨʔδʹDIVOLΛอଘ͢ΔΑ͏मਖ਼ w؅ཧऀ΍։ൃऀ͔Β͸1SPNFUIFVTͷετϨʔδ͚͕ͩେ͖͘ͳͬͨΑ͏ʹݟ͑Δ wϝτϦΫεͷࢀরʹඞཁͳͷͰ1SPNFUIFVTʹ͸े෼ͳ3".͕·ͩඞཁ

3FNPUF3FBE8SJUF"1* w1SPNFUIFVT͕֎෦ͷετϨʔδͱ࿈ܞͯ͠ϝτϦΫεΛಡΈॻ͖͢Δ࢓૊Έ w1SPNFUIFVTͷઃఆϑΝΠϧʹ"1*Λ࣮૷ͨ͠ΤϯυϙΠϯτΛࢦఆ͢Δ͚ͩͰ࢖͑Δ w$PSUFY΍*OqVY%#ͳͲɺଟ͘ͷ࣮૷͕͋Δ # Remote Write API ͷΤϯυϙΠϯτΛࢦఆ remote_write: - url: http://127.0.0.1:9091/write # Remote Read API ͷΤϯυϙΠϯτΛࢦఆ remote_read: - url: http://127.0.0.1:9091/read Prometheus ͷઃఆϑΝΠϧ

3FNPUF3FBE8SJUF"1*ͷ࣮૷ w3FBEͱ8SJUFͷΤϯυϙΠϯτΛͭͣͭ༻ҙ͢Δ͚ͩͷγϯϓϧͳ࣮૷ w1SPUPDPM#V⒎FSTͰΤϯίʔυ͞ΕͨϝτϦΫεΛ4OBQQZͰѹॖ͠ɺ)551Ͱసૹ͢Δ w1SPNFUIFVTϦϙδτϦʹ͋Δαϯϓϧ࣮૷<>͕ࢀߟʹͳΔ wυΩϡϝϯτʹ͸ʮࠓޙϓϩτίϧ͕มΘΔՄೳੑ͕͋Δʯͱॻ͔Ε͍ͯΔ H31$ <> [1]: https://github.com/prometheus/prometheus/tree/master/documentation/examples/remote_storage [2]: https://prometheus.io/docs/prometheus/latest/storage/

3FNPUF8SJUF"1*ར༻࣌ͷ஫ҙ఺ w1SPNFUIFVTʹ3FNPUF8SJUF"1*ͷΤϯυϙΠϯτΛෳ਺ࢦఆ͢Δͱʜ wશͯͷΤϯυϙΠϯτʹಉ͡ϝτϦΫε͕సૹ͞ΕΔ 'BO0VUʹͳΔ <> wϥ΢ϯυϩϏϯͰ͸ͳ͍ͷͰɺෛՙ෼ࢄʹ͸࢖͑ͳ͍ w1SPNFUIFVTWҎલΛར༻࣌ʹ3FNPUF8SJUF"1*ͷΤϯυϙΠϯτ͕μ΢ϯ͢Δͱʜ w1SPNFUIFVT͕సૹͷϦτϥΠΛ܁Γฦ͢ͷͰΤϯυϙΠϯτ෮ؼ࣌ͷෛՙ͕໰୊ʹ wసૹϦτϥΠதͷϝτϦΫε͸ϝϞϦʔʹอ࣋͞ΕΔͷͰɺࣦΘΕΔϦεΫ͕͋Δ w1SPNFUIFVTW͔Βసૹࣦഊͨ͠ϝτϦΫε΋8"-ʹอଘ͞ΕΔΑ͏ʹͳͬͨ<> wݱࡏ͸3FNPUF8SJUF"1*ར༻ऀ޲͚ͷνϡʔχϯάυΩϡϝϯτ΋ެ։͞Ε͍ͯΔ<> [1]: https://github.com/prometheus/prometheus/blob/master/documentation/internal_architecture.md [2]: https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/ [3]: https://prometheus.io/docs/practices/remote_write/

࣮૷Ҋ5IBOPT wΦϒδΣΫτετϨʔδʹϝτϦΫεΛอଘ͢Δ1SPNFUIFVTޓ׵ͷγεςϜ w1SPNFUIFVTͷ֤ػೳΛίϯϙʔωϯτʹ෼ׂͨ͠Α͏ͳߏ੒ʹͳ͍ͬͯΔ wඞཁͳίϯϙʔωϯτ͚ͩΛબΜͰ࢖͏͜ͱ΋Ͱ͖Δ w5IBOPTW͔Β3FNPUF8SJUF"1*ʹΑΔϝτϦΫεͷड৴͕Մೳʹ

5IBOPTͷίϯϙʔωϯτ

5IBOPTͷίϯϙʔωϯτ 4JEFDBS 1SPNFUIFVTͷσʔλσΟϨΫτϦΛ؂ࢹͯ͠ɺΦϒδΣΫτετϨʔδʹϝτϦΫεΛΞοϓϩʔυ͢ Δɻ1SPNFUIFVTͱͷؔ܎ͰσϓϩΠ͢Δɻ 4UPSF ΦϒδΣΫτετϨʔδʹอଘ͞ΕͨϝτϦΫεʹΞΫηε͢ΔͨΊͷ"1*ɻ௨ৗ͸2VFSZͱ૊Έ߹Θͤ ͯ࢖͏ɻ 2VFSZ 1SPN2-Λ࢖༻ͯ͠ϝτϦΫεΛूܭ͢Δɻ1SPNFUIFVTޓ׵ͷ6*ͱ"1*Λඋ͍͑ͯΔͨΊɺ 1SPNFUIFVTͱಉ͡Α͏ʹ࢖͑Δɻ 3VMF 1SPNFUIFVTͷϧʔϧϑΝΠϧΛಡΈࠐΈɺΞϥʔτΛੜ੒͢ΔɻΦϒδΣΫτετϨʔδʹอଘ͞Εͨ ϝτϦΫεશମʹϧʔϧΛద༻͍ͨ͠৔߹ͳͲʹ࢖༻ɻΞϥʔτͷૹ৴͸"MFSUNBOBHFSΛ࢖͏ɻ $PNQBDU ΦϒδΣΫτετϨʔδʹอଘ͞ΕͨϝτϦΫεΛఆظతʹू໿ͨ͠Γμ΢ϯαϯϓϦϯά͢Δɻϲ݄ܦ աͨ͠ϝτϦΫεΛ෼·ͨ͸࣌ؒ͝ͱʹαϯϓϦϯά͠ͳ͓͢ɺͱ͍ͬͨ͜ͱ͕Ͱ͖Δɻ 3FDFJWF 3FNPUF8SJUF"1*Λඋ͓͑ͯΓɺ֎෦͔ΒϝτϦΫεΛड͚෇͚ΔɻૹΒΕ͖ͯͨϝτϦΫε͸ 4JEFDBSͱಉ༷ʹΦϒδΣΫτετϨʔδʹอଘ͢Δɻ·࣮ͩݧతͳѻ͍ɻ

5IBOPT3FDFJWFΛར༻ͨ͠ߏ੒ w5IBOPT3FDFJWFΛར༻ͯ͠ɺಠࣗΞμϓλʔΛஔ͖׵͑ w؅ཧऀ΍։ൃऀ͸5IBOPT2VFSZΛ௨ͯ͡ϝτϦΫεΛࢀর͢Δ Prometheus Thanos Receive ؅ཧऀ ΦϒδΣΫτετϨʔδ Remote Write API ϝτϦΫεΛࢀর ϝτϦΫεΛอଘ Thanos Store Thanos Query ϝτϦΫεͷऔಘ ϝτϦΫεͷूܭ

5IBOPT3FDFJWFͷ৑௕Խ wෳ਺ͷ5IBOPT3FDFJWFΛฒ΂ͯϩʔυόϥϯαʔΛ഑ஔ͢Ε͹ෛՙ෼ࢄ͕Ͱ͖Δ wૹΒΕ͖ͯͨϝτϦΫεͷϥϕϧηοτͳͲ͔Β୲౰͢Δ5IBOPT3FDFJWF͕ܾ·Δ w୲౰Ͱͳ͍ϝτϦΫεσʔλΛड৴ͨ͠৔߹͸୲౰͢Δ5IBOPT3FDFJWFʹసૹ͢Δ Prometheus Thanos Receive #3 ϩʔυόϥϯαʔ Remote Write API ϦΫΤετૹ৴ ϦΫΤετసૹ Thanos Receive #2 Thanos Receive #1 Thanos Receive #2 ͕୲౰͢ΔϝτϦΫε Thanos Receive #1 ͕୲౰͢ΔϝτϦΫε

5IBOPT3FDFJWFʹΑΔϝτϦΫεͷϨϓϦέʔγϣϯ wෳ਺ͷ5IBOPT3FDFJWFͰಉ͡ϝτϦΫεΛอଘ͢Δ w3FQMJDBUJPO'BDUPSͷ஋͔ΒɺԿ୆ͷ5IBOPT3FDFJWF΁ͷॻ͖ࠐΈ͕ඞཁ͔Λܾఆ w 3FQMJDBUJPO'BDUPS ୆ͷ5IBOPT3FDFJWF΁ͷॻ͖ࠐΈʹࣦഊ͢ΔͱΤϥʔ Prometheus Thanos Receive #3 ϩʔυόϥϯαʔ Remote Write API ϦΫΤετૹ৴ ϨϓϦέʔγϣϯ Thanos Receive #2 Thanos Receive #1

5IBOPT3FDFJWFͷ࣮ߦ $ thanos receive \ --tsdb.path /var/lib/thanos \ # TSDB ϑΝΠϧͷύε --tsdb.retention 3h \ # TSDB ϑΝΠϧͷอ࣋ظݶ --objstore.config-file /etc/thanos/bucket_config.yaml \ # ΦϒδΣΫτετϨʔδͷઃఆ --receive.local-endpoint http://thanos1.local:19291/api/v1/receive \ # ࣗ਎ͷΤϯυϙΠϯτ໊ --receive.hashrings-file /etc/thanos/hashring.json \ # ৑௕ԽͷͨΊͷϋογϡϦϯάͷઃఆ --receive.replication-factor 3 # Replication Factor ͷઃఆ --label "thanos='1'" ɹɹɹɹɹɹ# Thanos ͷ External Label [ { "hashring": "cluster", "endpoints": [ "http://thanos1.local:19291/api/v1/receive", "http://thanos2.local:19291/api/v1/receive", "http://thanos3.local:19291/api/v1/receive" ] } ] /etc/thanos/hashring.json

5IBOPT3FDFJWFར༻࣌ͷ஫ҙ఺ w࣮ݧతͳѻ͍ͷίϯϙʔωϯτͰ͋Γɺ࢖͍ํʹؔ͢ΔυΩϡϝϯτ͕ͳ͍ wػೳ΍࢓༷͸1SPQPTBMυΩϡϝϯτͰ֬ೝͰ͖Δ<> wڍಈͷࡉ͔͍෦෼͸ιʔείʔυΛ௥͍͔͚͍ͯ͘ඞཁ͕͋Δ wthanos receiveίϚϯυͷιʔείʔυ͔Β௥͍͔͚͍ͯ͘ͱޮ཰͕Α͍ w--label ΦϓγϣϯͷࢦఆΛ͠ͳ͍ͱϝτϦΫεͷΞοϓϩʔυʹࣦഊ͢Δ w4JEFDBSͷ࣮૷Λྲྀ༻ͯ͠3FDFJWF͕࣮૷͞Εͨ͜ͱʹΑΔෆ۩߹ͱࢥΘΕΔ w4JEFDBSͰ͸1SPNFUIFVTͷ&YUFSOBM-BCFMͷઃఆΛ࢖༻͢Δ͕ɺ3FDFJWFʹ͸Ͱ͖ͳ͍ wద౰ͳϥϕϧΛࢦఆ͢Δ͜ͱͰͻͱ·ͣճආ͸Մೳ [1]: https://thanos.io/proposals/201812_thanos-remote-receive.md/

ݱঢ়ͱࠓޙͷ՝୊ wγϯϓϧͳ؀ڥʹ͓͍ͯ͸ɺઃఆͨ͠ΰʔϧ͸΄΅ୡ੒Ͱ͖͍ͯΔ wΑΓଟ͘ͷ1SPNFUIFVT͔ΒϝτϦΫεΛड৴͢Δʹ͸ҎԼͷΑ͏ͳରԠ͕ඞཁ wෳ਺ͷϝτϦΫεͷࠞࡏΛ๷͙࢓૊Έͷ࣮૷ w5IBOPT3FDFJWFͷύϑΥʔϚϯεͷݕূ w5IBOPT3FDFJWFͷ৑௕Խػೳͷར༻

Thank you !