$30 off During Our Annual Pro Sale. View Details »

Using Thanos as a long-term storage for your Prometheus metrics

Using Thanos as a long-term storage for your Prometheus metrics

Moto Ishizawa

January 15, 2020
Tweet

More Decks by Moto Ishizawa

Other Decks in Technology

Transcript

  1. Using Thanos as a long-term storage for your Prometheus metrics
    Moto Ishizawa
    Z Lab Corporation

    View Slide

  2. Moto Ishizawa
    @summerwind
    Infrastructure Engineer, Z Lab

    View Slide

  3. 1SPNFUIFVT

    View Slide

  4. എܠ
    wҎ্ͷ,VCFSOFUFTΫϥελʔ͕Քಇத
    w֤,VCFSOFUFTΫϥελʔͰ͸ઐ༻ͷ1SPNFUIFVT͕ىಈ͍ͯ͠Δ
    w֤,VCFSOFUFTΫϥελʔ͸ӬଓετϨʔδΛ࣋ͨͳ͍
    w֤,VCFSOFUFTΫϥελʔͷϊʔυ͸ϝϯςφϯεͷͨΊʹ਺िؒ͝ͱʹ࠶࡞੒͞ΕΔ
    wύϒϦοΫΫϥ΢υ΍4BB4͸࢖༻͍ͯ͠ͳ͍

    View Slide

  5. ՝୊
    w֤Ϋϥελʔͷ1SPNFUIFVTͷϝτϦΫε͕ϊʔυͷ࠶࡞੒࣌ʹࣦΘΕͯ͠·͏
    w؅ཧऀ΍։ൃऀ͸୹ظؒͷϝτϦΫε͔͠ࢀরͰ͖ͳ͍
    w֤ΫϥελʔͰ1SPNFUIFVTΛ࣮ߦ͢Δͷʹඞཁे෼ͳ3".Λ֬อ͠ͳ͚Ε͹ͳΒͳ͍

    View Slide

  6. w؅ཧऀ΍։ൃऀ͕೔Λ௒͑Δ௕ظؒͷϝτϦΫεΛࢀরͰ͖Δ
    wશͯͷϝτϦΫεΛ"NB[PO4ͷΑ͏ͳ֎෦ͷΦϒδΣΫτετϨʔδʹอଘ͢Δ
    w֤Ϋϥελʔʹ͍Δ1SPNFUIFVTͷ3".ͷফඅྔΛ࠷খݶʹ͢Δ
    ΰʔϧ

    View Slide



  7. Prometheus ΦϒδΣΫτετϨʔδ ؅ཧऀ
    ϝτϦΫεͷอଘ ϝτϦΫεͷࢀর
    wΫϥελʔͷ1SPNFUIFVT͸୹ظؒͷϝτϦΫε͚ͩΛอ࣋͢Δ
    w1SPNFUIFVT͕ऩूͨ͠ϝτϦΫε͸શͯΦϒδΣΫτετϨʔδʹอଘ͢Δ
    w؅ཧऀ΍։ൃऀ͸ΦϒδΣΫτετϨʔδʹอଘ͞ΕͨϝτϦΫεΛࢀরͰ͖Δ
    ͜Μͳ࢓૊Έ͕͋ΔͱΑͦ͞͏

    View Slide

  8. ࣮૷Ҋ$PSUFYͷར༻
    wεέʔϥϒϧͳ1SPNFUIFVTޓ׵γεςϜͷઌۦ͚
    wΦϒδΣΫτετϨʔδͰ͸ͳ͘σʔλετΞʹϝτϦΫεΛอଘ͢Δ %ZOBNP%#ͳͲ

    w8FBWF$MPVE΍(SBGBOB$MPVEʹ͓͍ͯӡ༻࣮੷͕͋Δ

    View Slide



  9. Prometheus ಠࣗΞμϓλʔ
    ؅ཧऀ ΦϒδΣΫτετϨʔδ
    Remote
    Read/Write API
    ϝτϦΫεΛࢀর ϝτϦΫεΛసૹ
    ࣮૷ҊಠࣗΞμϓλʔͷ։ൃ
    w1SPNFUIFVTͱΦϒδΣΫτετϨʔδΛܨ͙͚ͩͷ୯७ͳΞμϓλʔΛ։ൃ
    w1SPNFUIFVTͷ3FNPUF3FBE8SJUF"1*ͱ࣮ͯ͠૷
    w1SPNFUIFVTͷ54%#ΛGPSLͯ͠ɺΦϒδΣΫτετϨʔδʹDIVOLΛอଘ͢ΔΑ͏मਖ਼
    w؅ཧऀ΍։ൃऀ͔Β͸1SPNFUIFVTͷετϨʔδ͚͕ͩେ͖͘ͳͬͨΑ͏ʹݟ͑Δ
    wϝτϦΫεͷࢀরʹඞཁͳͷͰ1SPNFUIFVTʹ͸े෼ͳ3".͕·ͩඞཁ

    View Slide

  10. 3FNPUF3FBE8SJUF"1*
    w1SPNFUIFVT͕֎෦ͷετϨʔδͱ࿈ܞͯ͠ϝτϦΫεΛಡΈॻ͖͢Δ࢓૊Έ
    w1SPNFUIFVTͷઃఆϑΝΠϧʹ"1*Λ࣮૷ͨ͠ΤϯυϙΠϯτΛࢦఆ͢Δ͚ͩͰ࢖͑Δ
    w$PSUFY΍*OqVY%#ͳͲɺଟ͘ͷ࣮૷͕͋Δ
    # Remote Write API ͷΤϯυϙΠϯτΛࢦఆ
    remote_write:
    - url: http://127.0.0.1:9091/write
    # Remote Read API ͷΤϯυϙΠϯτΛࢦఆ
    remote_read:
    - url: http://127.0.0.1:9091/read
    Prometheus ͷઃఆϑΝΠϧ

    View Slide

  11. 3FNPUF3FBE8SJUF"1*ͷ࣮૷
    w3FBEͱ8SJUFͷΤϯυϙΠϯτΛͭͣͭ༻ҙ͢Δ͚ͩͷγϯϓϧͳ࣮૷
    w1SPUPDPM#V⒎FSTͰΤϯίʔυ͞ΕͨϝτϦΫεΛ4OBQQZͰѹॖ͠ɺ)551Ͱసૹ͢Δ
    w1SPNFUIFVTϦϙδτϦʹ͋Δαϯϓϧ࣮૷<>͕ࢀߟʹͳΔ
    wυΩϡϝϯτʹ͸ʮࠓޙϓϩτίϧ͕มΘΔՄೳੑ͕͋Δʯͱॻ͔Ε͍ͯΔ H31$
    <>
    [1]: https://github.com/prometheus/prometheus/tree/master/documentation/examples/remote_storage
    [2]: https://prometheus.io/docs/prometheus/latest/storage/

    View Slide

  12. 3FNPUF8SJUF"1*ར༻࣌ͷ஫ҙ఺
    w1SPNFUIFVTʹ3FNPUF8SJUF"1*ͷΤϯυϙΠϯτΛෳ਺ࢦఆ͢Δͱʜ
    wશͯͷΤϯυϙΠϯτʹಉ͡ϝτϦΫε͕సૹ͞ΕΔ 'BO0VUʹͳΔ
    <>
    wϥ΢ϯυϩϏϯͰ͸ͳ͍ͷͰɺෛՙ෼ࢄʹ͸࢖͑ͳ͍
    w1SPNFUIFVTWҎલΛར༻࣌ʹ3FNPUF8SJUF"1*ͷΤϯυϙΠϯτ͕μ΢ϯ͢Δͱʜ
    w1SPNFUIFVT͕సૹͷϦτϥΠΛ܁Γฦ͢ͷͰΤϯυϙΠϯτ෮ؼ࣌ͷෛՙ͕໰୊ʹ
    wసૹϦτϥΠதͷϝτϦΫε͸ϝϞϦʔʹอ࣋͞ΕΔͷͰɺࣦΘΕΔϦεΫ͕͋Δ
    w1SPNFUIFVTW͔Βసૹࣦഊͨ͠ϝτϦΫε΋8"-ʹอଘ͞ΕΔΑ͏ʹͳͬͨ<>
    wݱࡏ͸3FNPUF8SJUF"1*ར༻ऀ޲͚ͷνϡʔχϯάυΩϡϝϯτ΋ެ։͞Ε͍ͯΔ<>
    [1]: https://github.com/prometheus/prometheus/blob/master/documentation/internal_architecture.md
    [2]: https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/
    [3]: https://prometheus.io/docs/practices/remote_write/

    View Slide

  13. ࣮૷Ҋ5IBOPT
    wΦϒδΣΫτετϨʔδʹϝτϦΫεΛอଘ͢Δ1SPNFUIFVTޓ׵ͷγεςϜ
    w1SPNFUIFVTͷ֤ػೳΛίϯϙʔωϯτʹ෼ׂͨ͠Α͏ͳߏ੒ʹͳ͍ͬͯΔ
    wඞཁͳίϯϙʔωϯτ͚ͩΛબΜͰ࢖͏͜ͱ΋Ͱ͖Δ
    w5IBOPTW͔Β3FNPUF8SJUF"1*ʹΑΔϝτϦΫεͷड৴͕Մೳʹ

    View Slide

  14. 5IBOPTͷίϯϙʔωϯτ

    View Slide

  15. 5IBOPTͷίϯϙʔωϯτ
    4JEFDBS
    1SPNFUIFVTͷσʔλσΟϨΫτϦΛ؂ࢹͯ͠ɺΦϒδΣΫτετϨʔδʹϝτϦΫεΛΞοϓϩʔυ͢
    Δɻ1SPNFUIFVTͱͷؔ܎ͰσϓϩΠ͢Δɻ
    4UPSF
    ΦϒδΣΫτετϨʔδʹอଘ͞ΕͨϝτϦΫεʹΞΫηε͢ΔͨΊͷ"1*ɻ௨ৗ͸2VFSZͱ૊Έ߹Θͤ
    ͯ࢖͏ɻ
    2VFSZ
    1SPN2-Λ࢖༻ͯ͠ϝτϦΫεΛूܭ͢Δɻ1SPNFUIFVTޓ׵ͷ6*ͱ"1*Λඋ͍͑ͯΔͨΊɺ
    1SPNFUIFVTͱಉ͡Α͏ʹ࢖͑Δɻ
    3VMF
    1SPNFUIFVTͷϧʔϧϑΝΠϧΛಡΈࠐΈɺΞϥʔτΛੜ੒͢ΔɻΦϒδΣΫτετϨʔδʹอଘ͞Εͨ
    ϝτϦΫεશମʹϧʔϧΛద༻͍ͨ͠৔߹ͳͲʹ࢖༻ɻΞϥʔτͷૹ৴͸"MFSUNBOBHFSΛ࢖͏ɻ
    $PNQBDU
    ΦϒδΣΫτετϨʔδʹอଘ͞ΕͨϝτϦΫεΛఆظతʹू໿ͨ͠Γμ΢ϯαϯϓϦϯά͢Δɻϲ݄ܦ
    աͨ͠ϝτϦΫεΛ෼·ͨ͸࣌ؒ͝ͱʹαϯϓϦϯά͠ͳ͓͢ɺͱ͍ͬͨ͜ͱ͕Ͱ͖Δɻ
    3FDFJWF
    3FNPUF8SJUF"1*Λඋ͓͑ͯΓɺ֎෦͔ΒϝτϦΫεΛड͚෇͚ΔɻૹΒΕ͖ͯͨϝτϦΫε͸
    4JEFDBSͱಉ༷ʹΦϒδΣΫτετϨʔδʹอଘ͢Δɻ·࣮ͩݧతͳѻ͍ɻ

    View Slide

  16. 5IBOPT3FDFJWFΛར༻ͨ͠ߏ੒
    w5IBOPT3FDFJWFΛར༻ͯ͠ɺಠࣗΞμϓλʔΛஔ͖׵͑
    w؅ཧऀ΍։ൃऀ͸5IBOPT2VFSZΛ௨ͯ͡ϝτϦΫεΛࢀর͢Δ


    Prometheus Thanos Receive
    ؅ཧऀ
    ΦϒδΣΫτετϨʔδ
    Remote Write API
    ϝτϦΫεΛࢀর
    ϝτϦΫεΛอଘ
    Thanos Store
    Thanos Query
    ϝτϦΫεͷऔಘ
    ϝτϦΫεͷूܭ

    View Slide

  17. 5IBOPT3FDFJWFͷ৑௕Խ
    wෳ਺ͷ5IBOPT3FDFJWFΛฒ΂ͯϩʔυόϥϯαʔΛ഑ஔ͢Ε͹ෛՙ෼ࢄ͕Ͱ͖Δ
    wૹΒΕ͖ͯͨϝτϦΫεͷϥϕϧηοτͳͲ͔Β୲౰͢Δ5IBOPT3FDFJWF͕ܾ·Δ
    w୲౰Ͱͳ͍ϝτϦΫεσʔλΛड৴ͨ͠৔߹͸୲౰͢Δ5IBOPT3FDFJWFʹసૹ͢Δ
    Prometheus
    Thanos Receive #3
    ϩʔυόϥϯαʔ
    Remote Write API ϦΫΤετૹ৴
    ϦΫΤετసૹ
    Thanos Receive #2
    Thanos Receive #1
    Thanos Receive #2 ͕୲౰͢ΔϝτϦΫε
    Thanos Receive #1 ͕୲౰͢ΔϝτϦΫε

    View Slide

  18. 5IBOPT3FDFJWFʹΑΔϝτϦΫεͷϨϓϦέʔγϣϯ
    wෳ਺ͷ5IBOPT3FDFJWFͰಉ͡ϝτϦΫεΛอଘ͢Δ
    w3FQMJDBUJPO'BDUPSͷ஋͔ΒɺԿ୆ͷ5IBOPT3FDFJWF΁ͷॻ͖ࠐΈ͕ඞཁ͔Λܾఆ
    w 3FQMJDBUJPO'BDUPS
    ୆ͷ5IBOPT3FDFJWF΁ͷॻ͖ࠐΈʹࣦഊ͢ΔͱΤϥʔ
    Prometheus
    Thanos Receive #3
    ϩʔυόϥϯαʔ
    Remote Write API ϦΫΤετૹ৴
    ϨϓϦέʔγϣϯ
    Thanos Receive #2
    Thanos Receive #1

    View Slide

  19. 5IBOPT3FDFJWFͷ࣮ߦ
    $ thanos receive \
    --tsdb.path /var/lib/thanos \ # TSDB ϑΝΠϧͷύε
    --tsdb.retention 3h \ # TSDB ϑΝΠϧͷอ࣋ظݶ
    --objstore.config-file /etc/thanos/bucket_config.yaml \ # ΦϒδΣΫτετϨʔδͷઃఆ
    --receive.local-endpoint http://thanos1.local:19291/api/v1/receive \ # ࣗ਎ͷΤϯυϙΠϯτ໊
    --receive.hashrings-file /etc/thanos/hashring.json \ # ৑௕ԽͷͨΊͷϋογϡϦϯάͷઃఆ
    --receive.replication-factor 3 # Replication Factor ͷઃఆ
    --label "thanos='1'" ɹɹɹɹɹɹ# Thanos ͷ External Label
    [
    {
    "hashring": "cluster",
    "endpoints": [
    "http://thanos1.local:19291/api/v1/receive",
    "http://thanos2.local:19291/api/v1/receive",
    "http://thanos3.local:19291/api/v1/receive"
    ]
    }
    ]
    /etc/thanos/hashring.json

    View Slide

  20. 5IBOPT3FDFJWFར༻࣌ͷ஫ҙ఺
    w࣮ݧతͳѻ͍ͷίϯϙʔωϯτͰ͋Γɺ࢖͍ํʹؔ͢ΔυΩϡϝϯτ͕ͳ͍
    wػೳ΍࢓༷͸1SPQPTBMυΩϡϝϯτͰ֬ೝͰ͖Δ<>
    wڍಈͷࡉ͔͍෦෼͸ιʔείʔυΛ௥͍͔͚͍ͯ͘ඞཁ͕͋Δ
    wthanos receiveίϚϯυͷιʔείʔυ͔Β௥͍͔͚͍ͯ͘ͱޮ཰͕Α͍
    w--label ΦϓγϣϯͷࢦఆΛ͠ͳ͍ͱϝτϦΫεͷΞοϓϩʔυʹࣦഊ͢Δ
    w4JEFDBSͷ࣮૷Λྲྀ༻ͯ͠3FDFJWF͕࣮૷͞Εͨ͜ͱʹΑΔෆ۩߹ͱࢥΘΕΔ
    w4JEFDBSͰ͸1SPNFUIFVTͷ&YUFSOBM-BCFMͷઃఆΛ࢖༻͢Δ͕ɺ3FDFJWFʹ͸Ͱ͖ͳ͍
    wద౰ͳϥϕϧΛࢦఆ͢Δ͜ͱͰͻͱ·ͣճආ͸Մೳ
    [1]: https://thanos.io/proposals/201812_thanos-remote-receive.md/

    View Slide

  21. ݱঢ়ͱࠓޙͷ՝୊
    wγϯϓϧͳ؀ڥʹ͓͍ͯ͸ɺઃఆͨ͠ΰʔϧ͸΄΅ୡ੒Ͱ͖͍ͯΔ
    wΑΓଟ͘ͷ1SPNFUIFVT͔ΒϝτϦΫεΛड৴͢Δʹ͸ҎԼͷΑ͏ͳରԠ͕ඞཁ
    wෳ਺ͷϝτϦΫεͷࠞࡏΛ๷͙࢓૊Έͷ࣮૷
    w5IBOPT3FDFJWFͷύϑΥʔϚϯεͷݕূ
    w5IBOPT3FDFJWFͷ৑௕Խػೳͷར༻

    View Slide

  22. Thank you !

    View Slide