Using Thanos as a long-term storage for your Prometheus metrics

Transcript

1. Using Thanos as a long-term storage for your Prometheus metrics
   Moto Ishizawa
   Z Lab Corporation
   

   View Slide

2. Moto Ishizawa
   @summerwind
   Infrastructure Engineer, Z Lab
   

   View Slide

3. 1SPNFUIFVT
   

   View Slide

4. എܠ
   wҎ্ͷ,VCFSOFUFTΫϥελʔ͕Քಇத
   w֤,VCFSOFUFTΫϥελʔͰ͸ઐ༻ͷ1SPNFUIFVT͕ىಈ͍ͯ͠Δ
   w֤,VCFSOFUFTΫϥελʔ͸ӬଓετϨʔδΛ࣋ͨͳ͍
   w֤,VCFSOFUFTΫϥελʔͷϊʔυ͸ϝϯςφϯεͷͨΊʹ਺िؒ͝ͱʹ࠶࡞੒͞ΕΔ
   wύϒϦοΫΫϥ΢υ΍4BB4͸࢖༻͍ͯ͠ͳ͍
   

   View Slide

5. ՝୊
   w֤Ϋϥελʔͷ1SPNFUIFVTͷϝτϦΫε͕ϊʔυͷ࠶࡞੒࣌ʹࣦΘΕͯ͠·͏
   w؅ཧऀ΍։ൃऀ͸୹ظؒͷϝτϦΫε͔͠ࢀরͰ͖ͳ͍
   w֤ΫϥελʔͰ1SPNFUIFVTΛ࣮ߦ͢Δͷʹඞཁे෼ͳ3".Λ֬อ͠ͳ͚Ε͹ͳΒͳ͍
   

   View Slide

6. w؅ཧऀ΍։ൃऀ͕೔Λ௒͑Δ௕ظؒͷϝτϦΫεΛࢀরͰ͖Δ
   wશͯͷϝτϦΫεΛ"NB[PO4ͷΑ͏ͳ֎෦ͷΦϒδΣΫτετϨʔδʹอଘ͢Δ
   w֤Ϋϥελʔʹ͍Δ1SPNFUIFVTͷ3".ͷফඅྔΛ࠷খݶʹ͢Δ
   ΰʔϧ
   

   View Slide

7. 
   
   Prometheus ΦϒδΣΫτετϨʔδ ؅ཧऀ
   ϝτϦΫεͷอଘ ϝτϦΫεͷࢀর
   wΫϥελʔͷ1SPNFUIFVT͸୹ظؒͷϝτϦΫε͚ͩΛอ࣋͢Δ
   w1SPNFUIFVT͕ऩूͨ͠ϝτϦΫε͸શͯΦϒδΣΫτετϨʔδʹอଘ͢Δ
   w؅ཧऀ΍։ൃऀ͸ΦϒδΣΫτετϨʔδʹอଘ͞ΕͨϝτϦΫεΛࢀরͰ͖Δ
   ͜Μͳ࢓૊Έ͕͋ΔͱΑͦ͞͏
   

   View Slide

8. ࣮૷Ҋ$PSUFYͷར༻
   wεέʔϥϒϧͳ1SPNFUIFVTޓ׵γεςϜͷઌۦ͚
   wΦϒδΣΫτετϨʔδͰ͸ͳ͘σʔλετΞʹϝτϦΫεΛอଘ͢Δ %ZOBNP%#ͳͲ
   
   w8FBWF$MPVE΍(SBGBOB$MPVEʹ͓͍ͯӡ༻࣮੷͕͋Δ
   

   View Slide

9. 
   
   Prometheus ಠࣗΞμϓλʔ
   ؅ཧऀ ΦϒδΣΫτετϨʔδ
   Remote
   Read/Write API
   ϝτϦΫεΛࢀর ϝτϦΫεΛసૹ
   ࣮૷ҊಠࣗΞμϓλʔͷ։ൃ
   w1SPNFUIFVTͱΦϒδΣΫτετϨʔδΛܨ͙͚ͩͷ୯७ͳΞμϓλʔΛ։ൃ
   w1SPNFUIFVTͷ3FNPUF3FBE8SJUF"1*ͱ࣮ͯ͠૷
   w1SPNFUIFVTͷ54%#ΛGPSLͯ͠ɺΦϒδΣΫτετϨʔδʹDIVOLΛอଘ͢ΔΑ͏मਖ਼
   w؅ཧऀ΍։ൃऀ͔Β͸1SPNFUIFVTͷετϨʔδ͚͕ͩେ͖͘ͳͬͨΑ͏ʹݟ͑Δ
   wϝτϦΫεͷࢀরʹඞཁͳͷͰ1SPNFUIFVTʹ͸े෼ͳ3".͕·ͩඞཁ
   

   View Slide

10. 3FNPUF3FBE8SJUF"1*
    w1SPNFUIFVT͕֎෦ͷετϨʔδͱ࿈ܞͯ͠ϝτϦΫεΛಡΈॻ͖͢Δ࢓૊Έ
    w1SPNFUIFVTͷઃఆϑΝΠϧʹ"1*Λ࣮૷ͨ͠ΤϯυϙΠϯτΛࢦఆ͢Δ͚ͩͰ࢖͑Δ
    w$PSUFY΍*OqVY%#ͳͲɺଟ͘ͷ࣮૷͕͋Δ
    # Remote Write API ͷΤϯυϙΠϯτΛࢦఆ
    remote_write:
    - url: http://127.0.0.1:9091/write
    # Remote Read API ͷΤϯυϙΠϯτΛࢦఆ
    remote_read:
    - url: http://127.0.0.1:9091/read
    Prometheus ͷઃఆϑΝΠϧ
    

    View Slide

11. 3FNPUF3FBE8SJUF"1*ͷ࣮૷
    w3FBEͱ8SJUFͷΤϯυϙΠϯτΛͭͣͭ༻ҙ͢Δ͚ͩͷγϯϓϧͳ࣮૷
    w1SPUPDPM#V⒎FSTͰΤϯίʔυ͞ΕͨϝτϦΫεΛ4OBQQZͰѹॖ͠ɺ)551Ͱసૹ͢Δ
    w1SPNFUIFVTϦϙδτϦʹ͋Δαϯϓϧ࣮૷<>͕ࢀߟʹͳΔ
    wυΩϡϝϯτʹ͸ʮࠓޙϓϩτίϧ͕มΘΔՄೳੑ͕͋Δʯͱॻ͔Ε͍ͯΔ H31$
    <>
    [1]: https://github.com/prometheus/prometheus/tree/master/documentation/examples/remote_storage
    [2]: https://prometheus.io/docs/prometheus/latest/storage/
    

    View Slide

12. 3FNPUF8SJUF"1*ར༻࣌ͷ஫ҙ఺
    w1SPNFUIFVTʹ3FNPUF8SJUF"1*ͷΤϯυϙΠϯτΛෳ਺ࢦఆ͢Δͱʜ
    wશͯͷΤϯυϙΠϯτʹಉ͡ϝτϦΫε͕సૹ͞ΕΔ 'BO0VUʹͳΔ
    <>
    wϥ΢ϯυϩϏϯͰ͸ͳ͍ͷͰɺෛՙ෼ࢄʹ͸࢖͑ͳ͍
    w1SPNFUIFVTWҎલΛར༻࣌ʹ3FNPUF8SJUF"1*ͷΤϯυϙΠϯτ͕μ΢ϯ͢Δͱʜ
    w1SPNFUIFVT͕సૹͷϦτϥΠΛ܁Γฦ͢ͷͰΤϯυϙΠϯτ෮ؼ࣌ͷෛՙ͕໰୊ʹ
    wసૹϦτϥΠதͷϝτϦΫε͸ϝϞϦʔʹอ࣋͞ΕΔͷͰɺࣦΘΕΔϦεΫ͕͋Δ
    w1SPNFUIFVTW͔Βసૹࣦഊͨ͠ϝτϦΫε΋8"-ʹอଘ͞ΕΔΑ͏ʹͳͬͨ<>
    wݱࡏ͸3FNPUF8SJUF"1*ར༻ऀ޲͚ͷνϡʔχϯάυΩϡϝϯτ΋ެ։͞Ε͍ͯΔ<>
    [1]: https://github.com/prometheus/prometheus/blob/master/documentation/internal_architecture.md
    [2]: https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/
    [3]: https://prometheus.io/docs/practices/remote_write/
    

    View Slide

13. ࣮૷Ҋ5IBOPT
    wΦϒδΣΫτετϨʔδʹϝτϦΫεΛอଘ͢Δ1SPNFUIFVTޓ׵ͷγεςϜ
    w1SPNFUIFVTͷ֤ػೳΛίϯϙʔωϯτʹ෼ׂͨ͠Α͏ͳߏ੒ʹͳ͍ͬͯΔ
    wඞཁͳίϯϙʔωϯτ͚ͩΛબΜͰ࢖͏͜ͱ΋Ͱ͖Δ
    w5IBOPTW͔Β3FNPUF8SJUF"1*ʹΑΔϝτϦΫεͷड৴͕Մೳʹ
    

    View Slide

14. 5IBOPTͷίϯϙʔωϯτ
    

    View Slide

15. 5IBOPTͷίϯϙʔωϯτ
    4JEFDBS
    1SPNFUIFVTͷσʔλσΟϨΫτϦΛ؂ࢹͯ͠ɺΦϒδΣΫτετϨʔδʹϝτϦΫεΛΞοϓϩʔυ͢
    Δɻ1SPNFUIFVTͱͷؔ܎ͰσϓϩΠ͢Δɻ
    4UPSF
    ΦϒδΣΫτετϨʔδʹอଘ͞ΕͨϝτϦΫεʹΞΫηε͢ΔͨΊͷ"1*ɻ௨ৗ͸2VFSZͱ૊Έ߹Θͤ
    ͯ࢖͏ɻ
    2VFSZ
    1SPN2-Λ࢖༻ͯ͠ϝτϦΫεΛूܭ͢Δɻ1SPNFUIFVTޓ׵ͷ6*ͱ"1*Λඋ͍͑ͯΔͨΊɺ
    1SPNFUIFVTͱಉ͡Α͏ʹ࢖͑Δɻ
    3VMF
    1SPNFUIFVTͷϧʔϧϑΝΠϧΛಡΈࠐΈɺΞϥʔτΛੜ੒͢ΔɻΦϒδΣΫτετϨʔδʹอଘ͞Εͨ
    ϝτϦΫεશମʹϧʔϧΛద༻͍ͨ͠৔߹ͳͲʹ࢖༻ɻΞϥʔτͷૹ৴͸"MFSUNBOBHFSΛ࢖͏ɻ
    $PNQBDU
    ΦϒδΣΫτετϨʔδʹอଘ͞ΕͨϝτϦΫεΛఆظతʹू໿ͨ͠Γμ΢ϯαϯϓϦϯά͢Δɻϲ݄ܦ
    աͨ͠ϝτϦΫεΛ෼·ͨ͸࣌ؒ͝ͱʹαϯϓϦϯά͠ͳ͓͢ɺͱ͍ͬͨ͜ͱ͕Ͱ͖Δɻ
    3FDFJWF
    3FNPUF8SJUF"1*Λඋ͓͑ͯΓɺ֎෦͔ΒϝτϦΫεΛड͚෇͚ΔɻૹΒΕ͖ͯͨϝτϦΫε͸
    4JEFDBSͱಉ༷ʹΦϒδΣΫτετϨʔδʹอଘ͢Δɻ·࣮ͩݧతͳѻ͍ɻ
    

    View Slide

16. 5IBOPT3FDFJWFΛར༻ͨ͠ߏ੒
    w5IBOPT3FDFJWFΛར༻ͯ͠ɺಠࣗΞμϓλʔΛஔ͖׵͑
    w؅ཧऀ΍։ൃऀ͸5IBOPT2VFSZΛ௨ͯ͡ϝτϦΫεΛࢀর͢Δ
    
    
    Prometheus Thanos Receive
    ؅ཧऀ
    ΦϒδΣΫτετϨʔδ
    Remote Write API
    ϝτϦΫεΛࢀর
    ϝτϦΫεΛอଘ
    Thanos Store
    Thanos Query
    ϝτϦΫεͷऔಘ
    ϝτϦΫεͷूܭ
    

    View Slide

17. 5IBOPT3FDFJWFͷ৑௕Խ
    wෳ਺ͷ5IBOPT3FDFJWFΛฒ΂ͯϩʔυόϥϯαʔΛ഑ஔ͢Ε͹ෛՙ෼ࢄ͕Ͱ͖Δ
    wૹΒΕ͖ͯͨϝτϦΫεͷϥϕϧηοτͳͲ͔Β୲౰͢Δ5IBOPT3FDFJWF͕ܾ·Δ
    w୲౰Ͱͳ͍ϝτϦΫεσʔλΛड৴ͨ͠৔߹͸୲౰͢Δ5IBOPT3FDFJWFʹసૹ͢Δ
    Prometheus
    Thanos Receive #3
    ϩʔυόϥϯαʔ
    Remote Write API ϦΫΤετૹ৴
    ϦΫΤετసૹ
    Thanos Receive #2
    Thanos Receive #1
    Thanos Receive #2 ͕୲౰͢ΔϝτϦΫε
    Thanos Receive #1 ͕୲౰͢ΔϝτϦΫε
    

    View Slide

18. 5IBOPT3FDFJWFʹΑΔϝτϦΫεͷϨϓϦέʔγϣϯ
    wෳ਺ͷ5IBOPT3FDFJWFͰಉ͡ϝτϦΫεΛอଘ͢Δ
    w3FQMJDBUJPO'BDUPSͷ஋͔ΒɺԿ୆ͷ5IBOPT3FDFJWF΁ͷॻ͖ࠐΈ͕ඞཁ͔Λܾఆ
    w 3FQMJDBUJPO'BDUPS
    ୆ͷ5IBOPT3FDFJWF΁ͷॻ͖ࠐΈʹࣦഊ͢ΔͱΤϥʔ
    Prometheus
    Thanos Receive #3
    ϩʔυόϥϯαʔ
    Remote Write API ϦΫΤετૹ৴
    ϨϓϦέʔγϣϯ
    Thanos Receive #2
    Thanos Receive #1
    

    View Slide

19. 5IBOPT3FDFJWFͷ࣮ߦ
    $ thanos receive \
    --tsdb.path /var/lib/thanos \ # TSDB ϑΝΠϧͷύε
    --tsdb.retention 3h \ # TSDB ϑΝΠϧͷอ࣋ظݶ
    --objstore.config-file /etc/thanos/bucket_config.yaml \ # ΦϒδΣΫτετϨʔδͷઃఆ
    --receive.local-endpoint http://thanos1.local:19291/api/v1/receive \ # ࣗ਎ͷΤϯυϙΠϯτ໊
    --receive.hashrings-file /etc/thanos/hashring.json \ # ৑௕ԽͷͨΊͷϋογϡϦϯάͷઃఆ
    --receive.replication-factor 3 # Replication Factor ͷઃఆ
    --label "thanos='1'" ɹɹɹɹɹɹ# Thanos ͷ External Label
    [
    {
    "hashring": "cluster",
    "endpoints": [
    "http://thanos1.local:19291/api/v1/receive",
    "http://thanos2.local:19291/api/v1/receive",
    "http://thanos3.local:19291/api/v1/receive"
    ]
    }
    ]
    /etc/thanos/hashring.json
    

    View Slide

20. 5IBOPT3FDFJWFར༻࣌ͷ஫ҙ఺
    w࣮ݧతͳѻ͍ͷίϯϙʔωϯτͰ͋Γɺ࢖͍ํʹؔ͢ΔυΩϡϝϯτ͕ͳ͍
    wػೳ΍࢓༷͸1SPQPTBMυΩϡϝϯτͰ֬ೝͰ͖Δ<>
    wڍಈͷࡉ͔͍෦෼͸ιʔείʔυΛ௥͍͔͚͍ͯ͘ඞཁ͕͋Δ
    wthanos receiveίϚϯυͷιʔείʔυ͔Β௥͍͔͚͍ͯ͘ͱޮ཰͕Α͍
    w--label ΦϓγϣϯͷࢦఆΛ͠ͳ͍ͱϝτϦΫεͷΞοϓϩʔυʹࣦഊ͢Δ
    w4JEFDBSͷ࣮૷Λྲྀ༻ͯ͠3FDFJWF͕࣮૷͞Εͨ͜ͱʹΑΔෆ۩߹ͱࢥΘΕΔ
    w4JEFDBSͰ͸1SPNFUIFVTͷ&YUFSOBM-BCFMͷઃఆΛ࢖༻͢Δ͕ɺ3FDFJWFʹ͸Ͱ͖ͳ͍
    wద౰ͳϥϕϧΛࢦఆ͢Δ͜ͱͰͻͱ·ͣճආ͸Մೳ
    [1]: https://thanos.io/proposals/201812_thanos-remote-receive.md/
    

    View Slide

21. ݱঢ়ͱࠓޙͷ՝୊
    wγϯϓϧͳ؀ڥʹ͓͍ͯ͸ɺઃఆͨ͠ΰʔϧ͸΄΅ୡ੒Ͱ͖͍ͯΔ
    wΑΓଟ͘ͷ1SPNFUIFVT͔ΒϝτϦΫεΛड৴͢Δʹ͸ҎԼͷΑ͏ͳରԠ͕ඞཁ
    wෳ਺ͷϝτϦΫεͷࠞࡏΛ๷͙࢓૊Έͷ࣮૷
    w5IBOPT3FDFJWFͷύϑΥʔϚϯεͷݕূ
    w5IBOPT3FDFJWFͷ৑௕Խػೳͷར༻
    

    View Slide

22. Thank you !
    

    View Slide