Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Using Thanos as a long-term storage for your Prometheus metrics
Moto Ishizawa
January 15, 2020
Technology
1
9.3k
Using Thanos as a long-term storage for your Prometheus metrics
Moto Ishizawa
January 15, 2020
Tweet
Share
More Decks by Moto Ishizawa
See All by Moto Ishizawa
カンムにおけるプロダクトセキュリティのこれまでとこれから
summerwind
1
1.4k
Sharing test cases of internet protocols with Go and OCI Artifacts
summerwind
0
410
Using Kubernetes as a datastore for SPIRE
summerwind
1
730
Whitebox Controller
summerwind
5
1.6k
Managing Kubernetes manifests with Spruce
summerwind
2
3.6k
Understanding HTTP/2 prioritization
summerwind
17
5.4k
HTTP/2 Deep Dive: Priority & Server Push
summerwind
17
3.1k
HTTP/2 Server Push Considered Harmful
summerwind
1
1.9k
Implementing HTTP/2 client in 60 minutes
summerwind
3
2.3k
Other Decks in Technology
See All in Technology
GraphQLスキーマ設計の勘所
yukukotani
26
6.1k
S3とCloudWatch Logsの見直しから始めるコスト削減 / Cost saving S3 and CloudWatch Logs
shonansurvivors
0
210
日経電子版だけじゃない! 日経の新規Webメディアの開発 - NIKKEI Tech Talk #3
sztm
0
150
【NGK2023S】 ノードエディタ形式の画像処理ツール「Image-Processing-Node-Editor」
kazuhitotakahashi
0
250
re:Inventで発表があったIoT事例の紹介と考察
kizawa2020
0
170
PHPのimmutable arrayとは
hnw
1
150
メドレー エンジニア採用資料/ Medley Engineer Guide
medley
3
5k
証明書って何だっけ? 〜AWSの中間CA移行に備える〜
minorun365
3
2k
Oracle Transaction Manager for Microservices Free 22.3 製品概要
oracle4engineer
PRO
5
100
本社オフィスを移転し、 オフィスファシリティ・コーポレートIT を刷新した話
rotomx
3
1.2k
Multi-Cloud Gatewayでデータを統治せよ!/ Data Federation with MCG
tutsunom
1
180
金属加工屋の営業マンがSTマイクロで・・・
usashirou
0
150
Featured
See All Featured
Testing 201, or: Great Expectations
jmmastey
25
5.7k
The Power of CSS Pseudo Elements
geoffreycrofte
52
4.3k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
7
570
Into the Great Unknown - MozCon
thekraken
2
280
XXLCSS - How to scale CSS and keep your sanity
sugarenia
236
1.1M
A Tale of Four Properties
chriscoyier
149
21k
In The Pink: A Labor of Love
frogandcode
132
21k
Optimizing for Happiness
mojombo
365
64k
Producing Creativity
orderedlist
PRO
335
37k
A better future with KSS
kneath
230
16k
Git: the NoSQL Database
bkeepers
PRO
419
60k
KATA
mclloyd
12
9.7k
Transcript
Using Thanos as a long-term storage for your Prometheus metrics
Moto Ishizawa Z Lab Corporation
Moto Ishizawa @summerwind Infrastructure Engineer, Z Lab
1SPNFUIFVT
എܠ wҎ্ͷ,VCFSOFUFTΫϥελʔ͕Քಇத w֤,VCFSOFUFTΫϥελʔͰઐ༻ͷ1SPNFUIFVT͕ىಈ͍ͯ͠Δ w֤,VCFSOFUFTΫϥελʔӬଓετϨʔδΛ࣋ͨͳ͍ w֤,VCFSOFUFTΫϥελʔͷϊʔυϝϯςφϯεͷͨΊʹिؒ͝ͱʹ࠶࡞͞ΕΔ wύϒϦοΫΫϥυ4BB4༻͍ͯ͠ͳ͍
՝ w֤Ϋϥελʔͷ1SPNFUIFVTͷϝτϦΫε͕ϊʔυͷ࠶࡞࣌ʹࣦΘΕͯ͠·͏ wཧऀ։ൃऀظؒͷϝτϦΫε͔͠ࢀরͰ͖ͳ͍ w֤ΫϥελʔͰ1SPNFUIFVTΛ࣮ߦ͢Δͷʹඞཁेͳ3".Λ֬อ͠ͳ͚ΕͳΒͳ͍
wཧऀ։ൃऀ͕Λ͑ΔظؒͷϝτϦΫεΛࢀরͰ͖Δ wશͯͷϝτϦΫεΛ"NB[PO4ͷΑ͏ͳ֎෦ͷΦϒδΣΫτετϨʔδʹอଘ͢Δ w֤Ϋϥελʔʹ͍Δ1SPNFUIFVTͷ3".ͷফඅྔΛ࠷খݶʹ͢Δ ΰʔϧ
Prometheus ΦϒδΣΫτετϨʔδ ཧऀ ϝτϦΫεͷอଘ ϝτϦΫεͷࢀর wΫϥελʔͷ1SPNFUIFVTظؒͷϝτϦΫε͚ͩΛอ࣋͢Δ w1SPNFUIFVT͕ऩूͨ͠ϝτϦΫεશͯΦϒδΣΫτετϨʔδʹอଘ͢Δ wཧऀ։ൃऀΦϒδΣΫτετϨʔδʹอଘ͞ΕͨϝτϦΫεΛࢀরͰ͖Δ ͜ΜͳΈ͕͋ΔͱΑͦ͞͏
࣮Ҋ$PSUFYͷར༻ wεέʔϥϒϧͳ1SPNFUIFVTޓγεςϜͷઌۦ͚ wΦϒδΣΫτετϨʔδͰͳ͘σʔλετΞʹϝτϦΫεΛอଘ͢Δ %ZOBNP%#ͳͲ w8FBWF$MPVE(SBGBOB$MPVEʹ͓͍ͯӡ༻࣮͕͋Δ
Prometheus ಠࣗΞμϓλʔ ཧऀ ΦϒδΣΫτετϨʔδ Remote Read/Write API ϝτϦΫεΛࢀর ϝτϦΫεΛసૹ ࣮ҊಠࣗΞμϓλʔͷ։ൃ
w1SPNFUIFVTͱΦϒδΣΫτετϨʔδΛܨ͙͚ͩͷ୯७ͳΞμϓλʔΛ։ൃ w1SPNFUIFVTͷ3FNPUF3FBE8SJUF"1*ͱ࣮ͯ͠ w1SPNFUIFVTͷ54%#ΛGPSLͯ͠ɺΦϒδΣΫτετϨʔδʹDIVOLΛอଘ͢ΔΑ͏मਖ਼ wཧऀ։ൃऀ͔Β1SPNFUIFVTͷετϨʔδ͚͕ͩେ͖͘ͳͬͨΑ͏ʹݟ͑Δ wϝτϦΫεͷࢀরʹඞཁͳͷͰ1SPNFUIFVTʹेͳ3".͕·ͩඞཁ
3FNPUF3FBE8SJUF"1* w1SPNFUIFVT͕֎෦ͷετϨʔδͱ࿈ܞͯ͠ϝτϦΫεΛಡΈॻ͖͢ΔΈ w1SPNFUIFVTͷઃఆϑΝΠϧʹ"1*Λ࣮ͨ͠ΤϯυϙΠϯτΛࢦఆ͢Δ͚ͩͰ͑Δ w$PSUFY*OqVY%#ͳͲɺଟ͘ͷ࣮͕͋Δ # Remote Write API ͷΤϯυϙΠϯτΛࢦఆ remote_write:
- url: http://127.0.0.1:9091/write # Remote Read API ͷΤϯυϙΠϯτΛࢦఆ remote_read: - url: http://127.0.0.1:9091/read Prometheus ͷઃఆϑΝΠϧ
3FNPUF3FBE8SJUF"1*ͷ࣮ w3FBEͱ8SJUFͷΤϯυϙΠϯτΛͭͣͭ༻ҙ͢Δ͚ͩͷγϯϓϧͳ࣮ w1SPUPDPM#V⒎FSTͰΤϯίʔυ͞ΕͨϝτϦΫεΛ4OBQQZͰѹॖ͠ɺ)551Ͱసૹ͢Δ w1SPNFUIFVTϦϙδτϦʹ͋Δαϯϓϧ࣮<>͕ࢀߟʹͳΔ wυΩϡϝϯτʹʮࠓޙϓϩτίϧ͕มΘΔՄೳੑ͕͋Δʯͱॻ͔Ε͍ͯΔ H31$ <> [1]: https://github.com/prometheus/prometheus/tree/master/documentation/examples/remote_storage [2]:
https://prometheus.io/docs/prometheus/latest/storage/
3FNPUF8SJUF"1*ར༻࣌ͷҙ w1SPNFUIFVTʹ3FNPUF8SJUF"1*ͷΤϯυϙΠϯτΛෳࢦఆ͢Δͱʜ wશͯͷΤϯυϙΠϯτʹಉ͡ϝτϦΫε͕సૹ͞ΕΔ 'BO0VUʹͳΔ <> wϥϯυϩϏϯͰͳ͍ͷͰɺෛՙࢄʹ͑ͳ͍ w1SPNFUIFVTWҎલΛར༻࣌ʹ3FNPUF8SJUF"1*ͷΤϯυϙΠϯτ͕μϯ͢Δͱʜ w1SPNFUIFVT͕సૹͷϦτϥΠΛ܁Γฦ͢ͷͰΤϯυϙΠϯτ෮ؼ࣌ͷෛՙ͕ʹ wసૹϦτϥΠதͷϝτϦΫεϝϞϦʔʹอ࣋͞ΕΔͷͰɺࣦΘΕΔϦεΫ͕͋Δ w1SPNFUIFVTW͔Βసૹࣦഊͨ͠ϝτϦΫε8"-ʹอଘ͞ΕΔΑ͏ʹͳͬͨ<>
wݱࡏ3FNPUF8SJUF"1*ར༻ऀ͚ͷνϡʔχϯάυΩϡϝϯτެ։͞Ε͍ͯΔ<> [1]: https://github.com/prometheus/prometheus/blob/master/documentation/internal_architecture.md [2]: https://grafana.com/blog/2019/03/25/whats-new-in-prometheus-2.8-wal-based-remote-write/ [3]: https://prometheus.io/docs/practices/remote_write/
࣮Ҋ5IBOPT wΦϒδΣΫτετϨʔδʹϝτϦΫεΛอଘ͢Δ1SPNFUIFVTޓͷγεςϜ w1SPNFUIFVTͷ֤ػೳΛίϯϙʔωϯτʹׂͨ͠Α͏ͳߏʹͳ͍ͬͯΔ wඞཁͳίϯϙʔωϯτ͚ͩΛબΜͰ͏͜ͱͰ͖Δ w5IBOPTW͔Β3FNPUF8SJUF"1*ʹΑΔϝτϦΫεͷड৴͕Մೳʹ
5IBOPTͷίϯϙʔωϯτ
5IBOPTͷίϯϙʔωϯτ 4JEFDBS 1SPNFUIFVTͷσʔλσΟϨΫτϦΛࢹͯ͠ɺΦϒδΣΫτετϨʔδʹϝτϦΫεΛΞοϓϩʔυ͢ Δɻ1SPNFUIFVTͱͷؔͰσϓϩΠ͢Δɻ 4UPSF ΦϒδΣΫτετϨʔδʹอଘ͞ΕͨϝτϦΫεʹΞΫηε͢ΔͨΊͷ"1*ɻ௨ৗ2VFSZͱΈ߹Θͤ ͯ͏ɻ 2VFSZ 1SPN2-Λ༻ͯ͠ϝτϦΫεΛूܭ͢Δɻ1SPNFUIFVTޓͷ6*ͱ"1*Λඋ͍͑ͯΔͨΊɺ 1SPNFUIFVTͱಉ͡Α͏ʹ͑Δɻ
3VMF 1SPNFUIFVTͷϧʔϧϑΝΠϧΛಡΈࠐΈɺΞϥʔτΛੜ͢ΔɻΦϒδΣΫτετϨʔδʹอଘ͞Εͨ ϝτϦΫεશମʹϧʔϧΛద༻͍ͨ͠߹ͳͲʹ༻ɻΞϥʔτͷૹ৴"MFSUNBOBHFSΛ͏ɻ $PNQBDU ΦϒδΣΫτετϨʔδʹอଘ͞ΕͨϝτϦΫεΛఆظతʹूͨ͠ΓμϯαϯϓϦϯά͢Δɻϲ݄ܦ աͨ͠ϝτϦΫεΛ·ͨ࣌ؒ͝ͱʹαϯϓϦϯά͠ͳ͓͢ɺͱ͍ͬͨ͜ͱ͕Ͱ͖Δɻ 3FDFJWF 3FNPUF8SJUF"1*Λඋ͓͑ͯΓɺ֎෦͔ΒϝτϦΫεΛड͚͚ΔɻૹΒΕ͖ͯͨϝτϦΫε 4JEFDBSͱಉ༷ʹΦϒδΣΫτετϨʔδʹอଘ͢Δɻ·࣮ͩݧతͳѻ͍ɻ
5IBOPT3FDFJWFΛར༻ͨ͠ߏ w5IBOPT3FDFJWFΛར༻ͯ͠ɺಠࣗΞμϓλʔΛஔ͖͑ wཧऀ։ൃऀ5IBOPT2VFSZΛ௨ͯ͡ϝτϦΫεΛࢀর͢Δ Prometheus Thanos Receive ཧऀ ΦϒδΣΫτετϨʔδ Remote Write
API ϝτϦΫεΛࢀর ϝτϦΫεΛอଘ Thanos Store Thanos Query ϝτϦΫεͷऔಘ ϝτϦΫεͷूܭ
5IBOPT3FDFJWFͷԽ wෳͷ5IBOPT3FDFJWFΛฒͯϩʔυόϥϯαʔΛஔ͢Εෛՙࢄ͕Ͱ͖Δ wૹΒΕ͖ͯͨϝτϦΫεͷϥϕϧηοτͳͲ͔Β୲͢Δ5IBOPT3FDFJWF͕ܾ·Δ w୲Ͱͳ͍ϝτϦΫεσʔλΛड৴ͨ͠߹୲͢Δ5IBOPT3FDFJWFʹసૹ͢Δ Prometheus Thanos Receive #3 ϩʔυόϥϯαʔ Remote
Write API ϦΫΤετૹ৴ ϦΫΤετసૹ Thanos Receive #2 Thanos Receive #1 Thanos Receive #2 ͕୲͢ΔϝτϦΫε Thanos Receive #1 ͕୲͢ΔϝτϦΫε
5IBOPT3FDFJWFʹΑΔϝτϦΫεͷϨϓϦέʔγϣϯ wෳͷ5IBOPT3FDFJWFͰಉ͡ϝτϦΫεΛอଘ͢Δ w3FQMJDBUJPO'BDUPSͷ͔ΒɺԿͷ5IBOPT3FDFJWFͷॻ͖ࠐΈ͕ඞཁ͔Λܾఆ w 3FQMJDBUJPO'BDUPS ͷ5IBOPT3FDFJWFͷॻ͖ࠐΈʹࣦഊ͢ΔͱΤϥʔ Prometheus Thanos Receive
#3 ϩʔυόϥϯαʔ Remote Write API ϦΫΤετૹ৴ ϨϓϦέʔγϣϯ Thanos Receive #2 Thanos Receive #1
5IBOPT3FDFJWFͷ࣮ߦ $ thanos receive \ --tsdb.path /var/lib/thanos \ # TSDB
ϑΝΠϧͷύε --tsdb.retention 3h \ # TSDB ϑΝΠϧͷอ࣋ظݶ --objstore.config-file /etc/thanos/bucket_config.yaml \ # ΦϒδΣΫτετϨʔδͷઃఆ --receive.local-endpoint http://thanos1.local:19291/api/v1/receive \ # ࣗͷΤϯυϙΠϯτ໊ --receive.hashrings-file /etc/thanos/hashring.json \ # ԽͷͨΊͷϋογϡϦϯάͷઃఆ --receive.replication-factor 3 # Replication Factor ͷઃఆ --label "thanos='1'" ɹɹɹɹɹɹ# Thanos ͷ External Label [ { "hashring": "cluster", "endpoints": [ "http://thanos1.local:19291/api/v1/receive", "http://thanos2.local:19291/api/v1/receive", "http://thanos3.local:19291/api/v1/receive" ] } ] /etc/thanos/hashring.json
5IBOPT3FDFJWFར༻࣌ͷҙ w࣮ݧతͳѻ͍ͷίϯϙʔωϯτͰ͋Γɺ͍ํʹؔ͢ΔυΩϡϝϯτ͕ͳ͍ wػೳ༷1SPQPTBMυΩϡϝϯτͰ֬ೝͰ͖Δ<> wڍಈͷࡉ͔͍෦ιʔείʔυΛ͍͔͚͍ͯ͘ඞཁ͕͋Δ wthanos receiveίϚϯυͷιʔείʔυ͔Β͍͔͚͍ͯ͘ͱޮ͕Α͍ w--label ΦϓγϣϯͷࢦఆΛ͠ͳ͍ͱϝτϦΫεͷΞοϓϩʔυʹࣦഊ͢Δ w4JEFDBSͷ࣮Λྲྀ༻ͯ͠3FDFJWF͕࣮͞Εͨ͜ͱʹΑΔෆ۩߹ͱࢥΘΕΔ w4JEFDBSͰ1SPNFUIFVTͷ&YUFSOBM-BCFMͷઃఆΛ༻͢Δ͕ɺ3FDFJWFʹͰ͖ͳ͍
wదͳϥϕϧΛࢦఆ͢Δ͜ͱͰͻͱ·ͣճආՄೳ [1]: https://thanos.io/proposals/201812_thanos-remote-receive.md/
ݱঢ়ͱࠓޙͷ՝ wγϯϓϧͳڥʹ͓͍ͯɺઃఆͨ͠ΰʔϧ΄΅ୡͰ͖͍ͯΔ wΑΓଟ͘ͷ1SPNFUIFVT͔ΒϝτϦΫεΛड৴͢ΔʹҎԼͷΑ͏ͳରԠ͕ඞཁ wෳͷϝτϦΫεͷࠞࡏΛ͙Έͷ࣮ w5IBOPT3FDFJWFͷύϑΥʔϚϯεͷݕূ w5IBOPT3FDFJWFͷԽػೳͷར༻
Thank you !