Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
Prometheusでデータの水平分割を試みる/Let's split prometheus data
watawuwu
January 15, 2020
Technology
0
3.5k
Prometheusでデータの水平分割を試みる/Let's split prometheus data
watawuwu
January 15, 2020
Tweet
Share
More Decks by watawuwu
See All by watawuwu
watawuwu
5
560
watawuwu
10
1.2k
watawuwu
1
550
watawuwu
0
700
watawuwu
3
1.6k
Other Decks in Technology
See All in Technology
zaki134rp
1
180
vkbaba
0
120
shimacos
2
310
masashible
0
100
satotakeshi
2
420
am7cinnamon
2
2.7k
smzksts
0
220
minamizaki
0
580
ymas0315
0
150
line_developers
PRO
0
1.9k
hsano
0
120
yuuturn
1
130
Featured
See All Featured
michaelherold
225
8.5k
tenderlove
52
3.4k
qrush
285
18k
jcasabona
7
520
ddemaree
274
31k
matthewcrist
73
7.5k
chriscoyier
145
19k
destraynor
223
47k
mza
80
4.1k
danielanewman
1
470
jrom
114
7.1k
pedronauck
652
110k
Transcript
Prometheus で データ分割を試みる Prometheus Meetup Tokyo #3
profile: name: Wataru Matsui org: [ zlab.co.jp ] twitter: @watawuwu
Agenda • Motivation • Non-goal • How to respond to
increase • How to scale out • Configuration • Browse and Aelrts • Issue
Motivation
Address growing data
Non-goal
× Highly available ◦ Data redundancy × Long term storage
How to respond to increase in memory and storage usage
• Reduce data retention • Prolong scrape interval • Reduce
unnecessary metrics • Scale up • Scale out • Remote Write/Storage
How to scale out without remote storage
Prometheus is easy to scale out Pod Prometheus Pod Pod
Pod Prometheus
Configuration
A. Per scrape rule • For popular settings in Kubernetes
◦ Container Metrics(cAdvisor) ◦ Node Metrics ◦ Application Metrics cAdvisor Node Application
A. Per scrape rule • Application Metrics can be easily
split into multiple scrape rules App A App B - job_name: 'app-xxx' kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_app_xxx_scrape] action: keep
B. Per metrics(Not Time series) • Same scrape target, but
decide whether to scrape by metrics name metric_relabel_configs: - source_labels: [__name__] action: drop regex: 'container_fs' cAdvisor cAdvisor
C. Per label hash • Switch targets by label hash
- source_labels: [__address__] modulus: ${shard_total} target_label: __tmp_hash action: hashmod - source_labels: [__tmp_hash] regex: ${shard_num} action: keep cAdvisor cAdvisor addr: 10.26.80.18 addr: 10.26.80.19 shard_num: 0 shard_num: 1 shard_total: 2
Browse And Alerts
A. Aggregate using Remote read API remote_read: - url: http://prometheus-01:9090/api/v1/read
read_recent: true - url: http://prometheus-02:9090/api/v1/read read_recent: true - url: http://prometheus-03:9090/api/v1/read read_recent: true
B. Aggregate using Thanos Querier Ruler Sidecar Sidecar Sidecar
Issue
× Not autoscale × Complicated by redundancy × Can't resharding
or rebalancing
Thanks!