Prometheus で データ分割を試みる Prometheus Meetup Tokyo #3

profile: name: Wataru Matsui org: [ zlab.co.jp ] twitter: @watawuwu

Agenda ● Motivation ● Non-goal ● How to respond to increase ● How to scale out ● Configuration ● Browse and Aelrts ● Issue

Motivation

Address growing data

Non-goal

× Highly available ○ Data redundancy × Long term storage

How to respond to increase in memory and storage usage

● Reduce data retention ● Prolong scrape interval ● Reduce unnecessary metrics ● Scale up ● Scale out ● Remote Write/Storage

How to scale out without remote storage

Prometheus is easy to scale out Pod Prometheus Pod Pod Pod Prometheus

Configuration

A. Per scrape rule ● For popular settings in Kubernetes ○ Container Metrics(cAdvisor) ○ Node Metrics ○ Application Metrics cAdvisor Node Application

A. Per scrape rule ● Application Metrics can be easily split into multiple scrape rules App A App B - job_name: 'app-xxx' kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_app_xxx_scrape] action: keep

B. Per metrics(Not Time series) ● Same scrape target, but decide whether to scrape by metrics name metric_relabel_configs: - source_labels: [__name__] action: drop regex: 'container_fs' cAdvisor cAdvisor

C. Per label hash ● Switch targets by label hash - source_labels: [__address__] modulus: ${shard_total} target_label: __tmp_hash action: hashmod - source_labels: [__tmp_hash] regex: ${shard_num} action: keep cAdvisor cAdvisor addr: 10.26.80.18 addr: 10.26.80.19 shard_num: 0 shard_num: 1 shard_total: 2

Browse And Alerts

A. Aggregate using Remote read API remote_read: - url: http://prometheus-01:9090/api/v1/read read_recent: true - url: http://prometheus-02:9090/api/v1/read read_recent: true - url: http://prometheus-03:9090/api/v1/read read_recent: true

B. Aggregate using Thanos Querier Ruler Sidecar Sidecar Sidecar

Issue

× Not autoscale × Complicated by redundancy × Can't resharding or rebalancing

Thanks!