Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Prometheusでデータの水平分割を試みる/Let's split prometheus data

January 15, 2020

Prometheusでデータの水平分割を試みる/Let's split prometheus data


January 15, 2020

More Decks by watawuwu

Other Decks in Technology


  1. Agenda • Motivation • Non-goal • How to respond to

    increase • How to scale out • Configuration • Browse and Aelrts • Issue
  2. • Reduce data retention • Prolong scrape interval • Reduce

    unnecessary metrics • Scale up • Scale out • Remote Write/Storage
  3. A. Per scrape rule • For popular settings in Kubernetes

    ◦ Container Metrics(cAdvisor) ◦ Node Metrics ◦ Application Metrics cAdvisor Node Application
  4. A. Per scrape rule • Application Metrics can be easily

    split into multiple scrape rules App A App B - job_name: 'app-xxx' kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_app_xxx_scrape] action: keep
  5. B. Per metrics(Not Time series) • Same scrape target, but

    decide whether to scrape by metrics name metric_relabel_configs: - source_labels: [__name__] action: drop regex: 'container_fs' cAdvisor cAdvisor
  6. C. Per label hash • Switch targets by label hash

    - source_labels: [__address__] modulus: ${shard_total} target_label: __tmp_hash action: hashmod - source_labels: [__tmp_hash] regex: ${shard_num} action: keep cAdvisor cAdvisor addr: addr: shard_num: 0 shard_num: 1 shard_total: 2
  7. A. Aggregate using Remote read API remote_read: - url: http://prometheus-01:9090/api/v1/read

    read_recent: true - url: http://prometheus-02:9090/api/v1/read read_recent: true - url: http://prometheus-03:9090/api/v1/read read_recent: true