Prometheusでデータの水平分割を試みる/Let's split prometheus data

E7a86f7090ecce18bc3848324741b04c?s=47 watawuwu
January 15, 2020

Prometheusでデータの水平分割を試みる/Let's split prometheus data

E7a86f7090ecce18bc3848324741b04c?s=128

watawuwu

January 15, 2020
Tweet

Transcript

  1. Prometheus で データ分割を試みる Prometheus Meetup Tokyo #3

  2. profile: name: Wataru Matsui org: [ zlab.co.jp ] twitter: @watawuwu

  3. Agenda • Motivation • Non-goal • How to respond to

    increase • How to scale out • Configuration • Browse and Aelrts • Issue
  4. Motivation

  5. Address growing data

  6. Non-goal

  7. × Highly available ◦ Data redundancy × Long term storage

  8. How to respond to increase in memory and storage usage

  9. • Reduce data retention • Prolong scrape interval • Reduce

    unnecessary metrics • Scale up • Scale out • Remote Write/Storage
  10. How to scale out without remote storage

  11. Prometheus is easy to scale out Pod Prometheus Pod Pod

    Pod Prometheus
  12. Configuration

  13. A. Per scrape rule • For popular settings in Kubernetes

    ◦ Container Metrics(cAdvisor) ◦ Node Metrics ◦ Application Metrics cAdvisor Node Application
  14. A. Per scrape rule • Application Metrics can be easily

    split into multiple scrape rules App A App B - job_name: 'app-xxx' kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_app_xxx_scrape] action: keep
  15. B. Per metrics(Not Time series) • Same scrape target, but

    decide whether to scrape by metrics name metric_relabel_configs: - source_labels: [__name__] action: drop regex: 'container_fs' cAdvisor cAdvisor
  16. C. Per label hash • Switch targets by label hash

    - source_labels: [__address__] modulus: ${shard_total} target_label: __tmp_hash action: hashmod - source_labels: [__tmp_hash] regex: ${shard_num} action: keep cAdvisor cAdvisor addr: 10.26.80.18 addr: 10.26.80.19 shard_num: 0 shard_num: 1 shard_total: 2
  17. Browse And Alerts

  18. A. Aggregate using Remote read API remote_read: - url: http://prometheus-01:9090/api/v1/read

    read_recent: true - url: http://prometheus-02:9090/api/v1/read read_recent: true - url: http://prometheus-03:9090/api/v1/read read_recent: true
  19. B. Aggregate using Thanos Querier Ruler Sidecar Sidecar Sidecar

  20. Issue

  21. × Not autoscale × Complicated by redundancy × Can't resharding

    or rebalancing
  22. Thanks!