Prometheusでデータの水平分割を試みる/Let's split prometheus data

Transcript

1. Prometheus で データ分割を試みる Prometheus Meetup Tokyo #3

2. profile: name: Wataru Matsui org: [ zlab.co.jp ] twitter: @watawuwu

3. Agenda • Motivation • Non-goal • How to respond to

   increase • How to scale out • Configuration • Browse and Aelrts • Issue

4. Motivation

5. Address growing data

6. Non-goal

7. × Highly available ◦ Data redundancy × Long term storage

8. How to respond to increase in memory and storage usage

9. • Reduce data retention • Prolong scrape interval • Reduce

   unnecessary metrics • Scale up • Scale out • Remote Write/Storage

10. How to scale out without remote storage

11. Prometheus is easy to scale out Pod Prometheus Pod Pod

    Pod Prometheus

12. Configuration

13. A. Per scrape rule • For popular settings in Kubernetes

    ◦ Container Metrics(cAdvisor) ◦ Node Metrics ◦ Application Metrics cAdvisor Node Application

14. A. Per scrape rule • Application Metrics can be easily

    split into multiple scrape rules App A App B - job_name: 'app-xxx' kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_app_xxx_scrape] action: keep

15. B. Per metrics(Not Time series) • Same scrape target, but

    decide whether to scrape by metrics name metric_relabel_configs: - source_labels: [__name__] action: drop regex: 'container_fs' cAdvisor cAdvisor

16. C. Per label hash • Switch targets by label hash

    - source_labels: [__address__] modulus: ${shard_total} target_label: __tmp_hash action: hashmod - source_labels: [__tmp_hash] regex: ${shard_num} action: keep cAdvisor cAdvisor addr: 10.26.80.18 addr: 10.26.80.19 shard_num: 0 shard_num: 1 shard_total: 2

17. Browse And Alerts

18. A. Aggregate using Remote read API remote_read: - url: http://prometheus-01:9090/api/v1/read

    read_recent: true - url: http://prometheus-02:9090/api/v1/read read_recent: true - url: http://prometheus-03:9090/api/v1/read read_recent: true

19. B. Aggregate using Thanos Querier Ruler Sidecar Sidecar Sidecar

20. Issue

21. × Not autoscale × Complicated by redundancy × Can't resharding

    or rebalancing

22. Thanks!