Prometheusでデータの水平分割を試みる/Let's split prometheus data

Transcript

1. Prometheus で
   データ分割を試みる
   Prometheus Meetup Tokyo #3
   

   View Slide

2. profile:
   name: Wataru Matsui
   org: [ zlab.co.jp ]
   twitter: @watawuwu
   

   View Slide

3. Agenda
   ● Motivation
   ● Non-goal
   ● How to respond to increase
   ● How to scale out
   ● Configuration
   ● Browse and Aelrts
   ● Issue
   

   View Slide

4. Motivation
   

   View Slide

5. Address growing data
   

   View Slide

6. Non-goal
   

   View Slide

7. × Highly available
   ○ Data redundancy
   × Long term storage
   

   View Slide

8. How to respond
   to increase in
   memory and storage usage
   

   View Slide

9. ● Reduce data retention
   ● Prolong scrape interval
   ● Reduce unnecessary metrics
   ● Scale up
   ● Scale out
   ● Remote Write/Storage
   

   View Slide

10. How to scale out
    without remote storage
    

    View Slide

11. Prometheus is easy to
    scale out
    Pod
    Prometheus
    Pod
    Pod
    Pod Prometheus
    

    View Slide

12. Configuration
    

    View Slide

13. A. Per scrape rule
    ● For popular settings in Kubernetes
    ○ Container Metrics(cAdvisor)
    ○ Node Metrics
    ○ Application Metrics
    cAdvisor Node
    Application
    

    View Slide

14. A. Per scrape rule
    ● Application Metrics can be easily split
    into multiple scrape rules
    App A
    App B
    - job_name: 'app-xxx'
    kubernetes_sd_configs:
    - role: endpoints
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_annotation_app_xxx_scrape]
    action: keep
    

    View Slide

15. B. Per metrics(Not Time series)
    ● Same scrape target, but decide whether
    to scrape by metrics name
    metric_relabel_configs:
    - source_labels: [__name__]
    action: drop
    regex: 'container_fs'
    cAdvisor
    cAdvisor
    

    View Slide

16. C. Per label hash
    ● Switch targets by label hash
    - source_labels: [__address__]
    modulus: ${shard_total}
    target_label: __tmp_hash
    action: hashmod
    - source_labels: [__tmp_hash]
    regex: ${shard_num}
    action: keep
    cAdvisor
    cAdvisor
    addr: 10.26.80.18
    addr: 10.26.80.19
    shard_num: 0
    shard_num: 1
    shard_total: 2
    

    View Slide

17. Browse And Alerts
    

    View Slide

18. A. Aggregate using
    Remote read API
    remote_read:
    - url: http://prometheus-01:9090/api/v1/read
    read_recent: true
    - url: http://prometheus-02:9090/api/v1/read
    read_recent: true
    - url: http://prometheus-03:9090/api/v1/read
    read_recent: true
    

    View Slide

19. B. Aggregate using Thanos
    Querier
    Ruler
    Sidecar
    Sidecar
    Sidecar
    

    View Slide

20. Issue
    

    View Slide

21. × Not autoscale
    × Complicated by redundancy
    × Can't resharding
    or rebalancing
    

    View Slide

22. Thanks!
    

    View Slide