$30 off During Our Annual Pro Sale. View Details »

Prometheusでデータの水平分割を試みる/Let's split prometheus data

watawuwu
January 15, 2020

Prometheusでデータの水平分割を試みる/Let's split prometheus data

watawuwu

January 15, 2020
Tweet

More Decks by watawuwu

Other Decks in Technology

Transcript

  1. Prometheus で
    データ分割を試みる
    Prometheus Meetup Tokyo #3

    View Slide

  2. profile:
    name: Wataru Matsui
    org: [ zlab.co.jp ]
    twitter: @watawuwu

    View Slide

  3. Agenda
    ● Motivation
    ● Non-goal
    ● How to respond to increase
    ● How to scale out
    ● Configuration
    ● Browse and Aelrts
    ● Issue

    View Slide

  4. Motivation

    View Slide

  5. Address growing data

    View Slide

  6. Non-goal

    View Slide

  7. × Highly available
    ○ Data redundancy
    × Long term storage

    View Slide

  8. How to respond
    to increase in
    memory and storage usage

    View Slide

  9. ● Reduce data retention
    ● Prolong scrape interval
    ● Reduce unnecessary metrics
    ● Scale up
    ● Scale out
    ● Remote Write/Storage

    View Slide

  10. How to scale out
    without remote storage

    View Slide

  11. Prometheus is easy to
    scale out
    Pod
    Prometheus
    Pod
    Pod
    Pod Prometheus

    View Slide

  12. Configuration

    View Slide

  13. A. Per scrape rule
    ● For popular settings in Kubernetes
    ○ Container Metrics(cAdvisor)
    ○ Node Metrics
    ○ Application Metrics
    cAdvisor Node
    Application

    View Slide

  14. A. Per scrape rule
    ● Application Metrics can be easily split
    into multiple scrape rules
    App A
    App B
    - job_name: 'app-xxx'
    kubernetes_sd_configs:
    - role: endpoints
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_annotation_app_xxx_scrape]
    action: keep

    View Slide

  15. B. Per metrics(Not Time series)
    ● Same scrape target, but decide whether
    to scrape by metrics name
    metric_relabel_configs:
    - source_labels: [__name__]
    action: drop
    regex: 'container_fs'
    cAdvisor
    cAdvisor

    View Slide

  16. C. Per label hash
    ● Switch targets by label hash
    - source_labels: [__address__]
    modulus: ${shard_total}
    target_label: __tmp_hash
    action: hashmod
    - source_labels: [__tmp_hash]
    regex: ${shard_num}
    action: keep
    cAdvisor
    cAdvisor
    addr: 10.26.80.18
    addr: 10.26.80.19
    shard_num: 0
    shard_num: 1
    shard_total: 2

    View Slide

  17. Browse And Alerts

    View Slide

  18. A. Aggregate using
    Remote read API
    remote_read:
    - url: http://prometheus-01:9090/api/v1/read
    read_recent: true
    - url: http://prometheus-02:9090/api/v1/read
    read_recent: true
    - url: http://prometheus-03:9090/api/v1/read
    read_recent: true

    View Slide

  19. B. Aggregate using Thanos
    Querier
    Ruler
    Sidecar
    Sidecar
    Sidecar

    View Slide

  20. Issue

    View Slide

  21. × Not autoscale
    × Complicated by redundancy
    × Can't resharding
    or rebalancing

    View Slide

  22. Thanks!

    View Slide