ClickHouse exporter

ClickHouse exporter as simple Prometheus exporter

Transcript

1. CLICKHOUSE EXPORTER AS SIMPLE PROMETHEUS EXPORTER Presented by Yegor Andreenko / @f1yegor
2. WHOAMI Yegor Andreenko @f1yegor Software Engineer focused on data processing and mining
3. HOW DO WE MONITOR? Prometheus: From metrics to insight. Power your metrics and alerting with a leading open-source monitoring solution.
5. A lot of exporters
6. Blazingly fast open-source analytical column-oriented database You should try it!
7. CLICKHOUSE DOCUMENTATION Tutorial Reference Forum Benchmarks
8. # HELP clickhouse_compressed_read_buffer_blocks_total Number of CompressedReadBufferBlocks total processe # TYPE clickhouse_compressed_read_buffer_blocks_total counter clickhouse_compressed_read_buffer_blocks_total 7.770087e+06 # HELP clickhouse_replicas_sum_inserts_in_queue Number of ReplicasSumInsertsInQueue async processed # TYPE clickhouse_replicas_sum_inserts_in_queue gauge clickhouse_replicas_sum_inserts_in_queue 3 # HELP go_gc_duration_seconds A summary of the GC invocation durations. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile="0"} 0.000111917 go_gc_duration_seconds{quantile="0.25"} 0.000184783 go_gc_duration_seconds{quantile="0.5"} 0.000233279 go_gc_duration_seconds{quantile="0.75"} 0.000310706 go_gc_duration_seconds{quantile="1"} 0.004569921 go_gc_duration_seconds_sum 8.431114968 go_gc_duration_seconds_count 24784
9. // Exporter collects clickhouse stats from the given URI and exports them using // the prometheus metrics package. type Exporter struct { metricsURI string asyncMetricsURI string eventsURI string partsURI string mutex sync.RWMutex client *http.Client scrapeFailures prometheus.Counter gauges []*prometheus.GaugeVec counters []*prometheus.CounterVec }
10. for _, m := range metrics { newMetric := prometheus.NewGaugeVec(prometheus.GaugeOpts{ Namespace: namespace, Name: metricName(m.key), Help: "Number of " + m.key + " currently processed", }, []string{}).WithLabelValues() newMetric.Set(float64(m.value)) newMetric.Collect(ch) }
11. SOME GOTCHAS AND FUTURE IMPROVEMENTS // Describe describes all the metrics ever exported by the clickhouse exporter. It // implements prometheus.Collector. func (e *Exporter) Describe(ch chan<- *prometheus.Desc) { // We cannot know in advance what metrics the exporter will generate // from clickhouse. So we use the poor man's describe method: Run a collect // and send the descriptors of all the collected metrics. metricCh := make(chan prometheus.Metric) doneCh := make(chan struct{}) go func() { for m := range metricCh { ch <- m.Desc() } close(doneCh) }() e.Collect(metricCh) close(metricCh) <-doneCh }
12. SOME GOTCHAS AND FUTURE IMPROVEMENTS var _ Exporter = (*prometheus.Collector)(nil) Scrape the whole cluster instead of 1 host??? .gitlab integration test
13. QA Yegor Andreenko @f1yegor
14. Visit @dotGo via my referral link https://dotgo2017.eventbrite.com? discount=AMBASSADORYA100 with 20% discount.