What is Prometheus?（2/2） 参考 Prometheus and OpenMetrics, with Richard Hartmann https://kubernetespodcast.com/episode/037-prometheus-and-openmetrics/ Kubernetes Podcast「Prometheus and OpenMetrics, with Richard Hartmann」より Prometheus came into being because a few ex-Googlers were quite unhappy with what they found in the open-source and also in the paid software world. So they basically reimplemented large parts of Borgmon into Prometheus. In recent years, monitoring has undergone a Cambrian Explosion: Riemann, Heka, Bosun, and Prometheus have emerged as open source tools that are very similar to Borgmon’s time-series–based alerting. In particular, Prometheus shares many similarities with Borgmon, especially when you compare the two rule languages. The principles of variable collection and rule evaluation remain the same across all these tools and provide an environment with which you can experiment, and hopefully launch into production 「Site Reliability Engineering」より Prometheus は Borg を参考に作られた時系列ベースのアラートシステム 参考 Site Reliability Engineering https://sre.google/books/
What is Borgmon? Shortly after the job scheduling infrastructure Borg was created in 2003, a new monitoring system—Borgmon—was built to complement it. 「Site Reliability Engineering」より Borgmon は Borg のために作られた監視システム 参考 Site Reliability Engineering https://sre.google/books/
What is Borg? Borg is a distributed cluster operating system, similar to Apache Mesos. Borg manages its jobs at the cluster level. 「Site Reliability Engineering」より Borg は分散のクラスタ管理システム Kubernetes はこの Borg を参考に作られている 参考 Site Reliability Engineering https://sre.google/books/
まずは、SREについて知ろう Every implementation guide needs to start with a common base from which to build. In this case, the basic foundations of SRE include SLOs, monitoring, alerting, toil reduction, and simplicity. Getting these basics right will set you up well to succeed on your SRE journey. 「The Site Reliability Workbook」より SLO, monitoring, alerting の順番になっているので、まずは SLOが何か知ろう 参考 The Site Reliability Engineering Workbook https://sre.google/books/
SLI と SLO • サービスレベル指標：SLI (Service Level Indicator) – サービスの状態を測定するための指標 ( “成功したイベント数 / イベントの総数” がよく用いられている) • 成功したHTTPリクエストの数/ HTTPリクエストの総数（成功率） • 100ミリ秒未満で正常に完了したgRPC呼び出しの数/ gRPCリクエストの合計 • サービスレベル目標：SLO (Service Level Objective) – SLIを用いて定められるサービスの目標値 参考 The Site Reliability Engineering Workbook https://sre.google/books/ Even if you could achieve 100% reliability within your system, your customers would not experience 100% reliability. The chain of systems between you and your customers is often long and complex, and any of these components can fail. 「The Site Reliability Workbook」より 100% is the wrong reliability target for basically everything 「Site Reliability Engineering」より 参考 Site Reliability Engineering https://sre.google/books/
データ収集：Pull型 + Service discovery • Pull型 – Prometheusからターゲットに対して定期的にメトリクスを スクレイプするアーキテクチャを採用している • Service discovery – Pull型の欠点として,ターゲットが追加されるたびに Configの変更が必要なことが上げられるがこの機能により 自動化することができる • OpenMetrics – 対象の名前、値に加えてラベルもつ形式のデータ # HELP http_requests_total The total number of HTTP requests. # TYPE http_requests_total counter http_requests_total{method="post",code="200"} 1027 1395066363000 http_requests_total{method="post",code="400"} 3 1395066363000
データ収集：Exporter （2/2） • app自体がメトリクスを出力 # HELP http_requests_total The total number of HTTP requests. # TYPE http_requests_total counter http_requests_total{method="post",code="200"} 1027 1395066363000 http_requests_total{method="post",code="400"} 3 1395066363000 • exporterがメトリクスを出力 参考 Prometheus Introduction - Julius Volz, Prometheus https://sched.co/Zex4
なぜOpenMetricsが必要なの？ 参考 Prometheus and OpenMetrics, with Richard Hartmann https://kubernetespodcast.com/episode/037-prometheus-and-openmetrics/ OpenMetrics に対応しておけば どの監視ツールでも対応できるようになることが 期待される ADAM GLICK: Given all the work that you're doing with Prometheus, how did that lead you into creating the OpenMetrics Project? RICHARD HARTMANN: Politics. It's really hard for other projects, and especially for other companies, to support something with a different name on it. Even though Prometheus itself doesn't have a profit motive, so we don't have to have sales or anything, it was hard for others to accept stuff which is named after Prometheus in their own product. And this is basically why I decided to do that. • 他の企業が採用しやすいようにわざと切り出している • Prometheusのメトリクスのフォーマットを切り出したものがOpenMetiricsだが 今では、OpenMetiricsで定義してそれをPrometheusで取り込む形にシフトしている
What is Grafana? 参考 Grafana, with Torkel Ödegaard https://kubernetespodcast.com/episode/122-grafana/ So I took Kibana as a starting point when I started working on Grafana. And I really wanted that easy to use dashboard experience, combined with an easy query builder experience. Because that's where many of my teammates struggling, editing, and understanding the graphic queries. Because it's a small text box. It's a long query nested structure. Kubernetes Podcast「Grafana, with Torkel Ödegaard」より • 2014年にGithubにOSSとして公開 • Kibana v3 を参考に作られてる • 簡単にqueryを可視化できることを目指した
どのくらいの期間メトリクスを保管する？ This is a highly contentious point even within Prometheus team. The common wisdom used to be, you retain your data for two weeks, and you drop everything which is older. Personally, I kept all data since late 2015. So data which we collected back then is still available today. The truth is probably somewhere in the middle for most users. If you really care about persisting your data long time, you would probably be looking at something like Cortex or Thanos or Influx Data. Or one of those other tools where we have our remote read/write API, where you can just push data to those other systems and persist data over there. Kubernetes Podcast「Prometheus and OpenMetrics, with Richard Hartmann」より 参考 Prometheus and OpenMetrics, with Richard Hartmann https://kubernetespodcast.com/episode/037-prometheus-and-openmetrics/ 私個人の意見としては 内に保管するメトリクスは 週間分に留めておいて、 それ以上を保管したい場合は、 機能を使うのがおすすめです
監視システムの監視 • 監視システムの設計において、監視システム自身の監視の設計が必要になります • Prometheusでは、Prometheus自身もメトリクスを出力するため それを監視する形になります • 一番シンプルな構成は、Prometheus自身のメトリクスを監視する別のPrometheusを立てることで すが、他にもそのメトリクスだけを社内全体の監視基盤で監視したり、SaaSを使ったりするケースが あります # HELP go_goroutines Number of goroutines that currently exist. # TYPE go_goroutines gauge go_goroutines 289 # HELP go_info Information about the Go environment. # TYPE go_info gauge go_info{version="go1.15.2"} 1 # HELP go_memstats_alloc_bytes Number of bytes allocated and still in use. # TYPE go_memstats_alloc_bytes gauge go_memstats_alloc_bytes 3.63520904e+08 監視専用 一番シンプルな例 毎日誰かが定刻に画面を見てチェックするなどの人力ではなく システムとして設計した方が良いです のメトリクスの例
監視システムのテスト（1/2） 参考 The Site Reliability Engineering Workbook 「The Site Reliability Workbook」より 1. Binary reporting: Check that the exported metric variables change in value under certain conditions as expected. 2. Monitoring configurations: Make sure that rule evaluation produces expected results, and that specific conditions produce the expected alerts. 3. Alerting configurations: Test that generated alerts are routed to a predetermined destination, based on alert label values. メトリクスの正常性 の構成 期待したアラートが 出力されるか 監視システムに当然ながらテストが 必要になります
Amazon Managed Service for Prometheus • Amazon Web Services 側のブログ 参考 プレビュー開始 – Amazon Managed Service for Prometheus (AMP) https://aws.amazon.com/jp/blogs/news/join-the-preview-amazon-managed-service-for-prometheus-amp/ まだプレビューですが今後が楽しみです
Amazon Managed Service for Prometheus • Grafana Labs 側のブログ 参考 Our new partnership with AWS gives Grafana users more options https://grafana.com/blog/2020/12/15/announcing-amazon-managed-service-for-grafana/ をベースに実現しているので、 同じアーキテクチャを使用している にも 進展がないか期待しています
Openmetrics と OpenTelemetry の関係 参考 dd a README entry on how OM relates to the wider CNCF ecosystem https://github.com/OpenObservability/OpenMetrics/issues/137 参考 https://docs.google.com/document/d/17r3BW7-DBtdNNJ_PRvAlrnbj_RsMDkSv_lljOhI03HI/edit#heading=h.k8l30yl2bec4 それぞれの開発者同士で話し合いが行われているようですが、 統合するというよりは、スコープとして重なる部分があるので その辺りの境界を定めていく方向になるのかなと思っています
参考資料（2/7） • ブログ (1/2) – Borgmon 現代芸術美術館で空想にふける < https://cloud.google.com/blog/ja/products/gcp/welcome-to-the-museum-of-modern-borgmon-art > – プレビュー開始 – Amazon Managed Service for Prometheus (AMP) < https://aws.amazon.com/jp/blogs/news/join-the-preview-amazon-managed-service-for-prometheus-amp/ > – A Comprehensive Analysis of Open-Source Time Series Databases < https://www.alibabacloud.com/blog/a-comprehensive-analysis-of-open-source-time-series-databases-4_594733?spm=a2c41.12826437.0.0 > – Loki: Prometheus-inspired, open source logging for cloud natives < https://grafana.com/blog/2018/12/12/loki-prometheus-inspired-open-source-logging-for-cloud-natives/ > – An (only slightly technical) introduction to Loki, the Prometheus-inspired open source logging system < https://grafana.com/blog/2020/05/12/an-only-slightly-technical-introduction-to-loki-the-prometheus-inspired-open-source-logging-system/> – Prometheus vs. Graphite: Which Should You Choose for Time Series or Monitoring? < https://logz.io/blog/prometheus-vs-graphite > – Announcing Grafana Tempo, a massively scalable distributed tracing system < https://grafana.com/blog/2020/10/27/announcing-grafana-tempo-a-massively-scalable-distributed-tracing-system/ >
参考資料（3/7） • ブログ (2/2) – Loki 2.0 released: Transform logs as you’re querying them, and set up alerts within Loki < https://cloud.google.com/blog/ja/products/gcp/welcome-to-the-museum-of-modern-borgmon-art > – ObservabilityCON 2020: Your guide to the newest announcements from Grafana Lab < https://grafana.com/blog/2020/10/26/observabilitycon-2020-your-guide-to-the-newest-announcements-from-grafana-labs/ > – Our new partnership with AWS gives Grafana users more options < https://grafana.com/blog/2020/12/15/announcing-amazon-managed-service-for-grafana/> – Prometheusが利用するOpenMetricsの仕様がIETFに提出された < https://asnokaze.hatenablog.com/entry/2020/11/27/011732 > – An (only slightly technical) introduction to Loki, the Prometheus-inspired open source logging system < https://grafana.com/blog/2020/05/12/an-only-slightly-technical-introduction-to-loki-the-prometheus-inspired-open-source-logging-system/> – Prometheus vs. Graphite: Which Should You Choose for Time Series or Monitoring? < https://logz.io/blog/prometheus-vs-graphite > – Announcing Grafana Tempo, a massively scalable distributed tracing system < https://grafana.com/blog/2020/10/27/announcing-grafana-tempo-a-massively-scalable-distributed-tracing-system/ >
参考資料（4/7） • KubeCon + CloudNativeCon Europe 2019 – Grafana Loki: Like Prometheus, but for Logs - Tom Wilkie, Grafana Labs < https://sched.co/MPbj > • KubeCon + CloudNativeCon Europe 2020 Virtual – Prometheus Introduction - Julius Volz, Prometheus < https://sched.co/Zex4> – Scaling Prometheus: How We Got Some Thanos Into Cortex - Thor Hansen, HashiCorp & Marco Pracucci, Grafana Labs < https://sched.co/Zeuw > – Review and define compatibility or incompatibility between OpenMetrics and OpenTelemetry < https://sched.co/ekFx > – What You Need to Know About OpenMetrics - Brian Brazil, Robust Perception & Richard Hartmann, Grafana Labs < https://sched.co/ZevQ >
参考資料（5/7） • KubeCon + CloudNativeCon North America 2020 Virtual – Evolution of Metric Monitoring and Alerting: Upgrade Your Prometheus Today - Bartlomiej Płotka, Red Hat, Björn Rabenstein & Richard Hartmann, Grafana Labs, & Julius Volz, PromLabs < https://sched.co/ekHn > – Intro to Scaling Prometheus with Cortex - Tom Wilkie, Grafana Labs & Ken Haines, Microsoft < https://sched.co/ekHh > – Observing Cloud Native Observables with the New SIG Observability - Bartlomiej Płotka, Red Hat & Richard Hartmann, Grafana Labs < https://sched.co/ekFx > • ObservabilityCON – Observability with logs & Grafana < https://sched.co/ekHn >
参考資料（7/7） • Kubernetes Podcast – Prometheus and OpenMetrics, with Richard Hartmann < https://kubernetespodcast.com/episode/037-prometheus-and-openmetrics/ > – Grafana, with Torkel Ödegaard < https://kubernetespodcast.com/episode/122-grafana/ > – Monitoring, Metrics and M3, with Martin Mao and Rob Skillington < https://kubernetespodcast.com/episode/084-monitoring-metrics-m3/ >
yosshi_
linedevth
culvert
sugamasao
mploed
antimon2
line_developers
lemon
colopl
k_koheyi
bengo4com
lmi
whywaita
jmmastey
thekraken
lynnandtonic
tammielis
danielanewman
pedronauck
smashingmag
jeffersonlam
marcduiker
frogandcode
sachag
garrettdimon