Tweet
Tweet

Slide 1

Slide 1 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 1/31 Kubernetes Monitoring Introduction 1 / 31

Slide 2

Slide 2 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 2/31 Kyohei Mizumoto(@kyohmizu) C# Software Engineer Interests Docker/Kubernetes Go Security whoami 2 / 31

Slide 3

Slide 3 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 3/31 Required Kubernetesの概要程度の知識 Targets Kubernetes Monitoring未経験者 3 / 31

Slide 4

Slide 4 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 4/31 Contents 1. Monitoring Overview 2. Datadog 3. Prometheus 4. Demo 4 / 31

Slide 5

Slide 5 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 5/31 Monitoring Overview 5 / 31

Slide 6

Slide 6 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 6/31 Monitoring Problem Detection ダッシュボード、アラート Problem Resolution 根本原因の特定、トラブルシュート Continuous Improvement キャパシティ、コスト最適化 6 / 31

Slide 7

Slide 7 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 7/31 Observability A measure of how well internal states of a system can be inferred from knowledge of its external outputs 7 / 31

Slide 8

Slide 8 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 8/31 Landscape 8 / 31

Slide 9

Slide 9 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 9/31 Tools Datadog Prometheus Managed Service Azure Monitor for containers Stackdriver(for GKE) 9 / 31

Slide 10

Slide 10 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 10/31 Tools Datadog Prometheus Managed Service Azure Monitor for containers Stackdriver(for GKE) 10 / 31

Slide 11

Slide 11 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 11/31 Datadog 11 / 31

Slide 12

Slide 12 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 12/31 SaaS型の監視＆解析ツール 課⾦制 様々なサービスのメトリクスをシー ムレスに集約 Datadog 12 / 31

Slide 13

Slide 13 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 13/31 Dashboard 13 / 31

Slide 14

Slide 14 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 14/31 Metrics ホストのCPU使⽤率、ディスク使⽤率 ノード上のコンテナのCPU使⽤率 Deployment のPod数、要求起動数 Jobの成功数、失敗数 etc... 14 / 31

Slide 15

Slide 15 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 15/31 Prometheus 15 / 31

Slide 16

Slide 16 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 16/31 CNCFがホストするOSSの監視ツー ル 時系列でメトリクスを収集 Reliability Prometheus 16 / 31

Slide 17

Slide 17 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 17/31 Architecture pull metrics HDD / SSD Pushgateway Short-lived jobs Jobs / Exporters Storage Retrieval PromQL Prometheus Server Node Service Discovery find targets Prometheus Server Alertmanager push alerts Web UI Grafana API clients PagerDuty Email DNS Kubernetes Consul ... Custom integration notify ... 17 / 31

Slide 18

Slide 18 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 18/31 Architecture Prometheus Server メトリクスの収集(Pull型)、保存 Alert Manager Exporter 要求に応じてメトリクスを送信 Push Gateway メトリクスをPushしておく 18 / 31

Slide 19

Slide 19 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 19/31 Get Started 19 / 31

Slide 20

Slide 20 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 20/31 Use Helm Helmを使⽤してマニフェストを作成 公式サイト︓https://helm.sh/ # Chartのダウンロード $ helm fetch stable/prometheus --version 8.8.0 # Chartを元にマニフェストを作成 $ helm template --name sample-prometheus \ prometheus-8.8.0.tgz \ > sample-prometheus.yaml 20 / 31

Slide 21

Slide 21 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 21/31 Values LBとして外部に公開する場合は、valuesファイルを⽤意 しておく alertmanager: service: type: LoadBalancer pushgateway: service: type: LoadBalancer server: service: type: LoadBalancer 21 / 31

Slide 22

Slide 22 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 22/31 Deploy サンプルをデプロイ https://github.com/Kyohei-M/slide-k8s-monitoring/blob/master/sample- prometheus.yaml # Prometheusを起動 $ kubectl apply -f sample-prometheus.yaml 22 / 31

Slide 23

Slide 23 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 23/31 Demo 23 / 31

Slide 24

Slide 24 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 24/31 Prometheus Server 24 / 31

Slide 25

Slide 25 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 25/31 Prometheus Server 25 / 31

Slide 26

Slide 26 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 26/31 PromQL Prometheus Query Language # Example # apiserverのhttpリクエスト合計 http_requests_total{job="apiserver"} # 直近5分のhttpリクエストの増加率 rate(http_requests_total[5m])[30m:1m] # 直近1時間での空きメモリ量の差 delta(node_memory_MemFree_bytes[1h]) 26 / 31

Slide 27

Slide 27 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 27/31 Grafana OSSのデータ可視化ツール 27 / 31

Slide 28

Slide 28 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 28/31 Grafana helmでインストール $ helm install stable/grafana 28 / 31

Slide 29

Slide 29 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 29/31 Books 29 / 31

Slide 30

Slide 30 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 30/31 Links Monitoring and Observability https://thenewstack.io/monitoring-and-observability-whats-the- difference-and-why-does-it-matter/ Datadog https://www.datadoghq.com/ Prometheus https://prometheus.io/ Grafana https://grafana.com/ 30 / 31

Slide 31

Slide 31 text

2019/3/28 Kubernetes Monitoring Introduction 127.0.0.1:5500/index.html#2 31/31 Thank you! 31 / 31