Share
Embed
k8s_basic_monitoring.pdf
nnao45
July 27, 2018
Transcript
61 slides
-
MY NAME IS NNAO45 ࣗݾհ ˏnnao45 ✓ 26ࡀ(ฏ3ੜ·Ε) ✓ ࠓओʹΫϥυ
✓ ཱڭେֶཧֶ෦ԽֶՊଔۀ ✓ ςχεྺ10 ✓ ITྺ4 ✓ Go࠷ߴ!!zsh࠷ߴ!! ✓cndjpษڧձӡӦ ✓גࣜձࣾαΠόʔΤʔδΣϯτॴଐ ✓ ݩISPͷNW͕ϝΠϯ -
INTRODUCTION OF START UP THE K8S MONITORING. ͱΓ͋͑ͣKUBERNETESͷ෮शͷͭ ͱΓ͋͑ͣͯ͠Δͱ͔ݴ͓͖ͬͯΌ͍͍ײ͕͋ΔόζͬͯΔͭɻ ίϯςφཧʹඞཁͳϚΠΫϩαʔϏε܈ͬͯײ͡ɻ
https://www2.wwt.com/all-blog/kubernetes-101/kubernetes-101-3/ -
INTRODUCTION OF START UP THE K8S MONITORING. ͱΓ͋͑ͣKUBERNETESͷ෮शͷͭ ཁɺίϯςφʔͷΦʔέετϨʔλʔɻ ·ͨͦΕΛୡ͢ΔͨΊͷϚΠΫϩαʔϏε܈ɻ
http://dondocker.com/guardar-nuestras-imagenes-docker/ -
INTRODUCTION OF START UP THE K8S MONITORING. ͱΓ͋͑ͣKUBERNETESͷ෮शͷͭ ImmutableɾɾɾʮෆมʯͳΠϯϑϥɻ Declarationɾɾɾʮએݴʯతઃఆɻ
Self-healingɾɾɾࣗݾճ෮͢ΔγεςϜɻ Micro-serviceɾɾɾAPI࿈ܞʹΑΔίϯϙʔωϯτͷɻ AbstractionɾɾɾநԽʹΑΔϙʔλϏϦςΟɻ kubernetesͷࢥ -
INTRODUCTION OF START UP THE K8S MONITORING. ͱΓ͋͑ͣKUBERNETESͷ෮शͷͭ KubernetesΛΒͤΔɺϗετͷࣄΛࢦ͢ɻ ͪͳΈʹϚελʔϊʔυͱ͔ɺϫʔΧʔϊʔυͱ͔͍͏
දݱઆ໌ͷ্Ͱศར͕ͩ͋·Γਖ਼֬͡Όͳ͍ʢͱࢥͬͯΔʣɻ ίϯςφ ίϯςφ ϙου Node ίϯςφ ίϯςφ ϙου ϊʔυ -
INTRODUCTION OF START UP THE K8S MONITORING. ͱΓ͋͑ͣKUBERNETESͷ෮शͷͭ ̍ͭͷϚελʔϊʔυ(ਖ਼֬ʹҰͷϚελʔϓϩηε)Ͱ ཧ͞ΕΔϫʔΧʔϊʔυʢਖ਼֬ʹҙͷͷϫʔΧʔϓϩηεʣ܈
ίϯςφ ίϯςφ ϙου Cluster ίϯςφ ίϯςφ ϙου ϊʔυ ίϯςφ ίϯςφ ϙου ίϯςφ ίϯςφ ϙου ϊʔυ Ϋϥελʔ -
INTRODUCTION OF START UP THE K8S MONITORING. ͱΓ͋͑ͣKUBERNETESͷ෮शͷͭ ΫϥελʔͷதͰཧతʹϙουΛ ཧ͍ͨ͠ͱ͖ʹ͏୯Ґɻແࢦఆͷ߹defaultωʔϜεϖʔεɻ
ίϯςφ ίϯςφ ϙου NameSpace ίϯςφ ίϯςφ ϙου ϊʔυ ίϯςφ ίϯςφ ϙου ίϯςφ ίϯςφ ϙου ϊʔυ Ϋϥελʔ ίϯςφ ίϯςφ ϙου ίϯςφ ίϯςφ ϙου ωʔϜ εϖʔε ωʔϜ εϖʔε ϊʔυΛ·ͨ͛Δͷͬͯͨʁ -
LOOK AT THE K8S DEFAULT FUNCTION. kubernetesμΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱඋɻ Liveness ʮLiveness probeʯʹΑΔϓϩηεࢹ
ʮReadiness probeʯʹΑΔAPIࢹ Lifecycle Logging Kube-health -
LOOK AT THE K8S DEFAULT FUNCTION. kubernetesμΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱඋɻ Liveness ʮLiveness probeʯʹΑΔϓϩηεࢹ
ʮReadiness probeʯʹΑΔAPIࢹ Lifecycle Logging Kube-health -
ʮLiveness probeʯʹΑΔϓϩηεࢹ LOOK AT THE K8S DEFAULT FUNCTION. apiVersion: v1
kind: Pod ~snip~ livenessProbe: exec: command: - cat - /tmp/healthy initialDelaySeconds: 5 periodSeconds: 5 ~snip~ pods/probe/exec-liveness.yaml ✓ ͦͷίϯςφͷਖ਼ৗੑΛ httpϦΫΤετɺpingɺ ϑΝΠϧଘࡏͳͲͰ֬ೝ͢Δɻ ࣦഊͰ֘pod࠶ىಈ Pod Kubernetes cluster /tmp/healthy -
LOOK AT THE K8S DEFAULT FUNCTION. kubernetesμΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱඋɻ Liveness ʮLiveness probeʯʹΑΔϓϩηεࢹ
ʮReadiness probeʯʹΑΔAPIࢹ Lifecycle Logging Kube-health -
ʮReadiness probeʯʹΑΔ४උঢ়ଶͷࢹ LOOK AT THE K8S DEFAULT FUNCTION. apiVersion: v1
kind: Pod ~snip~ readinessProbe: tcpSocket: port: 8080 initialDelaySeconds: 40 periodSeconds: 10 ~snip~ pods/probe/exec-readiness.yaml ✓ ͦͷίϯςφ͕࠷ॳ ϦΫΤετΛड͚͚ͳ͍ঢ়ଶͰ ىಈͯ͠ɺReadiness͕OKޙɺ ϦΫΤετΛड͚ΔΑ͏ʹ͢Δɻ -
LOOK AT THE K8S DEFAULT FUNCTION. kubernetesμΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱඋɻ Liveness Lifecycle ʮReplica
SetʯʹΑΔ߹ܭPodཧɻ ʮDaemon SetʯʹΑΔper nodeͳPodࢹ Logging Kube-health -
LOOK AT THE K8S DEFAULT FUNCTION. kubernetesμΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱඋɻ Liveness Lifecycle ʮReplica
SetʯʹΑΔ߹ܭPodཧɻ ʮDaemon SetʯʹΑΔper nodeͳPodࢹ Logging Kube-health -
ʮReplica SetʯʹΑΔ߹ܭPodཧɻ LOOK AT THE K8S DEFAULT FUNCTION. Kubernetes Cluster
Worker Node Pod Pod Pod Worker Node ✓ʮશ෦Ͱ3ݸɺSlaveͷDB ͕ཉ͍͠ʯ࣌ͳͲʹ͏ɻ apiVersion: apps/v1 kind: ReplicaSet metadata: name: frontend labels: app: guestbook tier: frontend spec: # this replicas value is default # modify it according to your case replicas: 3 ~snip~ controllers/frontend.yaml -
LOOK AT THE K8S DEFAULT FUNCTION. Kubernetes Cluster Worker Node
Worker Node ෛՙ͕গͳ͍Node͔Β શ෦ͰPodΛ̏ݸ࡞Ζ͏ɻ ReplicaSet Controller DEPLOY DEPLOY DEPLOY apiVersion: apps/v1 kind: ReplicaSet metadata: name: frontend labels: app: guestbook tier: frontend spec: # this replicas value is default # modify it according to your case replicas: 3 ~snip~ controllers/frontend.yaml ʮReplica SetʯʹΑΔ߹ܭPodཧɻ -
LOOK AT THE K8S DEFAULT FUNCTION. ReplicaSet Controller Kubernetes Cluster
Worker Node Pod Pod Pod Worker Node શ෦Ͱ3ݸ͋Δ͔ΒOK apiVersion: apps/v1 kind: ReplicaSet metadata: name: frontend labels: app: guestbook tier: frontend spec: # this replicas value is default # modify it according to your case replicas: 3 ~snip~ controllers/frontend.yaml ʮReplica SetʯʹΑΔ߹ܭPodཧɻ -
LOOK AT THE K8S DEFAULT FUNCTION. ReplicaSet Controller Kubernetes Cluster
Worker Node Pod Pod Pod Worker Node ͋ɺ̍ݸམͪͨʂ apiVersion: apps/v1 kind: ReplicaSet metadata: name: frontend labels: app: guestbook tier: frontend spec: # this replicas value is default # modify it according to your case replicas: 3 ~snip~ controllers/frontend.yaml ʮReplica SetʯʹΑΔ߹ܭPodཧɻ -
LOOK AT THE K8S DEFAULT FUNCTION. apiVersion: apps/v1 kind: ReplicaSet
metadata: name: frontend labels: app: guestbook tier: frontend spec: # this replicas value is default # modify it according to your case replicas: 3 ~snip~ controllers/frontend.yaml ReplicaSet Controller Kubernetes Cluster Worker Node Pod Pod Pod Worker Node ̏ݸΛΩʔϓ͠ͳ͖Όɻ ̍ݸσϓϩΠʔʂ DEPLOY ʮReplica SetʯʹΑΔ߹ܭPodཧɻ -
LOOK AT THE K8S DEFAULT FUNCTION. apiVersion: apps/v1 kind: ReplicaSet
metadata: name: frontend labels: app: guestbook tier: frontend spec: # this replicas value is default # modify it according to your case replicas: 3 ~snip~ controllers/frontend.yaml ReplicaSet Controller Kubernetes Cluster Worker Node Pod Pod Pod Worker Node શ෦Ͱ3ݸ͋Δ͔ΒOK Pod ʮReplica SetʯʹΑΔ߹ܭPodཧɻ -
LOOK AT THE K8S DEFAULT FUNCTION. kubernetesμΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱඋɻ Liveness Lifecycle ʮReplica
SetʯʹΑΔ߹ܭPodཧɻ ʮDaemon SetʯʹΑΔper nodeͳPodࢹ Logging Kube-health -
ʮDaemon SetʯʹΑΔper nodeͳPodࢹ LOOK AT THE K8S DEFAULT FUNCTION. Kubernetes
Cluster Worker Node apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd-elasticsearch namespace: kube-system labels: k8s-app: log ~snip~ controllers/daemonset.yaml Worker Node Worker Node log log Pod Pod ✓ʮಛఆͷϊʔυʹ̍ݸͣͭɺ ϩάͷPod͕ཉ͍͠ʯͳͲ ͷ࣌ʹ͏ɻ -
ʮDaemon SetʯʹΑΔper nodeͳPodࢹ LOOK AT THE K8S DEFAULT FUNCTION. Kubernetes
Cluster Worker Node tagͰࢦఆͨ͠ϊʔυʹ ̍ݸͣͭͰσϓϩΠɻ ReplicaSet Controller DEPLOY DEPLOY apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd-elasticsearch namespace: kube-system labels: k8s-app: log ~snip~ controllers/daemonset.yaml Worker Node log log Worker Node -
ʮDaemon SetʯʹΑΔper nodeͳPodࢹ LOOK AT THE K8S DEFAULT FUNCTION. Kubernetes
Cluster Worker Node tagͰࢦఆͨ͠ϊʔυʹ ̍ݸͣͭ͋Δ͔ΒOKɻ ReplicaSet Controller apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd-elasticsearch namespace: kube-system labels: k8s-app: log ~snip~ controllers/daemonset.yaml Worker Node log log Pod Pod Worker Node -
σϞΓ·͢ɻ LOOK AT THE K8S DEFAULT FUNCTION. Kubernetes Cluster K8s-node-001
K8s-node-002 K8s-node-003 Master Worker Worker K8s-lab-001 VIP (LB) Metallb Metallb Master -
σϞᶃ LOOK AT THE K8S DEFAULT FUNCTION. Kubernetes Cluster K8s-node-001
K8s-node-002 K8s-node-003 Master Worker Worker K8s-lab-001 VIP (LB) Metallb Metallb Master ̍ճୟ͔Ε ͨΒࢮ͵ -
σϞᶄ LOOK AT THE K8S DEFAULT FUNCTION. Kubernetes Cluster K8s-node-001
K8s-node-002 K8s-node-003 Master Worker Worker K8s-lab-001 VIP (LB) Metallb Metallb Master 10ඵ ܦͬͨΒαʔ Ϗε։࢝ 10ඵ ܦͬͨΒαʔ Ϗε։࢝ -
LOOK AT THE K8S DEFAULT FUNCTION. kubernetesμΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱඋɻ Liveness Lifecycle Logging
ίϯςφϨϕϧͷpod͔ΒͷϩΪϯά ϊʔυϨϕϧͷpod͔ΒͷϩΪϯά ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά Kube-health -
LOOK AT THE K8S DEFAULT FUNCTION. kubernetesμΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱඋɻ Liveness Lifecycle Logging
ίϯςφϨϕϧͷpod͔ΒͷϩΪϯά ϊʔυϨϕϧͷpod͔ΒͷϩΪϯά ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά Kube-health -
ίϯςφϨϕϧͷpod͔ΒͷϩΪϯά LOOK AT THE K8S DEFAULT FUNCTION. apiVersion: v1 kind:
Pod metadata: name: counter spec: containers: - name: count image: busybox args: [/bin/sh, -c, 'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done'] debug/counter-pod.yaml Pod ✓ Ұ൪جຊతͳϩΪϯάͰɺ kubectl logsͰݟΔ Έͨͩpodsͷstdoutͱ stderrΛݟͯΔ͚ͩɻ Container stdout stderr ʼKUBECTL GET LOGS -
LOOK AT THE K8S DEFAULT FUNCTION. kubernetesμΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱඋɻ Liveness Lifecycle Logging
ίϯςφϨϕϧͷpod͔ΒͷϩΪϯά ϊʔυϨϕϧͷpod͔ΒͷϩΪϯά ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά Statement -
Node ϊʔυϨϕϧͷpod͔ΒͷϩΪϯά LOOK AT THE K8S DEFAULT FUNCTION. Pod ✓ϊʔυ୯ҐͰݟΔ߹ɺ
pod͔Βഉग़͞ΕΔstdoutͱ stderrΛॴଐ͢Δϊʔυͷ ϑΝΠϧʹॻ͖ग़͓ͯ͘͠ɻ Container stdout stderr stdout stderr Pod Container log rotate archive ✓ ͜ͷΑ͏ʹߏஙͨ͠߹ɺ ϩά͔ͳΓͷྔʹͳΔࣄ͕ ଟ͍ͷͰɺಉ࣌ʹϩάͷ ϩʔςʔτͷΈΛ࣮ ͓ͯ͘͠ࣄ͕΄΅ඞਢɻ -
LOOK AT THE K8S DEFAULT FUNCTION. kubernetesμΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱඋɻ Liveness Lifecycle Logging
ίϯςφϨϕϧͷpod͔ΒͷϩΪϯά ϊʔυϨϕϧͷpod͔ΒͷϩΪϯά ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά Kube-health -
ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά LOOK AT THE K8S DEFAULT FUNCTION. Pod ✓ ͖ͬΓݴͬͯɺ
Ϋϥελ୯ҐͰϩάΛݟ͍ͨ ߹Ͳ͏͢Δ͔࣮ʹΑΔɻ Container Pod Container ✓ ͱݴͬͯɺͦΕͰ ऴΘΓऐ͍͠ͷͰ දతͳϩΪϯάγεςϜ Λݟ͍͖ͯ·͠ΐ͏ Pod Container Pod Container Kubernetes Cluster ? -
Node ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά LOOK AT THE K8S DEFAULT FUNCTION. ✓ ϊʔυϩΪϯάΤʔδΣϯτํࣜ
Pod Container stdout stderr stdout stderr Pod Container Node Pod Container stdout stderr stdout stderr Pod Container Log agent log Backend ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά -
INTRODUCTION OF START UP THE K8S MONITORING. ͱΓ͋͑ͣKUBERNETESͷ෮शͷͭ Kubernetes্ͷpodͰɺ ҟͳΔίϯςφ͕ಉ͡σΟεΫʢྫ͑ϩάసૹίϯςϯπͷڞ༗ʣ
Λڞ༗͍ͨ͠߹ʹ͏ͭɻ Sidecarίϯςφ ίϯςφ ίϯςφ ϙου ϊʔυ σΟεΫ -
Node log Backend ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά LOOK AT THE K8S DEFAULT FUNCTION.
✓ ετϦʔϛϯάαΠυΧʔํࣜ Pod app streaming a.log node.log Pod app streaming b.log Pod app streaming node.log Pod app streaming Node a.log b.log stdout stderr stdout stderr stdout stderr stdout stderr Log agent -
Node ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά LOOK AT THE K8S DEFAULT FUNCTION. Pod App
log Backend ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά ✓ ϙουΠϯΤʔδΣϯτํࣜ˞ sidecar Log agent a.log Pod App sidecar Log agent b.log streaming streaming ※kubectl logs͔ΒϩάΛݟΔࣄ͕Ͱ͖ͳ͘ͳΔߏͰ͢ɻ -
LOOK AT THE K8S DEFAULT FUNCTION. kubernetesμΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱඋɻ Liveness Lifecycle Logging
Kube-health kubeࣗͷॏཁίϯϙʔωϯτͷࢹ ͦͷଞkube-systemͷࢹ HeapsterʹΑΔϦιʔεࢹ -
LOOK AT THE K8S DEFAULT FUNCTION. kubernetesμΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱඋɻ Liveness Lifecycle Logging
Kube-health kubeࣗͷॏཁίϯϙʔωϯτͷࢹ ͦͷଞkube-systemͷࢹ HeapsterʹΑΔϦιʔεࢹ -
kubeࣗͷॏཁίϯϙʔωϯτͷࢹ ᶃ kubelet LOOK AT THE K8S DEFAULT FUNCTION. ✓
kubernetes্ͰίϯςφΛಈ͔͢ϫʔΧϓϩηεͷ৺ଁ෦ɻ ✓ docker,rktlet,containerdͷϥϯλΠϜͱີͳؔʹ ίϯςφͱͯ͠ॴ༗͢Δ෦IPͷpingࢹ ίϯςφϥϯλΠϜͷσʔϞϯͷϓϩηεࢹ kubeletίϯςφ͕crashLoopͯ͠ͳ͍͔Ͳ͏͔ -
kubeࣗͷॏཁίϯϙʔωϯτͷࢹ ᶄ etcd LOOK AT THE K8S DEFAULT FUNCTION. ✓
kubernetes্ͰߏใΛอଘ͢ΔࢄKVSɻ ✓ etcd͕࣋ͭಠࣗͷΤϯυϙΠϯτΛࢹ Listenͯ͠Δ(:2379)ϙʔτͷࢹ ίϯςφͱͯ͠ॴ༗͢Δ෦IPͷpingࢹ /stats͔Β throughput, success, error, latencyϝτϦΫε /metrcs͔Βdisk ΦϖϨʔγϣϯ,gRPC ͳͲ෦ͷϝτϦΫε etcdίϯςφ͕crashLoopͯ͠ͳ͍͔Ͳ͏͔ -
LOOK AT THE K8S DEFAULT FUNCTION. kubernetesμΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱඋɻ Liveness Lifecycle Logging
Kube-health kubeࣗͷॏཁίϯϙʔωϯτͷࢹ ͦͷଞkube-systemͷࢹ metrics-serverʹΑΔϦιʔεࢹ -
ͦͷଞkube-systemͷࢹ ᶃ kube-apiserver ᶄ kube-schedulerɾɾɾͳͲ LOOK AT THE K8S DEFAULT
FUNCTION. ✓ ͍ΘΏΔϚελʔϊʔυͰಈ͍͍ͯΔϓϩηε ✓ σϑΥϧτͩͱ໊લۭؒkube-systemͰಈ͍͍ͯΔίϯςφ ίϯςφͱͯ͠ॴ༗͢Δ෦IPͷpingࢹ ίϯςφ͕crashLoopͯ͠ͳ͍͔Ͳ͏͔ ͱΓ໊͋͑ͣલۭؒkube-systemͷίϯςφݟ͓ͯ͘ -
LOOK AT THE K8S DEFAULT FUNCTION. kubernetesμΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱඋɻ Liveness Lifecycle Logging
Kube-health kubeࣗͷॏཁίϯϙʔωϯτͷࢹ ͦͷଞkube-systemͷࢹ metrics-serverʹΑΔϦιʔεࢹ -
LOOK AT THE K8S DEFAULT FUNCTION. ✓ kube1.7͘Β͍෦ͷϦιʔεࢹʮHeapsterʯͱ͍͏ ίϯϙʔωϯτ͕͍ͬͯͨɻ ✓
kube1.8͔Βʮmetrics-serverʯͱ͍͏ίϯϙʔωϯτʹஔ͖ ΘΓɺલड़ͨ͠ϙϯνֆͷΑ͏ͳΞʔΩςΫνϟʹͳΓɺΑ ΓϓϥΨϒϧͳΞʔΩςΫτʹɻ ✓ʮHeapsterʯطʹඇਪͳίϯϙʔωϯτɻʮmetrics- serverʯʹΑΔʮOSS monitoring architectureʯͷํͰษڧΛ ਐΊ͍͖ͯ·͠ΐ͏(^p^) metrics-serverʹΑΔϦιʔεࢹ -
Master Node LOOK AT THE K8S DEFAULT FUNCTION. Worker Node
Worker Node kubelet cAdvisor metrics- server Worker Node kubelet cAdvisor metrics.k8s.io api-server API CLIENT GET resource usage resource usage get data get data resource metrics pipelineʹΑΔجຊతͳϦιʔεऔಘ metrics-serverʹΑΔϦιʔεࢹ -
LOOK AT THE K8S DEFAULT FUNCTION. full monitoring pipelineʹΑΔϦονͳΧελϜϦιʔεऔಘ prometheusͱ͔͕͜ΕʹରԠ
metrics-serverʹΑΔϦιʔεࢹ ? https://www.weave.works/blog/kubernetes-horizontal-pod-autoscaler-and-prometheus