Upgrade to Pro — share decks privately, control downloads, hide ads and more …

k8s_basic_monitoring.pdf

nnao45
July 27, 2018
1.6k

 k8s_basic_monitoring.pdf

nnao45

July 27, 2018
Tweet

Transcript

  1. MY NAME IS NNAO45 ࣗݾ঺հ ˏnnao45 ✓ 26ࡀ(ฏ੒3೥ੜ·Ε) ✓ ࠓ͸ओʹΫϥ΢υ԰

    ✓ ཱڭେֶཧֶ෦ԽֶՊଔۀ ✓ ςχεྺ10೥ ✓ ITྺ4೥໨ ✓ Go࠷ߴ!!zsh࠷ߴ!! ✓cndjpษڧձӡӦ ✓גࣜձࣾαΠόʔΤʔδΣϯτॴଐ ✓ ݩISPͷNW͕ϝΠϯ
  2. INTRODUCTION OF START UP THE K8S MONITORING. ͱΓ͋͑ͣKUBERNETESͷ෮शͷ΍ͭ ImmutableɾɾɾʮෆมʯͳΠϯϑϥɻ Declarationɾɾɾʮએݴʯతઃఆɻ

    Self-healingɾɾɾࣗݾճ෮͢ΔγεςϜɻ Micro-serviceɾɾɾAPI࿈ܞʹΑΔίϯϙʔωϯτͷ෼཭ɻ Abstractionɾɾɾந৅ԽʹΑΔϙʔλϏϦςΟɻ kubernetesͷࢥ૝
  3. INTRODUCTION OF START UP THE K8S MONITORING. ͱΓ͋͑ͣKUBERNETESͷ෮शͷ΍ͭ KubernetesΛ૸ΒͤΔɺϗετͷࣄΛࢦ͢ɻ ͪͳΈʹϚελʔϊʔυͱ͔ɺϫʔΧʔϊʔυͱ͔͍͏

    දݱ͸આ໌ͷ্Ͱ͸ศར͕ͩ͋·Γਖ਼֬͡Όͳ͍ʢͱࢥͬͯΔʣɻ ίϯςφ ίϯςφ ϙου Node ίϯςφ ίϯςφ ϙου ϊʔυ
  4. INTRODUCTION OF START UP THE K8S MONITORING. ͱΓ͋͑ͣKUBERNETESͷ෮शͷ΍ͭ ΫϥελʔͷதͰ࿦ཧతʹϙουΛ ؅ཧ͍ͨ͠ͱ͖ʹ࢖͏୯Ґɻແࢦఆͷ৔߹defaultωʔϜεϖʔεɻ

    ίϯςφ ίϯςφ ϙου NameSpace ίϯςφ ίϯςφ ϙου ϊʔυ ίϯςφ ίϯςφ ϙου ίϯςφ ίϯςφ ϙου ϊʔυ Ϋϥελʔ ίϯςφ ίϯςφ ϙου ίϯςφ ίϯςφ ϙου ωʔϜ εϖʔε ωʔϜ εϖʔε ϊʔυΛ·ͨ͛Δͷ஌ͬͯͨʁ
  5. ʮLiveness probeʯʹΑΔϓϩηε؂ࢹ LOOK AT THE K8S DEFAULT FUNCTION. apiVersion: v1

    kind: Pod ~snip~ livenessProbe: exec: command: - cat - /tmp/healthy initialDelaySeconds: 5 periodSeconds: 5 ~snip~ pods/probe/exec-liveness.yaml ✓ ͦͷίϯςφͷਖ਼ৗੑΛ httpϦΫΤετ΍ɺpingɺ ϑΝΠϧଘࡏͳͲͰ֬ೝ͢Δɻ ࣦഊͰ֘౰pod࠶ىಈ Pod Kubernetes cluster /tmp/healthy
  6. ʮReadiness probeʯʹΑΔ४උঢ়ଶͷ؂ࢹ LOOK AT THE K8S DEFAULT FUNCTION. apiVersion: v1

    kind: Pod ~snip~ readinessProbe: tcpSocket: port: 8080 initialDelaySeconds: 40 periodSeconds: 10 ~snip~ pods/probe/exec-readiness.yaml ✓ ͦͷίϯςφ͕࠷ॳ͸ ϦΫΤετΛड͚෇͚ͳ͍ঢ়ଶͰ ىಈͯ͠ɺReadiness͕OKޙɺ ϦΫΤετΛड͚ΔΑ͏ʹ͢Δɻ
  7. LOOK AT THE K8S DEFAULT FUNCTION. kubernetes͸μΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱ૷උɻ Liveness Lifecycle ʮReplica

    SetʯʹΑΔ߹ܭPod਺؅ཧɻ ʮDaemon SetʯʹΑΔper nodeͳPod਺؂ࢹ Logging Kube-health
  8. LOOK AT THE K8S DEFAULT FUNCTION. kubernetes͸μΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱ૷උɻ Liveness Lifecycle ʮReplica

    SetʯʹΑΔ߹ܭPod਺؅ཧɻ ʮDaemon SetʯʹΑΔper nodeͳPod਺؂ࢹ Logging Kube-health
  9. ʮReplica SetʯʹΑΔ߹ܭPod਺؅ཧɻ LOOK AT THE K8S DEFAULT FUNCTION. Kubernetes Cluster

    Worker Node Pod Pod Pod Worker Node ✓ʮશ෦Ͱ3ݸɺSlaveͷDB ͕ཉ͍͠ʯ࣌ͳͲʹ࢖͏ɻ apiVersion: apps/v1 kind: ReplicaSet metadata: name: frontend labels: app: guestbook tier: frontend spec: # this replicas value is default # modify it according to your case replicas: 3 ~snip~ controllers/frontend.yaml
  10. LOOK AT THE K8S DEFAULT FUNCTION. Kubernetes Cluster Worker Node

    Worker Node ෛՙ͕গͳ͍Node͔Β શ෦ͰPodΛ̏ݸ࡞Ζ͏ɻ ReplicaSet Controller DEPLOY DEPLOY DEPLOY apiVersion: apps/v1 kind: ReplicaSet metadata: name: frontend labels: app: guestbook tier: frontend spec: # this replicas value is default # modify it according to your case replicas: 3 ~snip~ controllers/frontend.yaml ʮReplica SetʯʹΑΔ߹ܭPod਺؅ཧɻ
  11. LOOK AT THE K8S DEFAULT FUNCTION. ReplicaSet Controller Kubernetes Cluster

    Worker Node Pod Pod Pod Worker Node શ෦Ͱ3ݸ͋Δ͔ΒOK apiVersion: apps/v1 kind: ReplicaSet metadata: name: frontend labels: app: guestbook tier: frontend spec: # this replicas value is default # modify it according to your case replicas: 3 ~snip~ controllers/frontend.yaml ʮReplica SetʯʹΑΔ߹ܭPod਺؅ཧɻ
  12. LOOK AT THE K8S DEFAULT FUNCTION. ReplicaSet Controller Kubernetes Cluster

    Worker Node Pod Pod Pod Worker Node ͋ɺ̍ݸམͪͨʂ apiVersion: apps/v1 kind: ReplicaSet metadata: name: frontend labels: app: guestbook tier: frontend spec: # this replicas value is default # modify it according to your case replicas: 3 ~snip~ controllers/frontend.yaml ʮReplica SetʯʹΑΔ߹ܭPod਺؅ཧɻ
  13. LOOK AT THE K8S DEFAULT FUNCTION. apiVersion: apps/v1 kind: ReplicaSet

    metadata: name: frontend labels: app: guestbook tier: frontend spec: # this replicas value is default # modify it according to your case replicas: 3 ~snip~ controllers/frontend.yaml ReplicaSet Controller Kubernetes Cluster Worker Node Pod Pod Pod Worker Node ̏ݸΛΩʔϓ͠ͳ͖Όɻ ̍ݸσϓϩΠʔʂ DEPLOY ʮReplica SetʯʹΑΔ߹ܭPod਺؅ཧɻ
  14. LOOK AT THE K8S DEFAULT FUNCTION. apiVersion: apps/v1 kind: ReplicaSet

    metadata: name: frontend labels: app: guestbook tier: frontend spec: # this replicas value is default # modify it according to your case replicas: 3 ~snip~ controllers/frontend.yaml ReplicaSet Controller Kubernetes Cluster Worker Node Pod Pod Pod Worker Node શ෦Ͱ3ݸ͋Δ͔ΒOK Pod ʮReplica SetʯʹΑΔ߹ܭPod਺؅ཧɻ
  15. LOOK AT THE K8S DEFAULT FUNCTION. kubernetes͸μΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱ૷උɻ Liveness Lifecycle ʮReplica

    SetʯʹΑΔ߹ܭPod਺؅ཧɻ ʮDaemon SetʯʹΑΔper nodeͳPod਺؂ࢹ Logging Kube-health
  16. ʮDaemon SetʯʹΑΔper nodeͳPod਺؂ࢹ LOOK AT THE K8S DEFAULT FUNCTION. Kubernetes

    Cluster Worker Node apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd-elasticsearch namespace: kube-system labels: k8s-app: log ~snip~ controllers/daemonset.yaml Worker Node Worker Node log log Pod Pod ✓ʮಛఆͷϊʔυʹ̍ݸͣͭɺ ϩάͷPod͕ཉ͍͠ʯͳͲ ͷ࣌ʹ࢖͏ɻ
  17. ʮDaemon SetʯʹΑΔper nodeͳPod਺؂ࢹ LOOK AT THE K8S DEFAULT FUNCTION. Kubernetes

    Cluster Worker Node tagͰࢦఆͨ͠ϊʔυʹ ̍ݸͣͭͰσϓϩΠɻ ReplicaSet Controller DEPLOY DEPLOY apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd-elasticsearch namespace: kube-system labels: k8s-app: log ~snip~ controllers/daemonset.yaml Worker Node log log Worker Node
  18. ʮDaemon SetʯʹΑΔper nodeͳPod਺؂ࢹ LOOK AT THE K8S DEFAULT FUNCTION. Kubernetes

    Cluster Worker Node tagͰࢦఆͨ͠ϊʔυʹ ̍ݸͣͭ͋Δ͔ΒOKɻ ReplicaSet Controller apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd-elasticsearch namespace: kube-system labels: k8s-app: log ~snip~ controllers/daemonset.yaml Worker Node log log Pod Pod Worker Node
  19. σϞΓ·͢ɻ LOOK AT THE K8S DEFAULT FUNCTION. Kubernetes Cluster K8s-node-001

    K8s-node-002 K8s-node-003 Master Worker Worker K8s-lab-001 VIP (LB) Metallb Metallb Master
  20. σϞᶃ LOOK AT THE K8S DEFAULT FUNCTION. Kubernetes Cluster K8s-node-001

    K8s-node-002 K8s-node-003 Master Worker Worker K8s-lab-001 VIP (LB) Metallb Metallb Master ̍ճୟ͔Ε ͨΒࢮ͵
  21. σϞᶄ LOOK AT THE K8S DEFAULT FUNCTION. Kubernetes Cluster K8s-node-001

    K8s-node-002 K8s-node-003 Master Worker Worker K8s-lab-001 VIP (LB) Metallb Metallb Master 10ඵ ܦͬͨΒαʔ Ϗε։࢝ 10ඵ ܦͬͨΒαʔ Ϗε։࢝
  22. LOOK AT THE K8S DEFAULT FUNCTION. kubernetes͸μΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱ૷උɻ Liveness Lifecycle Logging

    ίϯςφϨϕϧͷpod͔ΒͷϩΪϯά ϊʔυϨϕϧͷpod͔ΒͷϩΪϯά ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά Kube-health
  23. LOOK AT THE K8S DEFAULT FUNCTION. kubernetes͸μΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱ૷උɻ Liveness Lifecycle Logging

    ίϯςφϨϕϧͷpod͔ΒͷϩΪϯά ϊʔυϨϕϧͷpod͔ΒͷϩΪϯά ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά Kube-health
  24. ίϯςφϨϕϧͷpod͔ΒͷϩΪϯά LOOK AT THE K8S DEFAULT FUNCTION. apiVersion: v1 kind:

    Pod metadata: name: counter spec: containers: - name: count image: busybox args: [/bin/sh, -c, 'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done'] debug/counter-pod.yaml Pod ✓ Ұ൪جຊతͳϩΪϯάͰɺ kubectl logsͰݟΔ ࢓૊Έ͸ͨͩpodsͷstdoutͱ stderrΛݟͯΔ͚ͩɻ Container stdout stderr ʼKUBECTL GET LOGS
  25. LOOK AT THE K8S DEFAULT FUNCTION. kubernetes͸μΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱ૷උɻ Liveness Lifecycle Logging

    ίϯςφϨϕϧͷpod͔ΒͷϩΪϯά ϊʔυϨϕϧͷpod͔ΒͷϩΪϯά ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά Statement
  26. Node ϊʔυϨϕϧͷpod͔ΒͷϩΪϯά LOOK AT THE K8S DEFAULT FUNCTION. Pod ✓ϊʔυ୯ҐͰݟΔ৔߹͸ɺ

    pod͔Βഉग़͞ΕΔstdoutͱ stderrΛॴଐ͢Δϊʔυͷ ϑΝΠϧʹॻ͖ग़͓ͯ͘͠ɻ Container stdout stderr stdout stderr Pod Container log rotate archive ✓ ͜ͷΑ͏ʹߏஙͨ͠৔߹ɺ ϩά͸͔ͳΓͷྔʹͳΔࣄ͕ ଟ͍ͷͰɺಉ࣌ʹϩάͷ ϩʔςʔτͷ࢓૊ΈΛ࣮૷ ͓ͯ͘͠ࣄ͕΄΅ඞਢɻ
  27. LOOK AT THE K8S DEFAULT FUNCTION. kubernetes͸μΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱ૷උɻ Liveness Lifecycle Logging

    ίϯςφϨϕϧͷpod͔ΒͷϩΪϯά ϊʔυϨϕϧͷpod͔ΒͷϩΪϯά ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά Kube-health
  28. ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά LOOK AT THE K8S DEFAULT FUNCTION. Pod ✓ ͸͖ͬΓݴͬͯɺ

    Ϋϥελ୯ҐͰϩάΛݟ͍ͨ ৔߹Ͳ͏͢Δ͔͸࣮૷ʹΑΔɻ Container Pod Container ✓ ͱ͸ݴͬͯ΋ɺͦΕͰ ऴΘΓ͸ऐ͍͠ͷͰ ୅දతͳϩΪϯάγεςϜ Λݟ͍͖ͯ·͠ΐ͏ Pod Container Pod Container Kubernetes Cluster ?
  29. Node ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά LOOK AT THE K8S DEFAULT FUNCTION. ✓ ϊʔυϩΪϯάΤʔδΣϯτํࣜ

    Pod Container stdout stderr stdout stderr Pod Container Node Pod Container stdout stderr stdout stderr Pod Container Log agent log Backend ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά
  30. Node log Backend ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά LOOK AT THE K8S DEFAULT FUNCTION.

    ✓ ετϦʔϛϯάαΠυΧʔํࣜ Pod app streaming a.log node.log Pod app streaming b.log Pod app streaming node.log Pod app streaming Node a.log b.log stdout stderr stdout stderr stdout stderr stdout stderr Log agent
  31. Node ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά LOOK AT THE K8S DEFAULT FUNCTION. Pod App

    log Backend ΫϥελϨϕϧͷpod͔ΒͷϩΪϯά ✓ ϙουΠϯΤʔδΣϯτํࣜ˞ sidecar Log agent a.log Pod App sidecar Log agent b.log streaming streaming ※kubectl logs͔ΒϩάΛݟΔࣄ͕Ͱ͖ͳ͘ͳΔߏ੒Ͱ͢ɻ
  32. LOOK AT THE K8S DEFAULT FUNCTION. kubernetes͸μΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱ૷උɻ Liveness Lifecycle Logging

    Kube-health kubeࣗ਎ͷॏཁίϯϙʔωϯτͷ؂ࢹ ͦͷଞkube-systemͷ؂ࢹ HeapsterʹΑΔϦιʔε؂ࢹ
  33. LOOK AT THE K8S DEFAULT FUNCTION. kubernetes͸μΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱ૷උɻ Liveness Lifecycle Logging

    Kube-health kubeࣗ਎ͷॏཁίϯϙʔωϯτͷ؂ࢹ ͦͷଞkube-systemͷ؂ࢹ HeapsterʹΑΔϦιʔε؂ࢹ
  34. kubeࣗ਎ͷॏཁίϯϙʔωϯτͷ؂ࢹ ᶃ kubelet LOOK AT THE K8S DEFAULT FUNCTION. ✓

    kubernetes্ͰίϯςφΛಈ͔͢ϫʔΧϓϩηεͷ৺ଁ෦ɻ ✓ docker,rktlet,containerd౳ͷϥϯλΠϜͱີ઀ͳؔ܎ʹ஫໨ ίϯςφͱͯ͠ॴ༗͢Δ಺෦IP΁ͷping؂ࢹ ίϯςφϥϯλΠϜͷσʔϞϯͷϓϩηε؂ࢹ kubeletίϯςφ͕crashLoopͯ͠ͳ͍͔Ͳ͏͔
  35. kubeࣗ਎ͷॏཁίϯϙʔωϯτͷ؂ࢹ ᶄ etcd LOOK AT THE K8S DEFAULT FUNCTION. ✓

    kubernetes্Ͱߏ੒৘ใΛอଘ͢Δ෼ࢄKVSɻ ✓ etcd͕࣋ͭಠࣗͷΤϯυϙΠϯτΛ؂ࢹ Listenͯ͠Δ(:2379౳)ϙʔτͷ؂ࢹ ίϯςφͱͯ͠ॴ༗͢Δ಺෦IP΁ͷping؂ࢹ /stats͔Β throughput, success, error, latencyϝτϦΫε /metrcs͔Βdisk ΦϖϨʔγϣϯ,gRPC ͳͲ಺෦ͷϝτϦΫε etcdίϯςφ͕crashLoopͯ͠ͳ͍͔Ͳ͏͔
  36. LOOK AT THE K8S DEFAULT FUNCTION. kubernetes͸μΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱ૷උɻ Liveness Lifecycle Logging

    Kube-health kubeࣗ਎ͷॏཁίϯϙʔωϯτͷ؂ࢹ ͦͷଞkube-systemͷ؂ࢹ metrics-serverʹΑΔϦιʔε؂ࢹ
  37. ͦͷଞkube-systemͷ؂ࢹ ᶃ kube-apiserver ᶄ kube-schedulerɾɾɾͳͲ LOOK AT THE K8S DEFAULT

    FUNCTION. ✓ ͍ΘΏΔϚελʔϊʔυͰಈ͍͍ͯΔϓϩηε ✓ σϑΥϧτͩͱ໊લۭؒkube-systemͰಈ͍͍ͯΔίϯςφ ίϯςφͱͯ͠ॴ༗͢Δ಺෦IP΁ͷping؂ࢹ ίϯςφ͕crashLoopͯ͠ͳ͍͔Ͳ͏͔ ͱΓ໊͋͑ͣલۭؒkube-systemͷίϯςφ਺͸ݟ͓ͯ͘
  38. LOOK AT THE K8S DEFAULT FUNCTION. kubernetes͸μΠφϛοΫͳϥΠϑαΠΫϧΛඪ४Ͱ૷උɻ Liveness Lifecycle Logging

    Kube-health kubeࣗ਎ͷॏཁίϯϙʔωϯτͷ؂ࢹ ͦͷଞkube-systemͷ؂ࢹ metrics-serverʹΑΔϦιʔε؂ࢹ
  39. LOOK AT THE K8S DEFAULT FUNCTION. ✓ kube1.7͘Β͍͸಺෦ͷϦιʔε؂ࢹ͸ʮHeapsterʯͱ͍͏ ίϯϙʔωϯτ͕΍͍ͬͯͨɻ ✓

    kube1.8͔Βʮmetrics-serverʯͱ͍͏ίϯϙʔωϯτʹஔ͖ ׵ΘΓɺલड़ͨ͠ϙϯνֆͷΑ͏ͳΞʔΩςΫνϟʹͳΓɺΑ ΓϓϥΨϒϧͳΞʔΩςΫτʹɻ ✓ʮHeapsterʯ͸طʹඇਪ঑ͳίϯϙʔωϯτɻʮmetrics- serverʯʹΑΔʮOSS monitoring architectureʯͷํͰษڧΛ ਐΊ͍͖ͯ·͠ΐ͏(^p^) metrics-serverʹΑΔϦιʔε؂ࢹ
  40. Master Node LOOK AT THE K8S DEFAULT FUNCTION. Worker Node

    Worker Node kubelet cAdvisor metrics- server Worker Node kubelet cAdvisor metrics.k8s.io api-server API CLIENT GET resource usage resource usage get data get data resource metrics pipelineʹΑΔجຊతͳϦιʔεऔಘ metrics-serverʹΑΔϦιʔε؂ࢹ
  41. LOOK AT THE K8S DEFAULT FUNCTION. full monitoring pipelineʹΑΔϦονͳΧελϜϦιʔεऔಘ prometheusͱ͔͕͜ΕʹରԠ

    metrics-serverʹΑΔϦιʔε؂ࢹ ? https://www.weave.works/blog/kubernetes-horizontal-pod-autoscaler-and-prometheus