CloudNativeDaysTokyo 2019 slide

F4b273c6b954b244a55f3e9370bbb3a6?s=47 hidenobu goto
July 22, 2019
1.5k

CloudNativeDaysTokyo 2019 slide

F4b273c6b954b244a55f3e9370bbb3a6?s=128

hidenobu goto

July 22, 2019
Tweet

Transcript

  1. KubernetesΛӡ༻ͨ͜͠ͱ ͰֶΜͩΞϯνύλʔϯ גࣜձࣾαΠόʔΤʔδΣϯτ ٕज़ຊ෦ Private Cloud Group
 ޙ౻ल৴ !1

  2. Profile • ໊લ ޙ౻ ल৴ • ॴଐ ٕज़ຊ෦ PrivateCloudGroup ։ൃνʔϜ

    • ܦྺ 2014 / 04 ~ 2016/08 SIer 2016 / 09 ~ ݱࡏ ࡏ੶த • ۀ຿ Kubernetes, OpenShiftͷॾʑ !2
  3. 1. ͜Ε·ͰͷPrivateCloudPlatform 2. Kubernetesͷߏ੒ 3. Ξϯνύλʔϯ8બ 4. ·ͱΊ !3

  4. ͜Ε·Ͱͷ PrivateCloudPlatform !4

  5. BareMetal Service (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ BareMetal ಺੡Ϋϥ΢υ Service VM BareMetal OpenStack

    Kilo
 Service VM BareMetal Kubernetes OpenStack Queens Service VM ͜Ε·ͰͷPrivateCloudPlatform 2013 - 2019 2015 - 2021 2019 - ݱࡏ΋Ұ෦ར༻த !5
  6. BareMetal Service (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ BareMetal ಺੡Ϋϥ΢υ Service VM BareMetal OpenStack

    Kilo
 Service VM BareMetal Kubernetes OpenStack Queen Service VM 2013 - 2019 2015 - 2021 2019 - ݱࡏ΋Ұ෦ར༻த ͜Ε·ͰͷPrivateCloudPlatform !6
  7. BareMetal Service (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ BareMetal ಺੡Ϋϥ΢υ Service VM BareMetal OpenStack

    Kilo
 Service VM BareMetal Kubernetes OpenStack Queen Service VM 2013 - 2019 2015 - 2021 2019 - ݱࡏ΋Ұ෦ར༻த ͜Ε·ͰͷPrivateCloudPlatform !7
  8. ͳͥͦ͜ʹKubernetesʁ !8

  9. BareMetal Service (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ BareMetal ಺੡Ϋϥ΢υ Service VM BareMetal OpenStack

    Kilo
 Service VM BareMetal Kubernetes OpenStack Queen Service VM 2013 - 2019 2015 - 2021 2019 - - 2013 ͜Ε·ͰͷPrivateCloudPlatform !9
  10. • ComputeϊʔυΛ৽ن௥Ճ͢Δͷʹ͕͔͔࣌ؒΔ(20-30෼) • ઃఆมߋΛ͢Δ৔߹΋Ansible४උͯ͠.... • ࠓճ͸dynamic inventory࢖͑ͳ͍͔ΒIP͖࣋ͬͯͯ... • ͋Εɺ͜ͷComputeϊʔυ͚ͩconf͕[changed]ʹͳͬͨΑʁ •

    όΠφϦมߋ͍ͨ͠Μ͚ͩͲґଘͱ͔৭ʑେৎ෉?? • devͱ؀ڥ͕ҧ͏ͷͰɺ͜ͷϊʔυ͸ςετϊʔυͱͯ͠௥Ճͯ͠ݕূ • ਃ͠ૹΒΕͨڊେͳAnsibleྲྀ͢ͷυΩυΩ͢Δ ͜Ε·Ͱ՝୊ !10
  11. ͜ͷΑ͏ͳ՝୊͕͋Γͭͭ
 ࣍ظPlatform͸Ͳ͏͠Α͏͔... !11

  12. !12

  13. ͋ͬͨ !13

  14. !14

  15. • ComputeϊʔυΛ৽ن௥Ճ͢Δͷʹ͕͔͔࣌ؒΔ(20-30෼) • ઃఆมߋΛ͢Δ৔߹΋Ansible४උͯ͠.... • ࠓճ͸dynamic inventory࢖͑ͳ͍͔ΒIP͖࣋ͬͯͯ... • ͋Εɺ͜ͷComputeϊʔυ͚ͩconf͕[changed]ʹͳͬͨΑʁ •

    όΠφϦมߋ͍ͨ͠Μ͚ͩͲґଘͱ͔৭ʑେৎ෉?? • devͱ؀ڥ͕ҧ͏ͷͰɺ͜ͷϊʔυ͸ςετϊʔυͱͯ͠௥Ճͯ͠ݕূ • ਃ͠ૹΒΕͨڊେͳAnsibleྲྀ͢ͷυΩυΩ͢Δ ͜Ε·Ͱ՝୊ !15
  16. • ComputeϊʔυΛ৽ن௥Ճ͢Δͷʹ͕͔͔࣌ؒΔ(20-30෼) • ઃఆมߋΛ͢Δ৔߹΋Ansible४උͯ͠.... • ࠓճ͸dynamic inventory࢖͑ͳ͍͔ΒIP͖࣋ͬͯͯ... • ͋Εɺ͜ͷComputeϊʔυ͚ͩconf͕[changed]ʹͳͬͨΑʁ •

    όΠφϦมߋ͍ͨ͠Μ͚ͩͲґଘͱ͔৭ʑେৎ෉?? • devͱ؀ڥ͕ҧ͏ͷͰɺ͜ͷϊʔυ͸ςετϊʔυͱͯ͠௥Ճͯ͠ݕূ • ਃ͠ૹΒΕͨڊେͳAnsibleྲྀ͢ͷυΩυΩ͢Δ ͜Ε·Ͱ՝୊ DaemonSetͰղܾ !16
  17. • ComputeϊʔυΛ৽ن௥Ճ͢Δͷʹ͕͔͔࣌ؒΔ(20-30෼) • ઃఆมߋΛ͢Δ৔߹΋Ansible४උͯ͠.... • ࠓճ͸dynamic inventory࢖͑ͳ͍͔ΒIP͖࣋ͬͯͯ... • ͋Εɺ͜ͷComputeϊʔυ͚ͩconf͕[changed]ʹͳͬͨΑʁ •

    όΠφϦมߋ͍ͨ͠Μ͚ͩͲґଘͱ͔৭ʑେৎ෉?? • devͱ؀ڥ͕ҧ͏ͷͰɺ͜ͷϊʔυ͸ςετϊʔυͱͯ͠௥Ճͯ͠ݕূ • ਃ͠ૹΒΕͨڊେͳAnsibleྲྀ͢ͷυΩυΩ͢Δ ͜Ε·Ͱ՝୊ DaemonSetͰղܾ ConfigMapͰղܾ !17
  18. • ComputeϊʔυΛ৽ن௥Ճ͢Δͷʹ͕͔͔࣌ؒΔ(20-30෼) • ઃఆมߋΛ͢Δ৔߹΋Ansible४උͯ͠.... • ࠓճ͸dynamic inventory࢖͑ͳ͍͔ΒIP͖࣋ͬͯͯ... • ͋Εɺ͜ͷComputeϊʔυ͚ͩconf͕[changed]ʹͳͬͨΑʁ •

    όΠφϦมߋ͍ͨ͠Μ͚ͩͲґଘͱ͔৭ʑେৎ෉?? • devͱ؀ڥ͕ҧ͏ͷͰɺ͜ͷϊʔυ͸ςετϊʔυͱͯ͠௥Ճͯ͠ݕূ • ਃ͠ૹΒΕͨڊେͳAnsibleྲྀ͢ͷυΩυΩ͢Δ ͜Ε·Ͱ՝୊ DaemonSetͰղܾ ConfigMapͰղܾ ContainerͰղܾ !18
  19. • ComputeϊʔυΛ৽ن௥Ճ͢Δͷʹ͕͔͔࣌ؒΔ(20-30෼) • ઃఆมߋΛ͢Δ৔߹΋Ansible४උͯ͠.... • ࠓճ͸dynamic inventory࢖͑ͳ͍͔ΒIP͖࣋ͬͯͯ... • ͋Εɺ͜ͷComputeϊʔυ͚ͩconf͕[changed]ʹͳͬͨΑʁ •

    όΠφϦมߋ͍ͨ͠Μ͚ͩͲґଘͱ͔৭ʑେৎ෉?? • devͱ؀ڥ͕ҧ͏ͷͰɺ͜ͷϊʔυ͸ςετϊʔυͱͯ͠௥Ճͯ͠ݕূ • ਃ͠ૹΒΕͨڊେͳAnsibleྲྀ͢ͷυΩυΩ͢Δ ͜Ε·Ͱ՝୊ DaemonSetͰղܾ ConfigMapͰղܾ ContainerͰղܾ Helm౳Ͱղܾ !19
  20. Kubernetesͷߏ੒ !20

  21. ৄࡉ৘ใ ,VCFSOFUFT  %PDLFS  6CVOUV  *OTUBMM ,VCFBEN $POUSPM1MBOF

    ୆ 8PSLFS/PEF ໿୆ !21
  22. Control Plane Node - baremetal Worker Node (OpenStack: Compute Node)

    - baremetal Neutron-proxy Nova-compute Neutron- openvswitch VM Ceph Openstack master Kubernetes master Nginx-ingress etc... Prometheus Harbor fluentd / exporter VM VM Maxsacle GaleraCluster × 5 × 135 (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ Control Plane Worker Node !22
  23. 
 ՝୊Λղܾ !23

  24. ͜͜·Ͱ͸ྑ͔ͬͨ
 (໿1೥൒લ) !24

  25. ͠͹Βͯ͘͠ !25

  26. !26

  27. !27

  28. ͱ͍͏͜ͱͰ !28

  29. Ξϯνύλʔϯ8બ • ग़͘Θͨ͠ΞϯνύλʔϯΛ8ݸ΄Ͳ͓఻͑͠·͢ • όʔδϣϯʹΑͬͯ౰ͯ͸·Βͳ͍৔߹΋͋Γ·͢ ※ ղܾํ๏ͳͲ͸͋͘·ͰݸਓతͳݟղͰ͢ ※ ຊࢿྉʹܝࡌ͞Εͨ಺༰ʹΑͬͯੜͨ͡ଛ֐౳ͷҰ੾ͷ੹೚Λෛ͍͔Ͷ·͢ͷͰ
 ྃ͝ঝ͍ͩ͘͞.

    ※ ຊࢿྉʹܝࡌ͞Εͨ಺༰Λར༻͢Δ৔߹͸ࣗݾ੹೚Ͱ͓ئ͍͠·͢ !29
  30. Ξϯνύλʔϯ8બ !30

  31. Ξϯνύλʔϯ1
 Control PlaneʹͦͷଞPod΋૬৐Γ !31

  32. • ͳͥͦ͏ͨ͠ʁ - ࠓճ͸OpenStackͷͨΊͷKubernetes
 ComputeϊʔυͰ͸ͳ͍෺ཧαʔόΛ؅ཧͨ͘͠ͳ͔ͬͨ(ӡ༻తͳ໘Ͱ)
 (ϊʔυ਺ΛՄೳͳݶΓ཈͍͑ͨཉٻ͸Ͳ͜ʹͰ΋͋Δ͸ͣ) Kubernetes Control Plane Nova-XXXX

    kube-apiserver kube-controller kube-scheduler Neutron-XXXX Cinder-XXXX Compute Worker Node Nova-XXXX Neutron-XXXX VM VM Compute Worker Node Nova-XXXX Neutron-XXXX VM VM (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ Control Planeʹͦͷଞpod΋૬৐Γ !32
  33. • Կ͕ىͬͨ͜ - ͔֬ʹ෺ཧαʔόΛ؅ཧ͢Δྔ͸গͳ͘ͳͬͨ - Control PlaneͷϝϯςΛߦ͏ͷ͕ඇৗʹ೉͘͠ͳͬͨ - ͦͷଞPodͷෆ۩߹ͷϝϞϦϦʔΫͳͲͰϗετ͕ෆௐʹͳΔͱӨڹΛ΋Ζʹड͚Δ Kubernetes

    Control Plane Nova-XXXX kube-apiserver kube-controller kube-scheduler Neutron-XXXX Cinder-XXXX Compute Worker Node Nova-XXXX Neutron-XXXX VM VM Compute Worker Node Nova-XXXX Neutron-XXXX VM VM (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ !33
  34. • Կ͕ىͬͨ͜ - ͔֬ʹ෺ཧαʔόΛ؅ཧ͢Δྔ͸গͳ͘ͳͬͨ - Control PlaneͷϝϯςΛߦ͏ͷ͕ඇৗʹ೉͘͠ͳͬͨ - ͦͷଞPodͷෆ۩߹ͷϝϞϦϦʔΫͳͲͰϗετ͕ෆௐʹͳΔͱӨڹΛ΋Ζʹड͚Δ Kubernetes

    Control Plane Nova-XXXX kube-apiserver kube-controller kube-scheduler Neutron-XXXX Cinder-XXXX Compute Worker Node Nova-XXXX Neutron-XXXX VM VM Compute Worker Node Nova-XXXX Neutron-XXXX VM VM (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ Original App !34
  35. • Կ͕ىͬͨ͜ - ͔֬ʹ෺ཧαʔόΛ؅ཧ͢Δྔ͸গͳ͘ͳͬͨ - Control PlaneͷϝϯςΛߦ͏ͷ͕ඇৗʹ೉͘͠ͳͬͨ - ͦͷଞPodͷෆ۩߹ͷϝϞϦϦʔΫͳͲͰϗετ͕ෆௐʹͳΔͱӨڹΛ΋Ζʹड͚Δ Kubernetes

    Control Plane Nova-XXXX lube-apiserver Kube-controller kube-scheduler Neutron-XXXX Cinder-XXXX Compute Worker Node Nova-XXXX Neutron-XXXX VM VM Compute Worker Node Nova-XXXX Neutron-XXXX VM VM (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ Original App !35
  36. • Կ͕ىͬͨ͜ - ͔֬ʹ෺ཧαʔόΛ؅ཧ͢Δྔ͸গͳ͘ͳͬͨ - Control PlaneͷϝϯςΛߦ͏ͷ͕ඇৗʹ೉͘͠ͳͬͨ - ͦͷଞPodͷෆ۩߹ͷϝϞϦϦʔΫͳͲͰϗετ͕ෆௐʹͳΔͱӨڹΛ΋Ζʹड͚Δ Kubernetes

    Control Plane Compute Worker Node Nova-XXXX Neutron-XXXX VM VM Compute Worker Node Nova-XXXX Neutron-XXXX VM VM (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ !36
  37. • Ͳ͏ͨ͠ํ͕ྑ͔ͬͨʁ - Control PlaneʹͦͷଞͷPod͕৐ͤͳ͍Α͏ʹͨ͠ํ͕ྑ͍
 ؅ཧ༻ͷInfra Workerϊʔυ౳Λ༻ҙ͢Δ (σϑΥϧτͰ͸৐Βͳ͍) Kubernetes Control

    Plane Infra Worker Node Nova-XXXX kube-apiserver kube-controller kube-scheduler Neutron-XXXX Cinder-XXXX Compute Worker Node Nova-XXXX Neutron-XXXX VM VM (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ !37
  38. • Ͳ͏ͨ͠ํ͕ྑ͔ͬͨʁ - Control PlaneʹͦͷଞͷPod͕৐ͤͳ͍Α͏ʹͨ͠ํ͕ྑ͍
 ؅ཧ༻ͷInfra Workerϊʔυ౳Λ༻ҙ͢Δ (σϑΥϧτͰ͸৐Βͳ͍) Kubernetes Control

    Plane Infra Worker Node Nova-XXXX kube-apiserver kube-controller kube-scheduler Neutron-XXXX Cinder-XXXX Compute Worker Node Nova-XXXX Neutron-XXXX VM VM (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ !38
  39. Ξϯνύλʔϯ2
 Ϋϥελ಺ͷϛχϚϜCeph !39

  40. • ͳͥͦ͏ͨ͠ʁ - ComputeϊʔυͰ͸ͳ͍෺ཧαʔόΛ؅ཧͨ͘͠ͳ͔ͬͨʢӡ༻తͳ໘Ͱʣ - osdͱ͔mon͕མͪͯ΋ࣗಈͰ෮چͯ͘͠ΕΔ - PVCͱ͔Ͱ࢖͍͍ͨͳ.....
 - CephΛߏங͢ΔVM͕͍ͳ͍


    - ΦϒδΣΫτετϨʔδͱͯ͠࢖͍͍ͨͳ.... Ϋϥελ಺ͷϛχϚϜCeph !40
  41. • Կ͕ىͬͨ͜ - ϊʔυͷϝϯςͷ࣮ࢪ + ো֐౳ͰCephͷΫϥελ͕յΕͨ - PVCΛར༻͍ͯ͠ΔPod͕མͪɺ࠷ऴతʹOpenStack΋࢖͑ͳ͘ͳͬͨ Kubernetes Control

    Plane osd - A Ceph-mon Kubernetes Control Plane Kubernetes Control Plane osd - A Ceph-mon osd - A Ceph-mon Ceph-mgr ࢒Γ
 ϝϯςத (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ !41
  42. • Կ͕ىͬͨ͜ - ϊʔυͷϝϯςͷ࣮ࢪ + ো֐౳ͰCephͷΫϥελ͕յΕͨ - PVCΛར༻͍ͯ͠ΔPod͕མͪɺ࠷ऴతʹOpenStack΋࢖͑ͳ͘ͳͬͨ Kubernetes Control

    Plane osd - A Ceph-mon Kubernetes Control Plane Kubernetes Control Plane osd - A Ceph-mon osd - A Ceph-mon Ceph-mgr DatabaseA DatabaseA DatabaseB DatabaseA (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ OpenStack OpenStack OpenStack ࢒Γ
 ϝϯςத !42
  43. • Կ͕ىͬͨ͜ - ϊʔυͷϝϯςͷ࣮ࢪ + ো֐౳ͰCephͷΫϥελ͕յΕͨ - PVCΛར༻͍ͯ͠ΔPod͕མͪɺ࠷ऴతʹOpenStack΋࢖͑ͳ͘ͳͬͨ
 Kubernetes Control

    Plane osd - A Ceph-mon Kubernetes Control Plane Kubernetes Control Plane osd - A Ceph-mon osd - A Ceph-mon Ceph-mgr DatabaseA DatabaseA DatabaseB DatabaseA (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ OpenStack OpenStack OpenStack ࢒Γ
 ϝϯςத !43
  44. • Կ͕ىͬͨ͜ - ϊʔυͷϝϯςͷ࣮ࢪ + ো֐౳ͰCephͷΫϥελ͕յΕͨ
 - PVCΛར༻͍ͯ͠ΔPod͕མͪɺ࠷ऴతʹOpenStack΋࢖͑ͳ͘ͳͬͨ Kubernetes Control

    Plane osd - A Ceph-mon Kubernetes Control Plane Kubernetes Control Plane osd - A Ceph-mon Ceph-mgr DatabaseA DatabaseB DatabaseA (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ OpenStack OpenStack ࢒Γ
 ϝϯςத !44
  45. • Կ͕ىͬͨ͜ - ϊʔυͷϝϯςͷ࣮ࢪ + ো֐౳ͰCephͷΫϥελ͕յΕͨ
 - PVCΛར༻͍ͯ͠ΔPod͕མͪɺ࠷ऴతʹOpenStack΋࢖͑ͳ͘ͳͬͨ Kubernetes Control

    Plane osd - A Ceph-mon Kubernetes Control Plane Kubernetes Control Plane osd - A Ceph-mon Ceph-mgr DatabaseA DatabaseB DatabaseA (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ OpenStack OpenStack ࢒Γ
 ϝϯςத !45
  46. • Կ͕ىͬͨ͜ - ϊʔυͷϝϯςͷ࣮ࢪ + ো֐౳ͰCephͷΫϥελ͕յΕͨ
 - PVCΛར༻͍ͯ͠ΔPod͕མͪɺ࠷ऴతʹOpenStack΋࢖͑ͳ͘ͳͬͨ Kubernetes Control

    Plane osd - A Ceph-mon Kubernetes Control Plane Kubernetes Control Plane osd - A Ceph-mon Ceph-mgr DatabaseA DatabaseB DatabaseA (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ OpenStack OpenStack ࢒Γ
 ϝϯςத !46
  47. • Կ͕ىͬͨ͜ - ϊʔυͷϝϯςͷ࣮ࢪ + ো֐౳ͰCephͷΫϥελ͕յΕͨ
 - PVCΛར༻͍ͯ͠ΔPod͕མͪɺ࠷ऴతʹOpenStack΋࢖͑ͳ͘ͳͬͨ Kubernetes Control

    Plane osd - A Ceph-mon Kubernetes Control Plane Kubernetes Control Plane osd - A Ceph-mon Ceph-mgr DatabaseA DatabaseB DatabaseA (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ OpenStack OpenStack ࢒Γ
 ϝϯςத !47
  48. Infra Worker Node osd - A Ceph-mon Infra Worker Node

    Infra Worker Node osd - A Ceph-mon osd - A Ceph-mon Ceph-mgr ɾɾN • Ͳ͏ͨ͠ํ͕ྑ͔ͬͨʁ - ϊʔυ਺͸ͨ͘͞Μ͋ͬͨํ͕ྑ͍ !48
  49. Ξϯνύλʔϯ3 Ϋϥελ಺ͷPrivate Registry !49

  50. • ͳͥͦ͏ͨ͠ʁ - Registryͷؔ࿈ίϯϙʔωϯτ͕མͪͯ΋࠶ىಈͯ͘͠ΕΔ͔Β͖ͬͱָ - CephͷPVC࢖͑͹؆୯Ͱ͸ʁʁ Kubernetes Control Plane Kubernetes

    
 Control Plane XXXXXX kube-apiserver kube-controller kube-scheduler XXXXXX Registry Infra Worker Node XXXXXX XXXXXX Kubernetes 
 Control Plane Ϋϥελ಺ͷPrivate Registry !50
  51. Kubernetes Control Plane Kubernetes 
 Control Plane XXXXXX kube-apiserver kube-controller

    kube-scheduler XXXXXX Registry Infra Worker Node XXXXXX XXXXXX Kubernetes 
 Control Plane • Կ͕ىͬͨ͜ - Registry͕ଘࡏ͢Δϊʔυ͕མͪͨ৔߹ɺRegistry͕෮چ͢Δ·ͰΠϝʔδΛϓϧͰ͖ͳ͍ ৔߹͕͋Γɺ௨ৗΑΓ෮چʹ͕͔͔࣌ؒΔ৔߹΋͋Δ - Ceph่͕յͨ͠λΠϛϯάͰ࢖͑ͳ͘ͳͬͨ !51
  52. Kubernetes Control Plane Kubernetes 
 Control Plane XXXXXX kube-apiserver kube-controller

    kube-scheduler XXXXXX Registry Infra Worker Node XXXXXX XXXXXX Kubernetes 
 Control Plane • Կ͕ىͬͨ͜ - Registry͕ଘࡏ͢Δϊʔυ͕མͪͨ৔߹ɺRegistry͕෮چ͢Δ·ͰΠϝʔδΛϓϧͰ͖ͳ͍ ৔߹͕͋Γɺ௨ৗΑΓ෮چʹ͕͔͔࣌ؒΔ৔߹΋͋Δ - Ceph่͕յͨ͠λΠϛϯάͰ࢖͑ͳ͘ͳͬͨ ͋Δϊʔυ͕μ΢ϯ
 (ϨδετϦͷؔ࿈ίϯϙʔωϯτ͕ଘࡏ) !52
  53. • Կ͕ىͬͨ͜ - Registry͕ଘࡏ͢Δϊʔυ͕མͪͨ৔߹ɺRegistry͕෮چ͢Δ·ͰΠϝʔδΛϓϧͰ͖ͳ͍ ৔߹͕͋Γɺ௨ৗΑΓ෮چʹ͕͔͔࣌ؒΔ৔߹΋͋Δ - Ceph่͕յͨ͠λΠϛϯάͰ࢖͑ͳ͘ͳͬͨ Kubernetes Control Plane

    Infra Worker Node kube-apiserver kube-controller kube-scheduler Registry Infra Worker Node XXXXXX XXXXXX ϨδετϦ͕෮چ͢Δ·Ͱ͸
 ϊʔυ্ʹPod͕࡞ΒΕͳ͍ XXXXXX XXXXXX Infra Worker Node XXXXXX XXXXXX Kubernetes 
 Control Plane Kubernetes 
 Control Plane Kubernetes 
 Control Plane !53
  54. • Ͳ͏ͨ͠ํ͕ྑ͔ͬͨʁ - k8sΫϥελͷ֎·ͨ͸ผk8sΫϥελʔͷ΋ͷΛར༻ͨ͠ํ͕ࣗ༝౓͸ߴ͍
 (Ҡઃ΋ߦ͍΍͍͢) - ৑௕͠΍͍͢RegistryΛ࢖͏ඞཁ͕͋Δ 
 (όοΫΤϯυʹDB͕͋Δ৔߹͸ͦΕΒ΋ؚΊͯ) Kubernetes

    Control Plane Infra Worker Node XXXXXX kube-apiserver kube-controller kube-scheduler XXXXXX Registry Infra Worker Node XXXXXX XXXXXX ͋Δϊʔυ͕μ΢ϯ Kubernetes 
 Control Plane Kubernetes 
 Control Plane !54
  55. • Ͳ͏ͨ͠ํ͕ྑ͔ͬͨʁ - k8sΫϥελͷ֎·ͨ͸ผk8sΫϥελʔͷ΋ͷΛར༻ͨ͠ํ͕ࣗ༝౓͸ߴ͍
 (Ҡઃ΋ߦ͍΍͍͢) - ৑௕͠΍͍͢RegistryΛ࢖͏ඞཁ͕͋Δ 
 (όοΫΤϯυʹDB͕͋Δ৔߹͸ͦΕΒ΋ؚΊͯ) Kubernetes

    Control Plane Infra Worker Node XXXXXX kube-apiserver kube-controller kube-scheduler XXXXXX Registry Infra Worker Node XXXXXX XXXXXX ଞͷϊʔυ্ʹpod͕࡞ΒΕΔ Kubernetes 
 Control Plane Kubernetes 
 Control Plane !55
  56. Ξϯνύλʔϯ4
 Ϋϥελ಺ͷPrometheus !56

  57. • ͳͥͦ͏ͨ͠ʁ - ؔ࿈ίϯϙʔωϯτ͕མͪͯ΋࠶ىಈͯ͘͠ΕΔ͔Β͖ͬͱָ - CephͷPVC࢖͑͹؆୯Ͱ͸ʁʁ Ϋϥελ಺ͷPrometheus !57

  58. • Կ͕ىͬͨ͜ - Ceph → Prometheus ͜ͷॱ൪Ͱར༻ෆՄʹͳͬͨ - ౰࣌֎ܗ͔Βͷ؂ࢹ͕ͳ͍ͷͰɺ্هͷ໰୊͕ى͖ͯ΋੩͔ͩͬͨ •

    Ͳ͏ͨ͠ํ͕ྑ͔ͬͨʁ - ֎ܗ؂ࢹΛ͢ΔͳΓɺ֎͔ΒKubernetsͷAPIΛୟ͍ͯSDͰ؂ࢹ͢Δύλʔϯ͕ྑ͔ͬͨ ฐࣾͷத੢͕Prometheus Tokyo Meetup #2 ʹͯৄࡉ͸ൃදͯ͠·͢ (ࢿྉ) !58
  59. Ξϯνύλʔϯ5
 େ͖ͳWorker NodeΛগͳ͍਺ͰͲʔΜ !59

  60. • ͳͥͦ͏ͨ͠ʁ - ComputeϊʔυͰ͸ͳ͍෺ཧαʔόΛ(ry • Կ͕ىͬͨ͜ - Ұ୆ʹ৐ΔPodͷྔ͕͔ͳΓଟ͘ͳΓɺҰ୆མͪͨͱ͖ͷΠϯύΫτ͕େ͖͘ͳͬͨ - ؾܰʹର৅ϊʔυͷϝϯςφϯεΛ͠ਏ͍

    - Ұ୆མͪͨͱ͖ʹPodͷߦ͖ઌ͕ͳ͘ͳ͍ͬͯͨ • Ͳ͏ͨ͠ํ͕ྑ͔ͬͨʁ - ͋Δఔ౓খ͞ͳϊʔυΛͨ͘͞Μฒ΂ͨํ͕ྑ͍(೔ʑͷӡ༻ϝϦοτ͸͋Δ) େ͖ͳWorker NodeΛগͳ͍਺ͰͲʔΜ !60
  61. Ξϯνύλʔϯ6
 ίϯςφͷϩάϩʔςΛ͍ͯ͠ͳ͍ !61

  62. • ͳͥͦ͏ͨ͠ʁ - લఏͱͯ͠໰୊͕ى͖ͨͷ͸ComputeϊʔυͷϗετͰ
 VM༻ʹϦιʔεΛ֬อ͢Δ΂͘KubernetesͰ࢖͏Ϧιʔε͸ͳΔ΂͘࢖Θͳ͍Α͏ʹ͍ͯͨ͠
 ͕ϩάϩʔςͷ͜ͱΛ׬શʹࣦ೦͍ͯͨ͠ ίϯςφͷϩάϩʔςΛ͍ͯ͠ͳ͍ !62

  63. • Կ͕ىͬͨ͜ - PodσϓϩΠ࣌ʹPodͷঢ়ଶ͕Pendingʹͳͬͨ (σϑΥϧτͷϩάϩʔςͩͱ଍Γͳ͍) • Ͳ͏ͨ͠ํ͕ྑ͔ͬͨʁ - docker-logdriverͷઃఆΛద੾ʹ͢Δ΂͖(αΠζ΍ੈ୅؅ཧ) -

    ෆཁͳdebugϩάͳͲ͸ग़͞ͳ͍Α͏ʹ͢Δ !63
  64. Ξϯνύλʔϯ7
 in_tailͱϩάϩʔςͷར༻ !64

  65. • ͳͥͦ͏ͨ͠ʁ - ϩάͷංେʹΑΓPod͕εέδϡʔϦϯά͞Εͳ͍Α͏ʹͳͬͨͷͰ
 ϩάΛfluentdͰtailͭͭ͠΋1࣌ؒຖʹlogrotateΛಈ͔͍ͯͨ͠ in_tailͱϩάϩʔςͷར༻ !65

  66. • Կ͕ىͬͨ͜ - 1࣌ؒͷස౓ͰlogrotateΛͯ͠ɺಉ࣌ʹfluentdͰ”*”Ͱin_tail͍ͯ͠Δͱ fluentd͕ߴස౓ͰLimitʹ౰ͨͬͯOOMͰkill͞ΕΔΑ͏ʹͳͬͨ ௚઀ؔ܎͋Δ͔͸ௐ΂͖Εͯͳ͍͕ɺ͜ͷΑ͏ͳissues΋͋Γ (https://github.com/fluent/fluentd-kubernetes-daemonset/issues/156) (https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/in_tail.rb#L255-L259) • Ͳ͏ͨ͠ํ͕ྑ͔ͬͨʁ

    - ϩάϩʔςΛ͢ΔλΠϛϯά͸௨ৗ௨Γʹ͠ɺੈ୅Λগͳ͘͢ΔΑ͏ʹ͢Δ - in_tailΛར༻͠ͳ͍ํ๏ͰؤுΔ !66
  67. Ξϯνύλʔϯ8 ա౓ͳCronJobͷར༻ !67

  68. • ͳͥͦ͏ͨ͠ʁ - όονॲཧΛຖ෼ߦͳ͍ͬͯΔ΋ͷ͕ෳ਺͍ͨ - ͦͷଞόοΫΞοϓͳͲ΋جຊతʹ͸CronJobͰճ͍ͯͨ͠ - ͜ΕΛControl PlaneͰճ͍ͯͨ͠ ա౓ͳCronJobͷར༻

    !68
  69. • Կ͕ىͬͨ͜ - Podͷ্͛Լ͕͛සൟʹߦΘΕΔ͜ͱͰɺϗετଆͷbuffer/cache͕ղ์͞Εͳ͍Ͱ free memory͕0%ʹͳΔ͜ͱͰϗετμ΢ϯ͕ൃੜͨ͠ ·ͨɺ/sys/fs/cgroup/memory഑Լ͕ҟৗʹͳ͍ͬͯͨ(Cannot Allocate Memory) -

    Control PlaneͰ͜ΕΒ͕ಈ͍͍ͯͨͨΊසൟʹControl Plane͕ෆ҆ఆʹͳͬͨ • Ͳ͏ͨ͠ํ͕ྑ͔͔ͬͨ - ৔౰ͨΓతͳରԠͱͳ͍ͬͯΔ͕ɺ ͦͷଞͷPodʹӨڹΛٴ΅͞ͳ͍Α͏ʹCronJobΛ࣮ߦ͢Δϊʔυ͸labelΛར༻ͯ͠ɺ CronJobՔಇઐ༻ͷWorker Nodeͱͯ͠ෳ਺୆Քಇ͍ͤͯ͞Δ !69
  70. ະͩରԠதͷ෦෼΋͋Γ
 ӡ༻Ͱؤு͍ͬͯΔ ߲໨΋͋Γ·͢ !70

  71. Ҏ্
 Ξϯνύλʔϯ8બͰͨ͠ !71

  72. ·ͱΊ !72

  73. ·ͱΊ • ౰࣌͸Kubernets͕͍͍ײ͡ʹ΍ͬͯ͘ΕΔͩΖ͏ͱґଘ͍͗ͯͨ͢͠ • Kubernetesͷࢥ૝͔Β͋·Γʹ΋ζϨͨ࢖͍ํΛ͢Δͱਏ͘ͳΔ • ӡ༻͕໘౗͔ͩΒͱ͍͏ࢹ఺͚ͩͩͱٯʹਏ͘ͳΔ • ͨͩɺKubernetesΛ̍೥΄Ͳӡ༻ͯ͠Έͯศརͳͱ͜Ζ΋
 ৭ʑͱײ͍ͯ͡Δ

    !73
  74. ·ͱΊ • ౰࣌͸Kubernets͕͍͍ײ͡ʹ΍ͬͯ͘ΕΔͩΖ͏ͱґଘ͍͗ͯͨ͢͠ • Kubernetesͷࢥ૝͔Β͋·Γʹ΋ζϨͨ࢖͍ํΛ͢Δͱਏ͘ͳΔ • ӡ༻͕໘౗͔ͩΒͱ͍͏ࢹ఺͚ͩͩͱٯʹਏ͘ͳΔ • ͨͩɺkubernetesΛ̍೥΄Ͳӡ༻ͯ͠Έͯศརͳͱ͜Ζ΋
 ৭ʑͱײ͍ͯ͡Δ

    ༻๏ɾ༰ྔΛकͬͯద੾ʹ !74
  75. ͋Γ͕ͱ͏͍͟͝·ͨ͠ !75