CloudNativeDaysTokyo 2019 slide

Transcript

1. KubernetesΛӡ༻ͨ͜͠ͱ ͰֶΜͩΞϯνύλʔϯ גࣜձࣾαΠόʔΤʔδΣϯτ ٕज़ຊ෦ Private Cloud Group
 ޙ౻ल৴ !1

2. Profile • ໊લ ޙ౻ ल৴ • ॴଐ ٕज़ຊ෦ PrivateCloudGroup ։ൃνʔϜ

   • ܦྺ 2014 / 04 ~ 2016/08 SIer 2016 / 09 ~ ݱࡏ ࡏ੶த • ۀ຿ Kubernetes, OpenShiftͷॾʑ !2

3. 1. ͜Ε·ͰͷPrivateCloudPlatform 2. Kubernetesͷߏ੒ 3. Ξϯνύλʔϯ8બ 4. ·ͱΊ !3

4. ͜Ε·Ͱͷ PrivateCloudPlatform !4

5. BareMetal Service (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ BareMetal ಺੡Ϋϥ΢υ Service VM BareMetal OpenStack

   Kilo
 Service VM BareMetal Kubernetes OpenStack Queens Service VM ͜Ε·ͰͷPrivateCloudPlatform 2013 - 2019 2015 - 2021 2019 - ݱࡏ΋Ұ෦ར༻த !5

6. BareMetal Service (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ BareMetal ಺੡Ϋϥ΢υ Service VM BareMetal OpenStack

   Kilo
 Service VM BareMetal Kubernetes OpenStack Queen Service VM 2013 - 2019 2015 - 2021 2019 - ݱࡏ΋Ұ෦ར༻த ͜Ε·ͰͷPrivateCloudPlatform !6

7. BareMetal Service (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ BareMetal ಺੡Ϋϥ΢υ Service VM BareMetal OpenStack

   Kilo
 Service VM BareMetal Kubernetes OpenStack Queen Service VM 2013 - 2019 2015 - 2021 2019 - ݱࡏ΋Ұ෦ར༻த ͜Ε·ͰͷPrivateCloudPlatform !7

8. ͳͥͦ͜ʹKubernetesʁ !8

9. BareMetal Service (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ BareMetal ಺੡Ϋϥ΢υ Service VM BareMetal OpenStack

   Kilo
 Service VM BareMetal Kubernetes OpenStack Queen Service VM 2013 - 2019 2015 - 2021 2019 - - 2013 ͜Ε·ͰͷPrivateCloudPlatform !9

10. • ComputeϊʔυΛ৽ن௥Ճ͢Δͷʹ͕͔͔࣌ؒΔ(20-30෼) • ઃఆมߋΛ͢Δ৔߹΋Ansible४උͯ͠.... • ࠓճ͸dynamic inventory࢖͑ͳ͍͔ΒIP͖࣋ͬͯͯ... • ͋Εɺ͜ͷComputeϊʔυ͚ͩconf͕[changed]ʹͳͬͨΑʁ •

    όΠφϦมߋ͍ͨ͠Μ͚ͩͲґଘͱ͔৭ʑେৎ෉?? • devͱ؀ڥ͕ҧ͏ͷͰɺ͜ͷϊʔυ͸ςετϊʔυͱͯ͠௥Ճͯ͠ݕূ • ਃ͠ૹΒΕͨڊେͳAnsibleྲྀ͢ͷυΩυΩ͢Δ ͜Ε·Ͱ՝୊ !10

11. ͜ͷΑ͏ͳ՝୊͕͋Γͭͭ
 ࣍ظPlatform͸Ͳ͏͠Α͏͔... !11

12. !12

13. ͋ͬͨ !13

14. !14

15. • ComputeϊʔυΛ৽ن௥Ճ͢Δͷʹ͕͔͔࣌ؒΔ(20-30෼) • ઃఆมߋΛ͢Δ৔߹΋Ansible४උͯ͠.... • ࠓճ͸dynamic inventory࢖͑ͳ͍͔ΒIP͖࣋ͬͯͯ... • ͋Εɺ͜ͷComputeϊʔυ͚ͩconf͕[changed]ʹͳͬͨΑʁ •

    όΠφϦมߋ͍ͨ͠Μ͚ͩͲґଘͱ͔৭ʑେৎ෉?? • devͱ؀ڥ͕ҧ͏ͷͰɺ͜ͷϊʔυ͸ςετϊʔυͱͯ͠௥Ճͯ͠ݕূ • ਃ͠ૹΒΕͨڊେͳAnsibleྲྀ͢ͷυΩυΩ͢Δ ͜Ε·Ͱ՝୊ !15

16. • ComputeϊʔυΛ৽ن௥Ճ͢Δͷʹ͕͔͔࣌ؒΔ(20-30෼) • ઃఆมߋΛ͢Δ৔߹΋Ansible४උͯ͠.... • ࠓճ͸dynamic inventory࢖͑ͳ͍͔ΒIP͖࣋ͬͯͯ... • ͋Εɺ͜ͷComputeϊʔυ͚ͩconf͕[changed]ʹͳͬͨΑʁ •

    όΠφϦมߋ͍ͨ͠Μ͚ͩͲґଘͱ͔৭ʑେৎ෉?? • devͱ؀ڥ͕ҧ͏ͷͰɺ͜ͷϊʔυ͸ςετϊʔυͱͯ͠௥Ճͯ͠ݕূ • ਃ͠ૹΒΕͨڊେͳAnsibleྲྀ͢ͷυΩυΩ͢Δ ͜Ε·Ͱ՝୊ DaemonSetͰղܾ !16

17. • ComputeϊʔυΛ৽ن௥Ճ͢Δͷʹ͕͔͔࣌ؒΔ(20-30෼) • ઃఆมߋΛ͢Δ৔߹΋Ansible४උͯ͠.... • ࠓճ͸dynamic inventory࢖͑ͳ͍͔ΒIP͖࣋ͬͯͯ... • ͋Εɺ͜ͷComputeϊʔυ͚ͩconf͕[changed]ʹͳͬͨΑʁ •

    όΠφϦมߋ͍ͨ͠Μ͚ͩͲґଘͱ͔৭ʑେৎ෉?? • devͱ؀ڥ͕ҧ͏ͷͰɺ͜ͷϊʔυ͸ςετϊʔυͱͯ͠௥Ճͯ͠ݕূ • ਃ͠ૹΒΕͨڊେͳAnsibleྲྀ͢ͷυΩυΩ͢Δ ͜Ε·Ͱ՝୊ DaemonSetͰղܾ ConfigMapͰղܾ !17

18. • ComputeϊʔυΛ৽ن௥Ճ͢Δͷʹ͕͔͔࣌ؒΔ(20-30෼) • ઃఆมߋΛ͢Δ৔߹΋Ansible४උͯ͠.... • ࠓճ͸dynamic inventory࢖͑ͳ͍͔ΒIP͖࣋ͬͯͯ... • ͋Εɺ͜ͷComputeϊʔυ͚ͩconf͕[changed]ʹͳͬͨΑʁ •

    όΠφϦมߋ͍ͨ͠Μ͚ͩͲґଘͱ͔৭ʑେৎ෉?? • devͱ؀ڥ͕ҧ͏ͷͰɺ͜ͷϊʔυ͸ςετϊʔυͱͯ͠௥Ճͯ͠ݕূ • ਃ͠ૹΒΕͨڊେͳAnsibleྲྀ͢ͷυΩυΩ͢Δ ͜Ε·Ͱ՝୊ DaemonSetͰղܾ ConfigMapͰղܾ ContainerͰղܾ !18

19. • ComputeϊʔυΛ৽ن௥Ճ͢Δͷʹ͕͔͔࣌ؒΔ(20-30෼) • ઃఆมߋΛ͢Δ৔߹΋Ansible४උͯ͠.... • ࠓճ͸dynamic inventory࢖͑ͳ͍͔ΒIP͖࣋ͬͯͯ... • ͋Εɺ͜ͷComputeϊʔυ͚ͩconf͕[changed]ʹͳͬͨΑʁ •

    όΠφϦมߋ͍ͨ͠Μ͚ͩͲґଘͱ͔৭ʑେৎ෉?? • devͱ؀ڥ͕ҧ͏ͷͰɺ͜ͷϊʔυ͸ςετϊʔυͱͯ͠௥Ճͯ͠ݕূ • ਃ͠ૹΒΕͨڊେͳAnsibleྲྀ͢ͷυΩυΩ͢Δ ͜Ε·Ͱ՝୊ DaemonSetͰղܾ ConfigMapͰղܾ ContainerͰղܾ Helm౳Ͱղܾ !19

20. Kubernetesͷߏ੒ !20

21. ৄࡉ৘ใ ,VCFSOFUFT  %PDLFS  6CVOUV  *OTUBMM ,VCFBEN $POUSPM
    1MBOF

    ୆ 8PSLFS
    /PEF ໿୆ !21

22. Control Plane Node - baremetal Worker Node (OpenStack: Compute Node)

    - baremetal Neutron-proxy Nova-compute Neutron- openvswitch VM Ceph Openstack master Kubernetes master Nginx-ingress etc... Prometheus Harbor fluentd / exporter VM VM Maxsacle GaleraCluster × 5 × 135 (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ Control Plane Worker Node !22

23. 
 ՝୊Λղܾ !23

24. ͜͜·Ͱ͸ྑ͔ͬͨ
 (໿1೥൒લ) !24

25. ͠͹Βͯ͘͠ !25

26. !26

27. !27

28. ͱ͍͏͜ͱͰ !28

29. Ξϯνύλʔϯ8બ • ग़͘Θͨ͠ΞϯνύλʔϯΛ8ݸ΄Ͳ͓఻͑͠·͢ • όʔδϣϯʹΑͬͯ౰ͯ͸·Βͳ͍৔߹΋͋Γ·͢ ※ ղܾํ๏ͳͲ͸͋͘·ͰݸਓతͳݟղͰ͢ ※ ຊࢿྉʹܝࡌ͞Εͨ಺༰ʹΑͬͯੜͨ͡ଛ֐౳ͷҰ੾ͷ੹೚Λෛ͍͔Ͷ·͢ͷͰ
 ྃ͝ঝ͍ͩ͘͞.

    ※ ຊࢿྉʹܝࡌ͞Εͨ಺༰Λར༻͢Δ৔߹͸ࣗݾ੹೚Ͱ͓ئ͍͠·͢ !29

30. Ξϯνύλʔϯ8બ !30

31. Ξϯνύλʔϯ1
 Control PlaneʹͦͷଞPod΋૬৐Γ !31

32. • ͳͥͦ͏ͨ͠ʁ - ࠓճ͸OpenStackͷͨΊͷKubernetes
 ComputeϊʔυͰ͸ͳ͍෺ཧαʔόΛ؅ཧͨ͘͠ͳ͔ͬͨ(ӡ༻తͳ໘Ͱ)
 (ϊʔυ਺ΛՄೳͳݶΓ཈͍͑ͨཉٻ͸Ͳ͜ʹͰ΋͋Δ͸ͣ) Kubernetes Control Plane Nova-XXXX

    kube-apiserver kube-controller kube-scheduler Neutron-XXXX Cinder-XXXX Compute Worker Node Nova-XXXX Neutron-XXXX VM VM Compute Worker Node Nova-XXXX Neutron-XXXX VM VM (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ Control Planeʹͦͷଞpod΋૬৐Γ !32

33. • Կ͕ىͬͨ͜ - ͔֬ʹ෺ཧαʔόΛ؅ཧ͢Δྔ͸গͳ͘ͳͬͨ - Control PlaneͷϝϯςΛߦ͏ͷ͕ඇৗʹ೉͘͠ͳͬͨ - ͦͷଞPodͷෆ۩߹ͷϝϞϦϦʔΫͳͲͰϗετ͕ෆௐʹͳΔͱӨڹΛ΋Ζʹड͚Δ Kubernetes

    Control Plane Nova-XXXX kube-apiserver kube-controller kube-scheduler Neutron-XXXX Cinder-XXXX Compute Worker Node Nova-XXXX Neutron-XXXX VM VM Compute Worker Node Nova-XXXX Neutron-XXXX VM VM (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ !33

34. • Կ͕ىͬͨ͜ - ͔֬ʹ෺ཧαʔόΛ؅ཧ͢Δྔ͸গͳ͘ͳͬͨ - Control PlaneͷϝϯςΛߦ͏ͷ͕ඇৗʹ೉͘͠ͳͬͨ - ͦͷଞPodͷෆ۩߹ͷϝϞϦϦʔΫͳͲͰϗετ͕ෆௐʹͳΔͱӨڹΛ΋Ζʹड͚Δ Kubernetes

    Control Plane Nova-XXXX kube-apiserver kube-controller kube-scheduler Neutron-XXXX Cinder-XXXX Compute Worker Node Nova-XXXX Neutron-XXXX VM VM Compute Worker Node Nova-XXXX Neutron-XXXX VM VM (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ Original App !34

35. • Կ͕ىͬͨ͜ - ͔֬ʹ෺ཧαʔόΛ؅ཧ͢Δྔ͸গͳ͘ͳͬͨ - Control PlaneͷϝϯςΛߦ͏ͷ͕ඇৗʹ೉͘͠ͳͬͨ - ͦͷଞPodͷෆ۩߹ͷϝϞϦϦʔΫͳͲͰϗετ͕ෆௐʹͳΔͱӨڹΛ΋Ζʹड͚Δ Kubernetes

    Control Plane Nova-XXXX lube-apiserver Kube-controller kube-scheduler Neutron-XXXX Cinder-XXXX Compute Worker Node Nova-XXXX Neutron-XXXX VM VM Compute Worker Node Nova-XXXX Neutron-XXXX VM VM (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ Original App !35

36. • Կ͕ىͬͨ͜ - ͔֬ʹ෺ཧαʔόΛ؅ཧ͢Δྔ͸গͳ͘ͳͬͨ - Control PlaneͷϝϯςΛߦ͏ͷ͕ඇৗʹ೉͘͠ͳͬͨ - ͦͷଞPodͷෆ۩߹ͷϝϞϦϦʔΫͳͲͰϗετ͕ෆௐʹͳΔͱӨڹΛ΋Ζʹड͚Δ Kubernetes

    Control Plane Compute Worker Node Nova-XXXX Neutron-XXXX VM VM Compute Worker Node Nova-XXXX Neutron-XXXX VM VM (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ !36

37. • Ͳ͏ͨ͠ํ͕ྑ͔ͬͨʁ - Control PlaneʹͦͷଞͷPod͕৐ͤͳ͍Α͏ʹͨ͠ํ͕ྑ͍
 ؅ཧ༻ͷInfra Workerϊʔυ౳Λ༻ҙ͢Δ (σϑΥϧτͰ͸৐Βͳ͍) Kubernetes Control

    Plane Infra Worker Node Nova-XXXX kube-apiserver kube-controller kube-scheduler Neutron-XXXX Cinder-XXXX Compute Worker Node Nova-XXXX Neutron-XXXX VM VM (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ !37

38. • Ͳ͏ͨ͠ํ͕ྑ͔ͬͨʁ - Control PlaneʹͦͷଞͷPod͕৐ͤͳ͍Α͏ʹͨ͠ํ͕ྑ͍
 ؅ཧ༻ͷInfra Workerϊʔυ౳Λ༻ҙ͢Δ (σϑΥϧτͰ͸৐Βͳ͍) Kubernetes Control

    Plane Infra Worker Node Nova-XXXX kube-apiserver kube-controller kube-scheduler Neutron-XXXX Cinder-XXXX Compute Worker Node Nova-XXXX Neutron-XXXX VM VM (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ !38

39. Ξϯνύλʔϯ2
 Ϋϥελ಺ͷϛχϚϜCeph !39

40. • ͳͥͦ͏ͨ͠ʁ - ComputeϊʔυͰ͸ͳ͍෺ཧαʔόΛ؅ཧͨ͘͠ͳ͔ͬͨʢӡ༻తͳ໘Ͱʣ - osdͱ͔mon͕མͪͯ΋ࣗಈͰ෮چͯ͘͠ΕΔ - PVCͱ͔Ͱ࢖͍͍ͨͳ.....
 - CephΛߏங͢ΔVM͕͍ͳ͍


    - ΦϒδΣΫτετϨʔδͱͯ͠࢖͍͍ͨͳ.... Ϋϥελ಺ͷϛχϚϜCeph !40

41. • Կ͕ىͬͨ͜ - ϊʔυͷϝϯςͷ࣮ࢪ + ো֐౳ͰCephͷΫϥελ͕յΕͨ - PVCΛར༻͍ͯ͠ΔPod͕མͪɺ࠷ऴతʹOpenStack΋࢖͑ͳ͘ͳͬͨ Kubernetes Control

    Plane osd - A Ceph-mon Kubernetes Control Plane Kubernetes Control Plane osd - A Ceph-mon osd - A Ceph-mon Ceph-mgr ࢒Γ
 ϝϯςத (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ !41

42. • Կ͕ىͬͨ͜ - ϊʔυͷϝϯςͷ࣮ࢪ + ো֐౳ͰCephͷΫϥελ͕յΕͨ - PVCΛར༻͍ͯ͠ΔPod͕མͪɺ࠷ऴతʹOpenStack΋࢖͑ͳ͘ͳͬͨ Kubernetes Control

    Plane osd - A Ceph-mon Kubernetes Control Plane Kubernetes Control Plane osd - A Ceph-mon osd - A Ceph-mon Ceph-mgr DatabaseA DatabaseA DatabaseB DatabaseA (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ OpenStack OpenStack OpenStack ࢒Γ
 ϝϯςத !42

43. • Կ͕ىͬͨ͜ - ϊʔυͷϝϯςͷ࣮ࢪ + ো֐౳ͰCephͷΫϥελ͕յΕͨ - PVCΛར༻͍ͯ͠ΔPod͕མͪɺ࠷ऴతʹOpenStack΋࢖͑ͳ͘ͳͬͨ
 Kubernetes Control

    Plane osd - A Ceph-mon Kubernetes Control Plane Kubernetes Control Plane osd - A Ceph-mon osd - A Ceph-mon Ceph-mgr DatabaseA DatabaseA DatabaseB DatabaseA (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ OpenStack OpenStack OpenStack ࢒Γ
 ϝϯςத !43

44. • Կ͕ىͬͨ͜ - ϊʔυͷϝϯςͷ࣮ࢪ + ো֐౳ͰCephͷΫϥελ͕յΕͨ
 - PVCΛར༻͍ͯ͠ΔPod͕མͪɺ࠷ऴతʹOpenStack΋࢖͑ͳ͘ͳͬͨ Kubernetes Control

    Plane osd - A Ceph-mon Kubernetes Control Plane Kubernetes Control Plane osd - A Ceph-mon Ceph-mgr DatabaseA DatabaseB DatabaseA (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ OpenStack OpenStack ࢒Γ
 ϝϯςத !44

45. • Կ͕ىͬͨ͜ - ϊʔυͷϝϯςͷ࣮ࢪ + ো֐౳ͰCephͷΫϥελ͕յΕͨ
 - PVCΛར༻͍ͯ͠ΔPod͕མͪɺ࠷ऴతʹOpenStack΋࢖͑ͳ͘ͳͬͨ Kubernetes Control

    Plane osd - A Ceph-mon Kubernetes Control Plane Kubernetes Control Plane osd - A Ceph-mon Ceph-mgr DatabaseA DatabaseB DatabaseA (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ OpenStack OpenStack ࢒Γ
 ϝϯςத !45

46. • Կ͕ىͬͨ͜ - ϊʔυͷϝϯςͷ࣮ࢪ + ো֐౳ͰCephͷΫϥελ͕յΕͨ
 - PVCΛར༻͍ͯ͠ΔPod͕མͪɺ࠷ऴతʹOpenStack΋࢖͑ͳ͘ͳͬͨ Kubernetes Control

    Plane osd - A Ceph-mon Kubernetes Control Plane Kubernetes Control Plane osd - A Ceph-mon Ceph-mgr DatabaseA DatabaseB DatabaseA (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ OpenStack OpenStack ࢒Γ
 ϝϯςத !46

47. • Կ͕ىͬͨ͜ - ϊʔυͷϝϯςͷ࣮ࢪ + ো֐౳ͰCephͷΫϥελ͕յΕͨ
 - PVCΛར༻͍ͯ͠ΔPod͕མͪɺ࠷ऴతʹOpenStack΋࢖͑ͳ͘ͳͬͨ Kubernetes Control

    Plane osd - A Ceph-mon Kubernetes Control Plane Kubernetes Control Plane osd - A Ceph-mon Ceph-mgr DatabaseA DatabaseB DatabaseA (஫) ਤ͸ΠϝʔδͷͨΊߏ੒΍഑ஔ͸ݫີʹ͸ҟͳΓ·͢ OpenStack OpenStack ࢒Γ
 ϝϯςத !47

48. Infra Worker Node osd - A Ceph-mon Infra Worker Node

    Infra Worker Node osd - A Ceph-mon osd - A Ceph-mon Ceph-mgr ɾɾN • Ͳ͏ͨ͠ํ͕ྑ͔ͬͨʁ - ϊʔυ਺͸ͨ͘͞Μ͋ͬͨํ͕ྑ͍ !48

49. Ξϯνύλʔϯ3 Ϋϥελ಺ͷPrivate Registry !49

50. • ͳͥͦ͏ͨ͠ʁ - Registryͷؔ࿈ίϯϙʔωϯτ͕མͪͯ΋࠶ىಈͯ͘͠ΕΔ͔Β͖ͬͱָ - CephͷPVC࢖͑͹؆୯Ͱ͸ʁʁ Kubernetes Control Plane Kubernetes

    
 Control Plane XXXXXX kube-apiserver kube-controller kube-scheduler XXXXXX Registry Infra Worker Node XXXXXX XXXXXX Kubernetes 
 Control Plane Ϋϥελ಺ͷPrivate Registry !50

51. Kubernetes Control Plane Kubernetes 
 Control Plane XXXXXX kube-apiserver kube-controller

    kube-scheduler XXXXXX Registry Infra Worker Node XXXXXX XXXXXX Kubernetes 
 Control Plane • Կ͕ىͬͨ͜ - Registry͕ଘࡏ͢Δϊʔυ͕མͪͨ৔߹ɺRegistry͕෮چ͢Δ·ͰΠϝʔδΛϓϧͰ͖ͳ͍ ৔߹͕͋Γɺ௨ৗΑΓ෮چʹ͕͔͔࣌ؒΔ৔߹΋͋Δ - Ceph่͕յͨ͠λΠϛϯάͰ࢖͑ͳ͘ͳͬͨ !51

52. Kubernetes Control Plane Kubernetes 
 Control Plane XXXXXX kube-apiserver kube-controller

    kube-scheduler XXXXXX Registry Infra Worker Node XXXXXX XXXXXX Kubernetes 
 Control Plane • Կ͕ىͬͨ͜ - Registry͕ଘࡏ͢Δϊʔυ͕མͪͨ৔߹ɺRegistry͕෮چ͢Δ·ͰΠϝʔδΛϓϧͰ͖ͳ͍ ৔߹͕͋Γɺ௨ৗΑΓ෮چʹ͕͔͔࣌ؒΔ৔߹΋͋Δ - Ceph่͕յͨ͠λΠϛϯάͰ࢖͑ͳ͘ͳͬͨ ͋Δϊʔυ͕μ΢ϯ
 (ϨδετϦͷؔ࿈ίϯϙʔωϯτ͕ଘࡏ) !52

53. • Կ͕ىͬͨ͜ - Registry͕ଘࡏ͢Δϊʔυ͕མͪͨ৔߹ɺRegistry͕෮چ͢Δ·ͰΠϝʔδΛϓϧͰ͖ͳ͍ ৔߹͕͋Γɺ௨ৗΑΓ෮چʹ͕͔͔࣌ؒΔ৔߹΋͋Δ - Ceph่͕յͨ͠λΠϛϯάͰ࢖͑ͳ͘ͳͬͨ Kubernetes Control Plane

    Infra Worker Node kube-apiserver kube-controller kube-scheduler Registry Infra Worker Node XXXXXX XXXXXX ϨδετϦ͕෮چ͢Δ·Ͱ͸
 ϊʔυ্ʹPod͕࡞ΒΕͳ͍ XXXXXX XXXXXX Infra Worker Node XXXXXX XXXXXX Kubernetes 
 Control Plane Kubernetes 
 Control Plane Kubernetes 
 Control Plane !53

54. • Ͳ͏ͨ͠ํ͕ྑ͔ͬͨʁ - k8sΫϥελͷ֎·ͨ͸ผk8sΫϥελʔͷ΋ͷΛར༻ͨ͠ํ͕ࣗ༝౓͸ߴ͍
 (Ҡઃ΋ߦ͍΍͍͢) - ৑௕͠΍͍͢RegistryΛ࢖͏ඞཁ͕͋Δ 
 (όοΫΤϯυʹDB͕͋Δ৔߹͸ͦΕΒ΋ؚΊͯ) Kubernetes

    Control Plane Infra Worker Node XXXXXX kube-apiserver kube-controller kube-scheduler XXXXXX Registry Infra Worker Node XXXXXX XXXXXX ͋Δϊʔυ͕μ΢ϯ Kubernetes 
 Control Plane Kubernetes 
 Control Plane !54

55. • Ͳ͏ͨ͠ํ͕ྑ͔ͬͨʁ - k8sΫϥελͷ֎·ͨ͸ผk8sΫϥελʔͷ΋ͷΛར༻ͨ͠ํ͕ࣗ༝౓͸ߴ͍
 (Ҡઃ΋ߦ͍΍͍͢) - ৑௕͠΍͍͢RegistryΛ࢖͏ඞཁ͕͋Δ 
 (όοΫΤϯυʹDB͕͋Δ৔߹͸ͦΕΒ΋ؚΊͯ) Kubernetes

    Control Plane Infra Worker Node XXXXXX kube-apiserver kube-controller kube-scheduler XXXXXX Registry Infra Worker Node XXXXXX XXXXXX ଞͷϊʔυ্ʹpod͕࡞ΒΕΔ Kubernetes 
 Control Plane Kubernetes 
 Control Plane !55

56. Ξϯνύλʔϯ4
 Ϋϥελ಺ͷPrometheus !56

57. • ͳͥͦ͏ͨ͠ʁ - ؔ࿈ίϯϙʔωϯτ͕མͪͯ΋࠶ىಈͯ͘͠ΕΔ͔Β͖ͬͱָ - CephͷPVC࢖͑͹؆୯Ͱ͸ʁʁ Ϋϥελ಺ͷPrometheus !57

58. • Կ͕ىͬͨ͜ - Ceph → Prometheus ͜ͷॱ൪Ͱར༻ෆՄʹͳͬͨ - ౰࣌֎ܗ͔Βͷ؂ࢹ͕ͳ͍ͷͰɺ্هͷ໰୊͕ى͖ͯ΋੩͔ͩͬͨ •

    Ͳ͏ͨ͠ํ͕ྑ͔ͬͨʁ - ֎ܗ؂ࢹΛ͢ΔͳΓɺ֎͔ΒKubernetsͷAPIΛୟ͍ͯSDͰ؂ࢹ͢Δύλʔϯ͕ྑ͔ͬͨ ฐࣾͷத੢͕Prometheus Tokyo Meetup #2 ʹͯৄࡉ͸ൃදͯ͠·͢ (ࢿྉ) !58

59. Ξϯνύλʔϯ5
 େ͖ͳWorker NodeΛগͳ͍਺ͰͲʔΜ !59

60. • ͳͥͦ͏ͨ͠ʁ - ComputeϊʔυͰ͸ͳ͍෺ཧαʔόΛ(ry • Կ͕ىͬͨ͜ - Ұ୆ʹ৐ΔPodͷྔ͕͔ͳΓଟ͘ͳΓɺҰ୆མͪͨͱ͖ͷΠϯύΫτ͕େ͖͘ͳͬͨ - ؾܰʹର৅ϊʔυͷϝϯςφϯεΛ͠ਏ͍

    - Ұ୆མͪͨͱ͖ʹPodͷߦ͖ઌ͕ͳ͘ͳ͍ͬͯͨ • Ͳ͏ͨ͠ํ͕ྑ͔ͬͨʁ - ͋Δఔ౓খ͞ͳϊʔυΛͨ͘͞Μฒ΂ͨํ͕ྑ͍(೔ʑͷӡ༻ϝϦοτ͸͋Δ) େ͖ͳWorker NodeΛগͳ͍਺ͰͲʔΜ !60

61. Ξϯνύλʔϯ6
 ίϯςφͷϩάϩʔςΛ͍ͯ͠ͳ͍ !61

62. • ͳͥͦ͏ͨ͠ʁ - લఏͱͯ͠໰୊͕ى͖ͨͷ͸ComputeϊʔυͷϗετͰ
 VM༻ʹϦιʔεΛ֬อ͢Δ΂͘KubernetesͰ࢖͏Ϧιʔε͸ͳΔ΂͘࢖Θͳ͍Α͏ʹ͍ͯͨ͠
 ͕ϩάϩʔςͷ͜ͱΛ׬શʹࣦ೦͍ͯͨ͠ ίϯςφͷϩάϩʔςΛ͍ͯ͠ͳ͍ !62

63. • Կ͕ىͬͨ͜ - PodσϓϩΠ࣌ʹPodͷঢ়ଶ͕Pendingʹͳͬͨ (σϑΥϧτͷϩάϩʔςͩͱ଍Γͳ͍) • Ͳ͏ͨ͠ํ͕ྑ͔ͬͨʁ - docker-logdriverͷઃఆΛద੾ʹ͢Δ΂͖(αΠζ΍ੈ୅؅ཧ) -

    ෆཁͳdebugϩάͳͲ͸ग़͞ͳ͍Α͏ʹ͢Δ !63

64. Ξϯνύλʔϯ7
 in_tailͱϩάϩʔςͷར༻ !64

65. • ͳͥͦ͏ͨ͠ʁ - ϩάͷංେʹΑΓPod͕εέδϡʔϦϯά͞Εͳ͍Α͏ʹͳͬͨͷͰ
 ϩάΛfluentdͰtailͭͭ͠΋1࣌ؒຖʹlogrotateΛಈ͔͍ͯͨ͠ in_tailͱϩάϩʔςͷར༻ !65

66. • Կ͕ىͬͨ͜ - 1࣌ؒͷස౓ͰlogrotateΛͯ͠ɺಉ࣌ʹfluentdͰ”*”Ͱin_tail͍ͯ͠Δͱ fluentd͕ߴස౓ͰLimitʹ౰ͨͬͯOOMͰkill͞ΕΔΑ͏ʹͳͬͨ ௚઀ؔ܎͋Δ͔͸ௐ΂͖Εͯͳ͍͕ɺ͜ͷΑ͏ͳissues΋͋Γ (https://github.com/fluent/fluentd-kubernetes-daemonset/issues/156) (https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/in_tail.rb#L255-L259) • Ͳ͏ͨ͠ํ͕ྑ͔ͬͨʁ

    - ϩάϩʔςΛ͢ΔλΠϛϯά͸௨ৗ௨Γʹ͠ɺੈ୅Λগͳ͘͢ΔΑ͏ʹ͢Δ - in_tailΛར༻͠ͳ͍ํ๏ͰؤுΔ !66

67. Ξϯνύλʔϯ8 ա౓ͳCronJobͷར༻ !67

68. • ͳͥͦ͏ͨ͠ʁ - όονॲཧΛຖ෼ߦͳ͍ͬͯΔ΋ͷ͕ෳ਺͍ͨ - ͦͷଞόοΫΞοϓͳͲ΋جຊతʹ͸CronJobͰճ͍ͯͨ͠ - ͜ΕΛControl PlaneͰճ͍ͯͨ͠ ա౓ͳCronJobͷར༻

    !68

69. • Կ͕ىͬͨ͜ - Podͷ্͛Լ͕͛සൟʹߦΘΕΔ͜ͱͰɺϗετଆͷbuffer/cache͕ղ์͞Εͳ͍Ͱ free memory͕0%ʹͳΔ͜ͱͰϗετμ΢ϯ͕ൃੜͨ͠ ·ͨɺ/sys/fs/cgroup/memory഑Լ͕ҟৗʹͳ͍ͬͯͨ(Cannot Allocate Memory) -

    Control PlaneͰ͜ΕΒ͕ಈ͍͍ͯͨͨΊසൟʹControl Plane͕ෆ҆ఆʹͳͬͨ • Ͳ͏ͨ͠ํ͕ྑ͔͔ͬͨ - ৔౰ͨΓతͳରԠͱͳ͍ͬͯΔ͕ɺ ͦͷଞͷPodʹӨڹΛٴ΅͞ͳ͍Α͏ʹCronJobΛ࣮ߦ͢Δϊʔυ͸labelΛར༻ͯ͠ɺ CronJobՔಇઐ༻ͷWorker Nodeͱͯ͠ෳ਺୆Քಇ͍ͤͯ͞Δ !69

70. ະͩରԠதͷ෦෼΋͋Γ
 ӡ༻Ͱؤு͍ͬͯΔ ߲໨΋͋Γ·͢ !70

71. Ҏ্
 Ξϯνύλʔϯ8બͰͨ͠ !71

72. ·ͱΊ !72

73. ·ͱΊ • ౰࣌͸Kubernets͕͍͍ײ͡ʹ΍ͬͯ͘ΕΔͩΖ͏ͱґଘ͍͗ͯͨ͢͠ • Kubernetesͷࢥ૝͔Β͋·Γʹ΋ζϨͨ࢖͍ํΛ͢Δͱਏ͘ͳΔ • ӡ༻͕໘౗͔ͩΒͱ͍͏ࢹ఺͚ͩͩͱٯʹਏ͘ͳΔ • ͨͩɺKubernetesΛ̍೥΄Ͳӡ༻ͯ͠Έͯศརͳͱ͜Ζ΋
 ৭ʑͱײ͍ͯ͡Δ

    !73

74. ·ͱΊ • ౰࣌͸Kubernets͕͍͍ײ͡ʹ΍ͬͯ͘ΕΔͩΖ͏ͱґଘ͍͗ͯͨ͢͠ • Kubernetesͷࢥ૝͔Β͋·Γʹ΋ζϨͨ࢖͍ํΛ͢Δͱਏ͘ͳΔ • ӡ༻͕໘౗͔ͩΒͱ͍͏ࢹ఺͚ͩͩͱٯʹਏ͘ͳΔ • ͨͩɺkubernetesΛ̍೥΄Ͳӡ༻ͯ͠Έͯศརͳͱ͜Ζ΋
 ৭ʑͱײ͍ͯ͡Δ

    ༻๏ɾ༰ྔΛकͬͯద੾ʹ !74

75. ͋Γ͕ͱ͏͍͟͝·ͨ͠ !75