GKEで作る動画サービス

Bdd614bcea1c5ceda5bbb7b0343b5838?s=47 umiyosh
June 28, 2017
960

 GKEで作る動画サービス

市ヶ谷Geek★Night#13 GCPはじめました。
https://ichigayageek.connpass.com/event/57962/
Session1:「GKEで作る動画サービス」

Bdd614bcea1c5ceda5bbb7b0343b5838?s=128

umiyosh

June 28, 2017
Tweet

Transcript

  1. GKEͰ࡞ΔಈըαʔϏε 1

  2. ͸͡Ί·ͯ͠ʂ 2

  3. 3

  4. AbemaTV αʔϏε঺հ 4

  5. 5

  6. 6

  7. 7

  8. 8

  9. 9

  10. ಈըͱ୊͍ͯ͠·͕͢ɺ ಈըٕज़ͷ࿩͸ͳ͍Ͱ͢ 10

  11. ΄΅GKEͱkubernetes ͷ࿩Λͤͯ͞௖͖·͢ 11

  12. AbemaTV Overview 12

  13. 13

  14. GKE 14

  15. GKE͸managed Kubernetes 15

  16. ࠓ೔ͷ࿩͸GKE্࢖͏ kubernetesͷ࿩Ͱ͢ 16

  17. kubernetes 17

  18. ൃԻ 18

  19. ΫʔόΝωΠςΟε 19

  20. Ͱ΋Googleͷਓ͸Ϋϕ ͱ͔Ͱ΋͍͍Αͱݴͬͯ ͨͷͰ͓޷͖ͳײ͡Ͱ 20

  21. Կނkubernetes͔ʁ 21

  22. Կނkubernetes͔ʁ • ΋ͱ΋ͱ͸Googleࣾ಺ʹ͸Googleݕࡧ΍Gmail΍YouTubeͳ ͲίϯϐϡʔλʔϦιʔε͕ڧ྽ʹඞཁͳΦϯϥΠϯαʔϏε Λ࠷దʹఏڙ͢ΔͨΊʹίϯςφͰӡ༻͍ͨ͠ͱ͍͏ڧྗͳχ ʔζ͕͋ͬͨ 22

  23. Կނkubernetes͔ʁ • Google͸ίϯςφϕʔεͰ10೥ӡ༻࣮ͯͨ͠੷͕͋Δ • googleͷΠϯϑϥ͸99%ίϯςφͰಈ͔ͯ͠ར༻Ͱ͖ΔΑ͏ʹͨ͠తͳ࿦จ • Large-scale cluster management at

    Google with Borg • Borg, Omega, and Kubernetes - ACM Queue • Googleࣾ಺ʹఏڙ͢ΔΠϯϑϥ͸ίϯςφϕʔεɻσʔληϯλʔͷϓϥΠϚϦʔΩʔΛα ʔό͔ΒΞϓϦέʔγϣϯ΁มԽͤͨ͞ • borg -> omegaΛ࡞ͬͯͨਓ͕ͨͪΦʔϓϯιʔεͱͯ͠࠶࣮૷ͨ͠΋ͷ͕kubernetes • KubernetesΛmanagedͳΫϥ΢υαʔϏεͱͯ͠ఏڙ͍ͯ͠Δͷ͕GKE • Kubernetes ͸GoogleΫϥ΢υઓུͷத֩ͱͷ͜ͱ 23
  24. Կނkubernetes͔ʁ • kubernetes͸αʔόͷڥքΛҙࣝ͠ͳ͍ΞϓϦέʔγϣϯσϓϩΠΛ Մೳʹ͢Δ • Data Center as a Computer

    ͱ͍͏ߟ͕͑ࠜͬ͜ʹ͋Δ • ͜ͷߟ͑͸ϚΠΫϩαʔϏεͱ਌࿨ੑ͕ߴ͍ • ϚΠΫϩαʔϏε୯ҐͰͷϦιʔεͷ૿ڧ/ॖୀ͕ϥΫ • Kubernetes1.6Ͱ͸, 5000node, 150000pods·Ͱͷεέʔϧ͕ Մೳ 24
  25. Կނkubernetes͔ʁ • ࣮ࡍKubernetesͷத৺ʹ͋Δߟ͑ํ:Ϣʔβ͕๬Ή΂͖ঢ়ଶΛࢦ ఆ͠Kubernetes͕ΫϥελΛͦͷΰʔϧͷঢ়ଶʹҠߦͤ͞Δ͜ ͱʹ͋Δ • GKE͸ෳ਺ͷίϯςφΠϝʔδΛϚωδϝϯτ͢Δʹ͸ɺ͓ͦΒ͘ ݱஈ֊Ͱ͍ͪ͹ΜखܰͰݎ࿚ͳํ๏ • ϝϦοτɿσϓϩΠͷ؆୯͞/ਝ଎͞ɺϦιʔε࢖༻ͷޮ཰ੑɺϙ

    ʔλϏϦςΟ 25
  26. Կނkubernetes͔ʁ • Opsࢹ఺ͰݟΔͱ͜Ε·Ͱconfiguration(chef, ansible)͠ ͯ࢖ͬͯͨVM͸managedͳnodeʹͳͬͨͷͰ, configuration ͸statelessͳίϯϙʔωϯτʹؔͯ͠͸࢖Θͳ͘ͳͬͨ • orchestration(fabric, capistrano)Ͱ΍ͬͯͨࣄ͸

    Docker build΍kubernetesʹ·͔ͤΔͷͰɺpodͷ͋Δ΂͖ ঢ়ଶΛهड़͢Δ͚ͩͰΑ͘ͳͬͨ • bootstrap(terraform)෦෼͚ͩ࢒ͬͨ 26
  27. Կނkubernetes͔ʁ • ࢖͍ࣺͯՄೳͳΠϛϡʔλϒϧͳProvisioning͸, ҆৺ײ͕ ߴ͍ 27

  28. Design 28

  29. Design • શͯΛKubernetesͰಈ͔͢ඞཁ͸ͳ͍ɻstateful or stateless ? • statelessͳΞϓϦέʔγϣϯ͸ίϯςφͱ૬ੑ͕ྑ͍ • ͱ͸͍͑statefulsetsΛ࢖ͬͯɺঢ়ଶΛ࣋ͬͨίϯςφΛӡ༻͢Δ͜ͱ΋Ͱ͖

    Δ • ઃܭ࣍ୈͩͱࢥ͍·͢ • AbemaTV͸ঢ়ଶΛ࣋ͭΩϟογϡ΋σʔλϕʔε΋ׂΓ੾ΓͰVMͰӡ༻ͯ͠·͢ • ΋͘͠͸managed ͳσʔλετΞΛ࠾༻͢Δͷ΋ྑ͍Ͱ͢Ͷ 29
  30. Overview 30

  31. Overview: ίϯϙʔωϯτ֓ཁ 31

  32. 32

  33. Overview: ίϯϙʔωϯτ֓ཁ • ReplicaSet: Pod(ίϯςφΠϝʔδ܈)ͷՔಈʹඞཁͳ৘ใΛදݱ͢ΔϦιʔε • Service: ֎෦/಺෦ͱͲͷΑ͏ʹ௨৴͢Δ͔(Endpoint)Λදݱ͢ΔϦιʔε • NodePool

    : Pod͕Քಇ͢ΔVMΛάϧʔϐϯάԽ͢ΔϦιʔε • Node : Pod͕Քಇ͢ΔVM • Pod : ίϯςφΠϝʔδ܈ • Master : kubernetesͷ༷ʑͳ؅ཧΛதԝूݖతʹ؅ཧ͢ΔVMɻmanagedαʔ ϏεͳͷͰීஈҙࣝ͢Δ͜ͱ͸͋·Γͳ͍ 33
  34. Overview: ίϯϙʔωϯτৄࡉ 34

  35. 35

  36. 36

  37. Overview: ίϯϙʔωϯτৄࡉ • master/api: kubernetesΛૢ࡞͢ΔͨΊͷAPIΛఏڙ͢Δ • master/controller-manager: ίϯςφͷঢ়ଶ؅ཧ΍ϊʔυͷ؅ཧͱݴ֤ͬͨछ ؅ཧ࡞ۀΛߦ͏ •

    master/scheduler: ֤ϊʔυʹର͠ίϯςφͷׂΓ౰ͯͳͲΛߦ͏ • master/etcd: ϦιʔεͷӬଓԽ૚ɻ෼ࢄKVS • node/proxy: ίϯςφ΁ͷωοτϫʔΫϧʔςΟϯά͓Αͼෛՙ෼ࢄΛߦ͏ • node/kubelet: ֤ϊʔυ্Ͱͷίϯςφ࡞੒/࡟আ΍ϘϦϡʔϜͷׂΓ౰ͯͳͲΛ ߦ͏ 37
  38. Overview: ίϯϙʔωϯτৄࡉ • node/kube-dns: ServiceΛఆٛ͢Δ͜ͱͰPodؒͰ௨৴͢ Δࡍͷ໊લΛղܾͯ͘͠Ε·͢ • node/fluentd-cloud-logging: Podͷඪ४ग़ྗΛ StackdriverLoggingʹૹ৴͠·͢

    • node/heapstar: Monitoring and Performance Analysis for Container Cluster 38
  39. Overview: ίϯϙʔωϯτৄࡉ • ͍Ζ͍Ζॻ͖·͕ͨ͠ɺ௨ৗӡ༻ঢ়ଶʹ͓͍ͯ͸kubectl΍ StackdriverLogging΍StackdriverΛ௨ͯ͡PodΛ؍࡯͠ ͨΓ੍ޚ͢ΔܗͰ͢ • ӡ༻໘Ͱ͸ͦ͜·ͰෳࡶͰ͸ͳ͍Ͱ͢ • node͸୆਺ͷௐ੔Λ͢Δ৔߹΍ΞοϓάϨʔυ࣌ʹ͸ҙࣝ͢Δ

    • ໰୊͕͋ͬͨ৔߹ɺnodeʹೖͬͯ৘ใऩू͢Δ͜ͱ͸͋Δ 39
  40. ىಈ͢Δ·ͰͷྲྀΕ 40

  41. 41

  42. • masterͱnodeͰ͍Ζ͍Ζ΍ΓऔΓͯ͠·͕͢ɺReadinessʹ ൓Ԡ͠RunningʹભҠ͢Δ·Ͱ͸ඵ୯Ґͷىಈ଎౓Ͱ͢ʢମ ײʣ 42

  43. Deployment 43

  44. • kubernetes v1.2͔Βར༻ՄೳͰ͕ͨ͠Rc͔ΒDeployment ʹҠߦ͠·ͨ͠ • ઌఔͷReplicaSet͸DeploymentʹΑͬͯࣗಈ؅ཧ͞Ε͍ͯ ·͢ • ͢͜͠લ͸ReplicationController࢖ͬͯ·͕ͨ͠ɺϚΠΫ ϩαʔϏε෦෼͸DeploymentʹҠߦ͠·ͨ͠

    44
  45. rc manifest apiVersion: v1 kind: ReplicationController metadata: name: nginx spec:

    replicas: 3 selector: app: nginx template: metadata: name: nginx labels: app: nginx env: production spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 45
  46. deployment manifest apiVersion: apps/v1beta1 # for versions before 1.6.0 use

    extensions/v1beta1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 3 template: metadata: name: nginx labels: app: nginx env: production spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 46
  47. ͋·ΓมΘΓ͕ແ͋͘Γ ͕͍ͨʂ 47

  48. 48

  49. • Rc࣌୅͸reload͸kubectl ͷwrapperπʔϧͰreloadͯ͠ ·ͨ͠ɻͦΕ΋ෆཁʹͳΓ·ͨ͠ • ͜ΕΒૢ࡞͸kubectl͔ΒͰ΋ग़དྷ·͕͢ɺAbemaTVͰ͸ ChatOpsʹΑͬͯdeployͯ͠·͢(΅͘ʹΑΔ࢓ࣄͰ͸ͳ͘ DevͷํʹΑΔ࢓ࣄͰ͢ɻ͋Γ͕ͨ΍) • Ͱ΋ඞཁʹԠͯ͡ɺkubectl΋࢖ͬͨΓ͸͠·͢(ো֐ରԠͳ

    Ͳ) 49
  50. ͱ͋Δ೔ͷdeploy • ͋Δϩοτ୯ҐͰrolling update͞Ε͍ͯΔ༷ࢠ͕؍࡯Ͱ͖Δ 50

  51. ϩά 51

  52. 52

  53. ϩά • GKE͸pod͕ඪ४ग़ྗʹग़ྗͨ݁͠ՌΛfluentd-cloud- loggingΛ௨ͯ͠StackdriverLoggingʹࣗಈసૹͯ͘͠Ε ͯͱͯ΋ϥΫ • ΞΫηεϩάͳͲϢʔβߦಈʹؔ܎͠ผ్ूܭ͍ͨ͠΋ͷ͸ Stream Insertܦ༝ͰBigqueryʹೖΕͯ·͢ •

    ෳ਺ͷςʔϒϧʹ෼ׂ͠ɺQuotaʹ͞͞Βͳ͍Α͏޻෉͕ඞཁ 53
  54. Network 54

  55. 55

  56. 56

  57. Network : Service 21:26:26 [0] % kubectl describe svc my-service

    Name: my-service Namespace: default Labels: name=my-service Selector: name=my-service Type: NodePort IP: xx.xx.xxx.xxx Port: http 80/TCP NodePort: http 30001/TCP Endpoints: xx.xx.x.x:80,xx.xx.xx.x:80,xx.xx.x.x:80 Session Affinity: None • Port: αʔϏεVIP্ͷԾ૝Port • NodePort: ϗετຖʹଘࡏ͢ΔNodePortͱPodͷContainerPortΛϚοϐϯά͢ΔPort 57
  58. 58

  59. 59

  60. 60

  61. 61

  62. Network : Service • ServiceΛఆٛ͢Δ͜ͱͰɺPod܈ʹର͢Δ઀ଓΛ؆୯ʹઃఆ Ͱ͖·͢ 62

  63. k8s͞ΘͬͯݟΑ͏ͱࢥͬͨํ΁ • Ұ൪खܰͳͷ͸GKEͩͱࢥ͍·͢ • GKEΞΧ΢ϯτͳ͍৔߹͸͜Μͳπʔϧ΋͋Γ·͢ 63

  64. Tool: minikube 64

  65. Tool: minikube localͷkubernetes؀ڥΛ࡞੒Ͱ͖·͢ % curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.7.1/minikube-darwin-amd64 && chmod

    +x minikube && sudo mv minikube /usr/local/bin/ % minikube start 65
  66. Tool: minikube • kubernetesͷίϯϙʔωϯτͰՔಇ͍ͯ͠ΔΘ͚Ͱ͸ͳ͍ͷ Ͱɺຊ֨తͳݕূʹ͸࢖Θͳ͍΄͏͕ྑ͍Ͱ͢ • VagrantͷΑ͏ͳҐஔ෇͚ͩͱࢥ͍·͢ 16:03:16 [0] %

    kubectl get pod --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default nginx-ydej2 1/1 Running 0 58m kube-system kube-addon-manager-minikubevm 1/1 Running 0 1d kube-system kubernetes-dashboard-xj6g7 1/1 Running 0 1d 66
  67. Tool: minikube GKE΍minikubeͳͲෳ਺ͷclusterΛ࢖ͬͯΔ৔߹͸ɺखݩͷ minikube clusterʹҎԼͰ੾Γସ͑Ͱ͖·͢ % kubectl config use-context minikube

    67
  68. tutorial: GuestBook 68

  69. 69

  70. tutorial: GuestBook • PHPΞϓϦέʔγϣϯͱRedisΛ࢖ͬͨ؆୯ͳtutorial • GKEͷ৔߹͸্هrepoͷREADME.mdʹैͬͯಈ͔ͯ͠ΈΔͱɺGKE ͷײ৮͕Θ͔ͬͯྑ͍ͱࢥ͍·͢ • minikubeͩͱDeploymentͳͲ৽͍͠Resource͕ಈ͔ͪ͘ΐͬͱ ࣗ৴͕ͳ͍

    • umiyosh/kubetutorial : 1೥લ͘Β͍ʹ࡞ͬͨ΍ͭͰ͕͢RC ϕʔεͳͷͰminikubeͰ΋ಈ͘ͱࢥ͍·͢ 70
  71. ݟͱ͍ͨ΄͏͕͍͍΍ͭ 71

  72. ݟͱ͍ͨ΄͏͕͍͍΍ͭ • ໌Β͔ʹkube࢖͍࢝Ίͨͱ͖ʹཉ͔ͬͨ͠΍ͭ • kube࢖͏৔߹ɺ֤resourceΛΘ͔Γ΍͘͢આ໌ͯ͘͠ΕͯΔͷͰઈରಡΜͩ΄͏͕͍͍Ͱ͢ 72

  73. ݟͱ͍ͨ΄͏͕͍͍΍ͭ % kubectl explain pod.spec.nodeSelector FIELD: nodeSelector <map[string]string> DESCRIPTION: NodeSelector

    is a selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node. More info: http://releases.k8s.io/release-1.2/docs/user-guide/node-selection/README.md • → ؆қͳઆ໌ʴkubernetes.io͔kubernetes/ kubernetes/docʹ༠ಋͯ͘͠ΕΔ 73
  74. ݟͱ͍ͨ΄͏͕͍͍΍ͭ • ެࣜυΩϡϝϯτ 74

  75. ݟͱ͍ͨ΄͏͕͍͍΍ͭ • ެࣜϦϙδτϦ 75

  76. ݟͱ͍ͨ΄͏͕͍͍΍ͭ • gcpug slack ͷ k8s_gke_ja νϟϯωϧ 76

  77. ݟͱ͍ͨ΄͏͕͍͍΍ͭ • kubernetes slackνϟωϧ 77

  78. • Kubernetes - Production-Grade Container orchestration • kubernetes/kubernetes: Production-Grade Container

    Scheduling and Management • k8s Slack • gcpug Slack 78
  79. ·ͱΊ 79

  80. ·ͱΊ • DockerͷΦʔέετϨʔγϣϯ͸kubernetes͕ͱͯ΋ϥΫͰ ݱ࣮త • KubernetesΛ࢖͏ͳΒGKE͕ϥΫͰݱ࣮త • ϋΠεϧʔϓοτͷαʔϏεͰ΋҆ఆͯ͠࢖͍͑ͯ·͢ 80

  81. σʔληϯλʔͷϓϥΠ ϚϦʔΩʔΛαʔό͔Β ΞϓϦέʔγϣϯ΁มԽ ͤ͞·͠ΐ͏ 81

  82. ͓͠·͍ 82