GKEで作る動画サービス

市ヶ谷Geek★Night#13 GCPはじめました。
https://ichigayageek.connpass.com/event/57962/
Session1：「GKEで作る動画サービス」

Transcript

1. GKEͰ࡞ΔಈըαʔϏε 1
2. ͸͡Ί·ͯ͠ʂ 2
4. AbemaTV αʔϏε঺հ 4
10. ಈըͱ୊͍ͯ͠·͕͢ɺ ಈըٕज़ͷ࿩͸ͳ͍Ͱ͢ 10
11. ΄΅GKEͱkubernetes ͷ࿩Λͤͯ͞௖͖·͢ 11
12. AbemaTV Overview 12
14. GKE 14
15. GKE͸managed Kubernetes 15
16. ࠓ೔ͷ࿩͸GKE্࢖͏ kubernetesͷ࿩Ͱ͢ 16
17. kubernetes 17
18. ൃԻ 18
19. ΫʔόΝωΠςΟε 19
20. Ͱ΋Googleͷਓ͸Ϋϕ ͱ͔Ͱ΋͍͍Αͱݴͬͯ ͨͷͰ͓޷͖ͳײ͡Ͱ 20
21. Կނkubernetes͔ʁ 21
22. Կނkubernetes͔ʁ • ΋ͱ΋ͱ͸Googleࣾ಺ʹ͸Googleݕࡧ΍Gmail΍YouTubeͳ ͲίϯϐϡʔλʔϦιʔε͕ڧ྽ʹඞཁͳΦϯϥΠϯαʔϏε Λ࠷దʹఏڙ͢ΔͨΊʹίϯςφͰӡ༻͍ͨ͠ͱ͍͏ڧྗͳχ ʔζ͕͋ͬͨ 22
23. Կނkubernetes͔ʁ • Google͸ίϯςφϕʔεͰ10೥ӡ༻࣮ͯͨ͠੷͕͋Δ • googleͷΠϯϑϥ͸99%ίϯςφͰಈ͔ͯ͠ར༻Ͱ͖ΔΑ͏ʹͨ͠తͳ࿦จ • Large-scale cluster management at Google with Borg • Borg, Omega, and Kubernetes - ACM Queue • Googleࣾ಺ʹఏڙ͢ΔΠϯϑϥ͸ίϯςφϕʔεɻσʔληϯλʔͷϓϥΠϚϦʔΩʔΛα ʔό͔ΒΞϓϦέʔγϣϯ΁มԽͤͨ͞ • borg -> omegaΛ࡞ͬͯͨਓ͕ͨͪΦʔϓϯιʔεͱͯ͠࠶࣮૷ͨ͠΋ͷ͕kubernetes • KubernetesΛmanagedͳΫϥ΢υαʔϏεͱͯ͠ఏڙ͍ͯ͠Δͷ͕GKE • Kubernetes ͸GoogleΫϥ΢υઓུͷத֩ͱͷ͜ͱ 23
24. Կނkubernetes͔ʁ • kubernetes͸αʔόͷڥքΛҙࣝ͠ͳ͍ΞϓϦέʔγϣϯσϓϩΠΛ Մೳʹ͢Δ • Data Center as a Computer ͱ͍͏ߟ͕͑ࠜͬ͜ʹ͋Δ • ͜ͷߟ͑͸ϚΠΫϩαʔϏεͱ਌࿨ੑ͕ߴ͍ • ϚΠΫϩαʔϏε୯ҐͰͷϦιʔεͷ૿ڧ/ॖୀ͕ϥΫ • Kubernetes1.6Ͱ͸, 5000node, 150000pods·Ͱͷεέʔϧ͕ Մೳ 24
25. Կނkubernetes͔ʁ • ࣮ࡍKubernetesͷத৺ʹ͋Δߟ͑ํ:Ϣʔβ͕๬Ή΂͖ঢ়ଶΛࢦ ఆ͠Kubernetes͕ΫϥελΛͦͷΰʔϧͷঢ়ଶʹҠߦͤ͞Δ͜ ͱʹ͋Δ • GKE͸ෳ਺ͷίϯςφΠϝʔδΛϚωδϝϯτ͢Δʹ͸ɺ͓ͦΒ͘ ݱஈ֊Ͱ͍ͪ͹ΜखܰͰݎ࿚ͳํ๏ • ϝϦοτɿσϓϩΠͷ؆୯͞/ਝ଎͞ɺϦιʔε࢖༻ͷޮ཰ੑɺϙ ʔλϏϦςΟ 25
26. Կނkubernetes͔ʁ • Opsࢹ఺ͰݟΔͱ͜Ε·Ͱconfiguration(chef, ansible)͠ ͯ࢖ͬͯͨVM͸managedͳnodeʹͳͬͨͷͰ, configuration ͸statelessͳίϯϙʔωϯτʹؔͯ͠͸࢖Θͳ͘ͳͬͨ • orchestration(fabric, capistrano)Ͱ΍ͬͯͨࣄ͸ Docker build΍kubernetesʹ·͔ͤΔͷͰɺpodͷ͋Δ΂͖ ঢ়ଶΛهड़͢Δ͚ͩͰΑ͘ͳͬͨ • bootstrap(terraform)෦෼͚ͩ࢒ͬͨ 26
27. Կނkubernetes͔ʁ • ࢖͍ࣺͯՄೳͳΠϛϡʔλϒϧͳProvisioning͸, ҆৺ײ͕ ߴ͍ 27
28. Design 28
29. Design • શͯΛKubernetesͰಈ͔͢ඞཁ͸ͳ͍ɻstateful or stateless ? • statelessͳΞϓϦέʔγϣϯ͸ίϯςφͱ૬ੑ͕ྑ͍ • ͱ͸͍͑statefulsetsΛ࢖ͬͯɺঢ়ଶΛ࣋ͬͨίϯςφΛӡ༻͢Δ͜ͱ΋Ͱ͖ Δ • ઃܭ࣍ୈͩͱࢥ͍·͢ • AbemaTV͸ঢ়ଶΛ࣋ͭΩϟογϡ΋σʔλϕʔε΋ׂΓ੾ΓͰVMͰӡ༻ͯ͠·͢ • ΋͘͠͸managed ͳσʔλετΞΛ࠾༻͢Δͷ΋ྑ͍Ͱ͢Ͷ 29
30. Overview 30
31. Overview: ίϯϙʔωϯτ֓ཁ 31
33. Overview: ίϯϙʔωϯτ֓ཁ • ReplicaSet: Pod(ίϯςφΠϝʔδ܈)ͷՔಈʹඞཁͳ৘ใΛදݱ͢ΔϦιʔε • Service: ֎෦/಺෦ͱͲͷΑ͏ʹ௨৴͢Δ͔(Endpoint)Λදݱ͢ΔϦιʔε • NodePool : Pod͕Քಇ͢ΔVMΛάϧʔϐϯάԽ͢ΔϦιʔε • Node : Pod͕Քಇ͢ΔVM • Pod : ίϯςφΠϝʔδ܈ • Master : kubernetesͷ༷ʑͳ؅ཧΛதԝूݖతʹ؅ཧ͢ΔVMɻmanagedαʔ ϏεͳͷͰීஈҙࣝ͢Δ͜ͱ͸͋·Γͳ͍ 33
34. Overview: ίϯϙʔωϯτৄࡉ 34
37. Overview: ίϯϙʔωϯτৄࡉ • master/api: kubernetesΛૢ࡞͢ΔͨΊͷAPIΛఏڙ͢Δ • master/controller-manager: ίϯςφͷঢ়ଶ؅ཧ΍ϊʔυͷ؅ཧͱݴ֤ͬͨछ ؅ཧ࡞ۀΛߦ͏ • master/scheduler: ֤ϊʔυʹର͠ίϯςφͷׂΓ౰ͯͳͲΛߦ͏ • master/etcd: ϦιʔεͷӬଓԽ૚ɻ෼ࢄKVS • node/proxy: ίϯςφ΁ͷωοτϫʔΫϧʔςΟϯά͓Αͼෛՙ෼ࢄΛߦ͏ • node/kubelet: ֤ϊʔυ্Ͱͷίϯςφ࡞੒/࡟আ΍ϘϦϡʔϜͷׂΓ౰ͯͳͲΛ ߦ͏ 37
38. Overview: ίϯϙʔωϯτৄࡉ • node/kube-dns: ServiceΛఆٛ͢Δ͜ͱͰPodؒͰ௨৴͢ Δࡍͷ໊લΛղܾͯ͘͠Ε·͢ • node/fluentd-cloud-logging: Podͷඪ४ग़ྗΛ StackdriverLoggingʹૹ৴͠·͢ • node/heapstar: Monitoring and Performance Analysis for Container Cluster 38
39. Overview: ίϯϙʔωϯτৄࡉ • ͍Ζ͍Ζॻ͖·͕ͨ͠ɺ௨ৗӡ༻ঢ়ଶʹ͓͍ͯ͸kubectl΍ StackdriverLogging΍StackdriverΛ௨ͯ͡PodΛ؍࡯͠ ͨΓ੍ޚ͢ΔܗͰ͢ • ӡ༻໘Ͱ͸ͦ͜·ͰෳࡶͰ͸ͳ͍Ͱ͢ • node͸୆਺ͷௐ੔Λ͢Δ৔߹΍ΞοϓάϨʔυ࣌ʹ͸ҙࣝ͢Δ • ໰୊͕͋ͬͨ৔߹ɺnodeʹೖͬͯ৘ใऩू͢Δ͜ͱ͸͋Δ 39
40. ىಈ͢Δ·ͰͷྲྀΕ 40
42. • masterͱnodeͰ͍Ζ͍Ζ΍ΓऔΓͯ͠·͕͢ɺReadinessʹ ൓Ԡ͠RunningʹભҠ͢Δ·Ͱ͸ඵ୯Ґͷىಈ଎౓Ͱ͢ʢମ ײʣ 42
43. Deployment 43
44. • kubernetes v1.2͔Βར༻ՄೳͰ͕ͨ͠Rc͔ΒDeployment ʹҠߦ͠·ͨ͠ • ઌఔͷReplicaSet͸DeploymentʹΑͬͯࣗಈ؅ཧ͞Ε͍ͯ ·͢ • ͢͜͠લ͸ReplicationController࢖ͬͯ·͕ͨ͠ɺϚΠΫ ϩαʔϏε෦෼͸DeploymentʹҠߦ͠·ͨ͠ 44
45. rc manifest apiVersion: v1 kind: ReplicationController metadata: name: nginx spec: replicas: 3 selector: app: nginx template: metadata: name: nginx labels: app: nginx env: production spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 45
46. deployment manifest apiVersion: apps/v1beta1 # for versions before 1.6.0 use extensions/v1beta1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 3 template: metadata: name: nginx labels: app: nginx env: production spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 46
47. ͋·ΓมΘΓ͕ແ͋͘Γ ͕͍ͨʂ 47
49. • Rc࣌୅͸reload͸kubectl ͷwrapperπʔϧͰreloadͯ͠ ·ͨ͠ɻͦΕ΋ෆཁʹͳΓ·ͨ͠ • ͜ΕΒૢ࡞͸kubectl͔ΒͰ΋ग़དྷ·͕͢ɺAbemaTVͰ͸ ChatOpsʹΑͬͯdeployͯ͠·͢(΅͘ʹΑΔ࢓ࣄͰ͸ͳ͘ DevͷํʹΑΔ࢓ࣄͰ͢ɻ͋Γ͕ͨ΍) • Ͱ΋ඞཁʹԠͯ͡ɺkubectl΋࢖ͬͨΓ͸͠·͢(ো֐ରԠͳ Ͳ) 49
50. ͱ͋Δ೔ͷdeploy • ͋Δϩοτ୯ҐͰrolling update͞Ε͍ͯΔ༷ࢠ͕؍࡯Ͱ͖Δ 50
51. ϩά 51
53. ϩά • GKE͸pod͕ඪ४ग़ྗʹग़ྗͨ݁͠ՌΛfluentd-cloud- loggingΛ௨ͯ͠StackdriverLoggingʹࣗಈసૹͯ͘͠Ε ͯͱͯ΋ϥΫ • ΞΫηεϩάͳͲϢʔβߦಈʹؔ܎͠ผ్ूܭ͍ͨ͠΋ͷ͸ Stream Insertܦ༝ͰBigqueryʹೖΕͯ·͢ • ෳ਺ͷςʔϒϧʹ෼ׂ͠ɺQuotaʹ͞͞Βͳ͍Α͏޻෉͕ඞཁ 53
54. Network 54
57. Network : Service 21:26:26 [0] % kubectl describe svc my-service Name: my-service Namespace: default Labels: name=my-service Selector: name=my-service Type: NodePort IP: xx.xx.xxx.xxx Port: http 80/TCP NodePort: http 30001/TCP Endpoints: xx.xx.x.x:80,xx.xx.xx.x:80,xx.xx.x.x:80 Session Affinity: None • Port: αʔϏεVIP্ͷԾ૝Port • NodePort: ϗετຖʹଘࡏ͢ΔNodePortͱPodͷContainerPortΛϚοϐϯά͢ΔPort 57
62. Network : Service • ServiceΛఆٛ͢Δ͜ͱͰɺPod܈ʹର͢Δ઀ଓΛ؆୯ʹઃఆ Ͱ͖·͢ 62
63. k8s͞ΘͬͯݟΑ͏ͱࢥͬͨํ΁ • Ұ൪खܰͳͷ͸GKEͩͱࢥ͍·͢ • GKEΞΧ΢ϯτͳ͍৔߹͸͜Μͳπʔϧ΋͋Γ·͢ 63
64. Tool: minikube 64
65. Tool: minikube localͷkubernetes؀ڥΛ࡞੒Ͱ͖·͢ % curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.7.1/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/ % minikube start 65
66. Tool: minikube • kubernetesͷίϯϙʔωϯτͰՔಇ͍ͯ͠ΔΘ͚Ͱ͸ͳ͍ͷ Ͱɺຊ֨తͳݕূʹ͸࢖Θͳ͍΄͏͕ྑ͍Ͱ͢ • VagrantͷΑ͏ͳҐஔ෇͚ͩͱࢥ͍·͢ 16:03:16 [0] % kubectl get pod --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default nginx-ydej2 1/1 Running 0 58m kube-system kube-addon-manager-minikubevm 1/1 Running 0 1d kube-system kubernetes-dashboard-xj6g7 1/1 Running 0 1d 66
67. Tool: minikube GKE΍minikubeͳͲෳ਺ͷclusterΛ࢖ͬͯΔ৔߹͸ɺखݩͷ minikube clusterʹҎԼͰ੾Γସ͑Ͱ͖·͢ % kubectl config use-context minikube 67
68. tutorial: GuestBook 68
70. tutorial: GuestBook • PHPΞϓϦέʔγϣϯͱRedisΛ࢖ͬͨ؆୯ͳtutorial • GKEͷ৔߹͸্هrepoͷREADME.mdʹैͬͯಈ͔ͯ͠ΈΔͱɺGKE ͷײ৮͕Θ͔ͬͯྑ͍ͱࢥ͍·͢ • minikubeͩͱDeploymentͳͲ৽͍͠Resource͕ಈ͔ͪ͘ΐͬͱ ࣗ৴͕ͳ͍ • umiyosh/kubetutorial : 1೥લ͘Β͍ʹ࡞ͬͨ΍ͭͰ͕͢RC ϕʔεͳͷͰminikubeͰ΋ಈ͘ͱࢥ͍·͢ 70
71. ݟͱ͍ͨ΄͏͕͍͍΍ͭ 71
72. ݟͱ͍ͨ΄͏͕͍͍΍ͭ • ໌Β͔ʹkube࢖͍࢝Ίͨͱ͖ʹཉ͔ͬͨ͠΍ͭ • kube࢖͏৔߹ɺ֤resourceΛΘ͔Γ΍͘͢આ໌ͯ͘͠ΕͯΔͷͰઈରಡΜͩ΄͏͕͍͍Ͱ͢ 72
73. ݟͱ͍ͨ΄͏͕͍͍΍ͭ % kubectl explain pod.spec.nodeSelector FIELD: nodeSelector <map[string]string> DESCRIPTION: NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node. More info: http://releases.k8s.io/release-1.2/docs/user-guide/node-selection/README.md • → ؆қͳઆ໌ʴkubernetes.io͔kubernetes/ kubernetes/docʹ༠ಋͯ͘͠ΕΔ 73
74. ݟͱ͍ͨ΄͏͕͍͍΍ͭ • ެࣜυΩϡϝϯτ 74
75. ݟͱ͍ͨ΄͏͕͍͍΍ͭ • ެࣜϦϙδτϦ 75
76. ݟͱ͍ͨ΄͏͕͍͍΍ͭ • gcpug slack ͷ k8s_gke_ja νϟϯωϧ 76
77. ݟͱ͍ͨ΄͏͕͍͍΍ͭ • kubernetes slackνϟωϧ 77
78. • Kubernetes - Production-Grade Container orchestration • kubernetes/kubernetes: Production-Grade Container Scheduling and Management • k8s Slack • gcpug Slack 78
79. ·ͱΊ 79
80. ·ͱΊ • DockerͷΦʔέετϨʔγϣϯ͸kubernetes͕ͱͯ΋ϥΫͰ ݱ࣮త • KubernetesΛ࢖͏ͳΒGKE͕ϥΫͰݱ࣮త • ϋΠεϧʔϓοτͷαʔϏεͰ΋҆ఆͯ͠࢖͍͑ͯ·͢ 80
81. σʔληϯλʔͷϓϥΠ ϚϦʔΩʔΛαʔό͔Β ΞϓϦέʔγϣϯ΁มԽ ͤ͞·͠ΐ͏ 81
82. ͓͠·͍ 82