GKEで作る動画サービス

by umiyosh

Slide 1

Slide 1 text

GKEͰ࡞ΔಈըαʔϏε 1

Slide 2

Slide 2 text

͸͡Ί·ͯ͠ʂ 2

Slide 3

Slide 3 text

Slide 4

Slide 4 text

AbemaTV αʔϏε঺հ 4

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

ಈըͱ୊͍ͯ͠·͕͢ɺ ಈըٕज़ͷ࿩͸ͳ͍Ͱ͢ 10

Slide 11

Slide 11 text

΄΅GKEͱkubernetes ͷ࿩Λͤͯ͞௖͖·͢ 11

Slide 12

Slide 12 text

AbemaTV Overview 12

Slide 13

Slide 13 text

Slide 14

Slide 14 text

GKE 14

Slide 15

Slide 15 text

GKE͸managed Kubernetes 15

Slide 16

Slide 16 text

ࠓ೔ͷ࿩͸GKE্࢖͏ kubernetesͷ࿩Ͱ͢ 16

Slide 17

Slide 17 text

kubernetes 17

Slide 18

Slide 18 text

ൃԻ 18

Slide 19

Slide 19 text

ΫʔόΝωΠςΟε 19

Slide 20

Slide 20 text

Ͱ΋Googleͷਓ͸Ϋϕ ͱ͔Ͱ΋͍͍Αͱݴͬͯ ͨͷͰ͓޷͖ͳײ͡Ͱ 20

Slide 21

Slide 21 text

Կނkubernetes͔ʁ 21

Slide 22

Slide 22 text

Կނkubernetes͔ʁ • ΋ͱ΋ͱ͸Googleࣾ಺ʹ͸Googleݕࡧ΍Gmail΍YouTubeͳ ͲίϯϐϡʔλʔϦιʔε͕ڧ྽ʹඞཁͳΦϯϥΠϯαʔϏε Λ࠷దʹఏڙ͢ΔͨΊʹίϯςφͰӡ༻͍ͨ͠ͱ͍͏ڧྗͳχ ʔζ͕͋ͬͨ 22

Slide 23

Slide 23 text

Կނkubernetes͔ʁ • Google͸ίϯςφϕʔεͰ10೥ӡ༻࣮ͯͨ͠੷͕͋Δ • googleͷΠϯϑϥ͸99%ίϯςφͰಈ͔ͯ͠ར༻Ͱ͖ΔΑ͏ʹͨ͠తͳ࿦จ • Large-scale cluster management at Google with Borg • Borg, Omega, and Kubernetes - ACM Queue • Googleࣾ಺ʹఏڙ͢ΔΠϯϑϥ͸ίϯςφϕʔεɻσʔληϯλʔͷϓϥΠϚϦʔΩʔΛα ʔό͔ΒΞϓϦέʔγϣϯ΁มԽͤͨ͞ • borg -> omegaΛ࡞ͬͯͨਓ͕ͨͪΦʔϓϯιʔεͱͯ͠࠶࣮૷ͨ͠΋ͷ͕kubernetes • KubernetesΛmanagedͳΫϥ΢υαʔϏεͱͯ͠ఏڙ͍ͯ͠Δͷ͕GKE • Kubernetes ͸GoogleΫϥ΢υઓུͷத֩ͱͷ͜ͱ 23

Slide 24

Slide 24 text

Կނkubernetes͔ʁ • kubernetes͸αʔόͷڥքΛҙࣝ͠ͳ͍ΞϓϦέʔγϣϯσϓϩΠΛ Մೳʹ͢Δ • Data Center as a Computer ͱ͍͏ߟ͕͑ࠜͬ͜ʹ͋Δ • ͜ͷߟ͑͸ϚΠΫϩαʔϏεͱ਌࿨ੑ͕ߴ͍ • ϚΠΫϩαʔϏε୯ҐͰͷϦιʔεͷ૿ڧ/ॖୀ͕ϥΫ • Kubernetes1.6Ͱ͸, 5000node, 150000pods·Ͱͷεέʔϧ͕ Մೳ 24

Slide 25

Slide 25 text

Կނkubernetes͔ʁ • ࣮ࡍKubernetesͷத৺ʹ͋Δߟ͑ํ:Ϣʔβ͕๬Ή΂͖ঢ়ଶΛࢦ ఆ͠Kubernetes͕ΫϥελΛͦͷΰʔϧͷঢ়ଶʹҠߦͤ͞Δ͜ ͱʹ͋Δ • GKE͸ෳ਺ͷίϯςφΠϝʔδΛϚωδϝϯτ͢Δʹ͸ɺ͓ͦΒ͘ ݱஈ֊Ͱ͍ͪ͹ΜखܰͰݎ࿚ͳํ๏ • ϝϦοτɿσϓϩΠͷ؆୯͞/ਝ଎͞ɺϦιʔε࢖༻ͷޮ཰ੑɺϙ ʔλϏϦςΟ 25

Slide 26

Slide 26 text

Կނkubernetes͔ʁ • Opsࢹ఺ͰݟΔͱ͜Ε·Ͱconfiguration(chef, ansible)͠ ͯ࢖ͬͯͨVM͸managedͳnodeʹͳͬͨͷͰ, configuration ͸statelessͳίϯϙʔωϯτʹؔͯ͠͸࢖Θͳ͘ͳͬͨ • orchestration(fabric, capistrano)Ͱ΍ͬͯͨࣄ͸ Docker build΍kubernetesʹ·͔ͤΔͷͰɺpodͷ͋Δ΂͖ ঢ়ଶΛهड़͢Δ͚ͩͰΑ͘ͳͬͨ • bootstrap(terraform)෦෼͚ͩ࢒ͬͨ 26

Slide 27

Slide 27 text

Կނkubernetes͔ʁ • ࢖͍ࣺͯՄೳͳΠϛϡʔλϒϧͳProvisioning͸, ҆৺ײ͕ ߴ͍ 27

Slide 28

Slide 28 text

Design 28

Slide 29

Slide 29 text

Design • શͯΛKubernetesͰಈ͔͢ඞཁ͸ͳ͍ɻstateful or stateless ? • statelessͳΞϓϦέʔγϣϯ͸ίϯςφͱ૬ੑ͕ྑ͍ • ͱ͸͍͑statefulsetsΛ࢖ͬͯɺঢ়ଶΛ࣋ͬͨίϯςφΛӡ༻͢Δ͜ͱ΋Ͱ͖ Δ • ઃܭ࣍ୈͩͱࢥ͍·͢ • AbemaTV͸ঢ়ଶΛ࣋ͭΩϟογϡ΋σʔλϕʔε΋ׂΓ੾ΓͰVMͰӡ༻ͯ͠·͢ • ΋͘͠͸managed ͳσʔλετΞΛ࠾༻͢Δͷ΋ྑ͍Ͱ͢Ͷ 29

Slide 30

Slide 30 text

Overview 30

Slide 31

Slide 31 text

Overview: ίϯϙʔωϯτ֓ཁ 31

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Overview: ίϯϙʔωϯτ֓ཁ • ReplicaSet: Pod(ίϯςφΠϝʔδ܈)ͷՔಈʹඞཁͳ৘ใΛදݱ͢ΔϦιʔε • Service: ֎෦/಺෦ͱͲͷΑ͏ʹ௨৴͢Δ͔(Endpoint)Λදݱ͢ΔϦιʔε • NodePool : Pod͕Քಇ͢ΔVMΛάϧʔϐϯάԽ͢ΔϦιʔε • Node : Pod͕Քಇ͢ΔVM • Pod : ίϯςφΠϝʔδ܈ • Master : kubernetesͷ༷ʑͳ؅ཧΛதԝूݖతʹ؅ཧ͢ΔVMɻmanagedαʔ ϏεͳͷͰීஈҙࣝ͢Δ͜ͱ͸͋·Γͳ͍ 33

Slide 34

Slide 34 text

Overview: ίϯϙʔωϯτৄࡉ 34

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Overview: ίϯϙʔωϯτৄࡉ • master/api: kubernetesΛૢ࡞͢ΔͨΊͷAPIΛఏڙ͢Δ • master/controller-manager: ίϯςφͷঢ়ଶ؅ཧ΍ϊʔυͷ؅ཧͱݴ֤ͬͨछ ؅ཧ࡞ۀΛߦ͏ • master/scheduler: ֤ϊʔυʹର͠ίϯςφͷׂΓ౰ͯͳͲΛߦ͏ • master/etcd: ϦιʔεͷӬଓԽ૚ɻ෼ࢄKVS • node/proxy: ίϯςφ΁ͷωοτϫʔΫϧʔςΟϯά͓Αͼෛՙ෼ࢄΛߦ͏ • node/kubelet: ֤ϊʔυ্Ͱͷίϯςφ࡞੒/࡟আ΍ϘϦϡʔϜͷׂΓ౰ͯͳͲΛ ߦ͏ 37

Slide 38

Slide 38 text

Overview: ίϯϙʔωϯτৄࡉ • node/kube-dns: ServiceΛఆٛ͢Δ͜ͱͰPodؒͰ௨৴͢ Δࡍͷ໊લΛղܾͯ͘͠Ε·͢ • node/fluentd-cloud-logging: Podͷඪ४ग़ྗΛ StackdriverLoggingʹૹ৴͠·͢ • node/heapstar: Monitoring and Performance Analysis for Container Cluster 38

Slide 39

Slide 39 text

Overview: ίϯϙʔωϯτৄࡉ • ͍Ζ͍Ζॻ͖·͕ͨ͠ɺ௨ৗӡ༻ঢ়ଶʹ͓͍ͯ͸kubectl΍ StackdriverLogging΍StackdriverΛ௨ͯ͡PodΛ؍࡯͠ ͨΓ੍ޚ͢ΔܗͰ͢ • ӡ༻໘Ͱ͸ͦ͜·ͰෳࡶͰ͸ͳ͍Ͱ͢ • node͸୆਺ͷௐ੔Λ͢Δ৔߹΍ΞοϓάϨʔυ࣌ʹ͸ҙࣝ͢Δ • ໰୊͕͋ͬͨ৔߹ɺnodeʹೖͬͯ৘ใऩू͢Δ͜ͱ͸͋Δ 39

Slide 40

Slide 40 text

ىಈ͢Δ·ͰͷྲྀΕ 40

Slide 41

Slide 41 text

Slide 42

Slide 42 text

• masterͱnodeͰ͍Ζ͍Ζ΍ΓऔΓͯ͠·͕͢ɺReadinessʹ ൓Ԡ͠RunningʹભҠ͢Δ·Ͱ͸ඵ୯Ґͷىಈ଎౓Ͱ͢ʢମ ײʣ 42

Slide 43

Slide 43 text

Deployment 43

Slide 44

Slide 44 text

• kubernetes v1.2͔Βར༻ՄೳͰ͕ͨ͠Rc͔ΒDeployment ʹҠߦ͠·ͨ͠ • ઌఔͷReplicaSet͸DeploymentʹΑͬͯࣗಈ؅ཧ͞Ε͍ͯ ·͢ • ͢͜͠લ͸ReplicationController࢖ͬͯ·͕ͨ͠ɺϚΠΫ ϩαʔϏε෦෼͸DeploymentʹҠߦ͠·ͨ͠ 44

Slide 45

Slide 45 text

rc manifest apiVersion: v1 kind: ReplicationController metadata: name: nginx spec: replicas: 3 selector: app: nginx template: metadata: name: nginx labels: app: nginx env: production spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 45

Slide 46

Slide 46 text

deployment manifest apiVersion: apps/v1beta1 # for versions before 1.6.0 use extensions/v1beta1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 3 template: metadata: name: nginx labels: app: nginx env: production spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 46

Slide 47

Slide 47 text

͋·ΓมΘΓ͕ແ͋͘Γ ͕͍ͨʂ 47

Slide 48

Slide 48 text

Slide 49

Slide 49 text

• Rc࣌୅͸reload͸kubectl ͷwrapperπʔϧͰreloadͯ͠ ·ͨ͠ɻͦΕ΋ෆཁʹͳΓ·ͨ͠ • ͜ΕΒૢ࡞͸kubectl͔ΒͰ΋ग़དྷ·͕͢ɺAbemaTVͰ͸ ChatOpsʹΑͬͯdeployͯ͠·͢(΅͘ʹΑΔ࢓ࣄͰ͸ͳ͘ DevͷํʹΑΔ࢓ࣄͰ͢ɻ͋Γ͕ͨ΍) • Ͱ΋ඞཁʹԠͯ͡ɺkubectl΋࢖ͬͨΓ͸͠·͢(ো֐ରԠͳ Ͳ) 49

Slide 50

Slide 50 text

ͱ͋Δ೔ͷdeploy • ͋Δϩοτ୯ҐͰrolling update͞Ε͍ͯΔ༷ࢠ͕؍࡯Ͱ͖Δ 50

Slide 51

Slide 51 text

ϩά 51

Slide 52

Slide 52 text

Slide 53

Slide 53 text

ϩά • GKE͸pod͕ඪ४ग़ྗʹग़ྗͨ݁͠ՌΛfluentd-cloud- loggingΛ௨ͯ͠StackdriverLoggingʹࣗಈసૹͯ͘͠Ε ͯͱͯ΋ϥΫ • ΞΫηεϩάͳͲϢʔβߦಈʹؔ܎͠ผ్ूܭ͍ͨ͠΋ͷ͸ Stream Insertܦ༝ͰBigqueryʹೖΕͯ·͢ • ෳ਺ͷςʔϒϧʹ෼ׂ͠ɺQuotaʹ͞͞Βͳ͍Α͏޻෉͕ඞཁ 53

Slide 54

Slide 54 text

Network 54

Slide 55

Slide 55 text

Slide 56

Slide 56 text

Slide 57

Slide 57 text

Network : Service 21:26:26 [0] % kubectl describe svc my-service Name: my-service Namespace: default Labels: name=my-service Selector: name=my-service Type: NodePort IP: xx.xx.xxx.xxx Port: http 80/TCP NodePort: http 30001/TCP Endpoints: xx.xx.x.x:80,xx.xx.xx.x:80,xx.xx.x.x:80 Session Affinity: None • Port: αʔϏεVIP্ͷԾ૝Port • NodePort: ϗετຖʹଘࡏ͢ΔNodePortͱPodͷContainerPortΛϚοϐϯά͢ΔPort 57

Slide 58

Slide 58 text

Slide 59

Slide 59 text

Slide 60

Slide 60 text

Slide 61

Slide 61 text

Slide 62

Slide 62 text

Network : Service • ServiceΛఆٛ͢Δ͜ͱͰɺPod܈ʹର͢Δ઀ଓΛ؆୯ʹઃఆ Ͱ͖·͢ 62

Slide 63

Slide 63 text

k8s͞ΘͬͯݟΑ͏ͱࢥͬͨํ΁ • Ұ൪खܰͳͷ͸GKEͩͱࢥ͍·͢ • GKEΞΧ΢ϯτͳ͍৔߹͸͜Μͳπʔϧ΋͋Γ·͢ 63

Slide 64

Slide 64 text

Tool: minikube 64

Slide 65

Slide 65 text

Tool: minikube localͷkubernetes؀ڥΛ࡞੒Ͱ͖·͢ % curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.7.1/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/ % minikube start 65

Slide 66

Slide 66 text

Tool: minikube • kubernetesͷίϯϙʔωϯτͰՔಇ͍ͯ͠ΔΘ͚Ͱ͸ͳ͍ͷ Ͱɺຊ֨తͳݕূʹ͸࢖Θͳ͍΄͏͕ྑ͍Ͱ͢ • VagrantͷΑ͏ͳҐஔ෇͚ͩͱࢥ͍·͢ 16:03:16 [0] % kubectl get pod --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default nginx-ydej2 1/1 Running 0 58m kube-system kube-addon-manager-minikubevm 1/1 Running 0 1d kube-system kubernetes-dashboard-xj6g7 1/1 Running 0 1d 66

Slide 67

Slide 67 text

Tool: minikube GKE΍minikubeͳͲෳ਺ͷclusterΛ࢖ͬͯΔ৔߹͸ɺखݩͷ minikube clusterʹҎԼͰ੾Γସ͑Ͱ͖·͢ % kubectl config use-context minikube 67

Slide 68

Slide 68 text

tutorial: GuestBook 68

Slide 69

Slide 69 text

Slide 70

Slide 70 text

tutorial: GuestBook • PHPΞϓϦέʔγϣϯͱRedisΛ࢖ͬͨ؆୯ͳtutorial • GKEͷ৔߹͸্هrepoͷREADME.mdʹैͬͯಈ͔ͯ͠ΈΔͱɺGKE ͷײ৮͕Θ͔ͬͯྑ͍ͱࢥ͍·͢ • minikubeͩͱDeploymentͳͲ৽͍͠Resource͕ಈ͔ͪ͘ΐͬͱ ࣗ৴͕ͳ͍ • umiyosh/kubetutorial : 1೥લ͘Β͍ʹ࡞ͬͨ΍ͭͰ͕͢RC ϕʔεͳͷͰminikubeͰ΋ಈ͘ͱࢥ͍·͢ 70

Slide 71

Slide 71 text

ݟͱ͍ͨ΄͏͕͍͍΍ͭ 71

Slide 72

Slide 72 text

ݟͱ͍ͨ΄͏͕͍͍΍ͭ • ໌Β͔ʹkube࢖͍࢝Ίͨͱ͖ʹཉ͔ͬͨ͠΍ͭ • kube࢖͏৔߹ɺ֤resourceΛΘ͔Γ΍͘͢આ໌ͯ͘͠ΕͯΔͷͰઈରಡΜͩ΄͏͕͍͍Ͱ͢ 72

Slide 73

Slide 73 text

ݟͱ͍ͨ΄͏͕͍͍΍ͭ % kubectl explain pod.spec.nodeSelector FIELD: nodeSelector DESCRIPTION: NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node. More info: http://releases.k8s.io/release-1.2/docs/user-guide/node-selection/README.md • → ؆қͳઆ໌ʴkubernetes.io͔kubernetes/ kubernetes/docʹ༠ಋͯ͘͠ΕΔ 73

Slide 74

Slide 74 text

ݟͱ͍ͨ΄͏͕͍͍΍ͭ • ެࣜυΩϡϝϯτ 74

Slide 75

Slide 75 text

ݟͱ͍ͨ΄͏͕͍͍΍ͭ • ެࣜϦϙδτϦ 75

Slide 76

Slide 76 text

ݟͱ͍ͨ΄͏͕͍͍΍ͭ • gcpug slack ͷ k8s_gke_ja νϟϯωϧ 76

Slide 77

Slide 77 text

ݟͱ͍ͨ΄͏͕͍͍΍ͭ • kubernetes slackνϟωϧ 77

Slide 78

Slide 78 text

• Kubernetes - Production-Grade Container orchestration • kubernetes/kubernetes: Production-Grade Container Scheduling and Management • k8s Slack • gcpug Slack 78

Slide 79

Slide 79 text

·ͱΊ 79

Slide 80

Slide 80 text

·ͱΊ • DockerͷΦʔέετϨʔγϣϯ͸kubernetes͕ͱͯ΋ϥΫͰ ݱ࣮త • KubernetesΛ࢖͏ͳΒGKE͕ϥΫͰݱ࣮త • ϋΠεϧʔϓοτͷαʔϏεͰ΋҆ఆͯ͠࢖͍͑ͯ·͢ 80

Slide 81

Slide 81 text

σʔληϯλʔͷϓϥΠ ϚϦʔΩʔΛαʔό͔Β ΞϓϦέʔγϣϯ΁มԽ ͤ͞·͠ΐ͏ 81

Slide 82

Slide 82 text

͓͠·͍ 82