Upgrade to Pro — share decks privately, control downloads, hide ads and more …

GKEで作る動画サービス

umiyosh
June 28, 2017
1.3k

 GKEで作る動画サービス

市ヶ谷Geek★Night#13 GCPはじめました。
https://ichigayageek.connpass.com/event/57962/
Session1:「GKEで作る動画サービス」

umiyosh

June 28, 2017
Tweet

Transcript

  1. GKEͰ࡞ΔಈըαʔϏε
    1

    View full-size slide

  2. ͸͡Ί·ͯ͠ʂ
    2

    View full-size slide

  3. AbemaTV αʔϏε঺հ
    4

    View full-size slide

  4. ಈըͱ୊͍ͯ͠·͕͢ɺ
    ಈըٕज़ͷ࿩͸ͳ͍Ͱ͢
    10

    View full-size slide

  5. ΄΅GKEͱkubernetes
    ͷ࿩Λͤͯ͞௖͖·͢
    11

    View full-size slide

  6. AbemaTV Overview
    12

    View full-size slide

  7. GKE͸managed
    Kubernetes
    15

    View full-size slide

  8. ࠓ೔ͷ࿩͸GKE্࢖͏
    kubernetesͷ࿩Ͱ͢
    16

    View full-size slide

  9. kubernetes
    17

    View full-size slide

  10. ΫʔόΝωΠςΟε
    19

    View full-size slide

  11. Ͱ΋Googleͷਓ͸Ϋϕ
    ͱ͔Ͱ΋͍͍Αͱݴͬͯ
    ͨͷͰ͓޷͖ͳײ͡Ͱ
    20

    View full-size slide

  12. Կނkubernetes͔ʁ
    21

    View full-size slide

  13. Կނkubernetes͔ʁ
    • ΋ͱ΋ͱ͸Googleࣾ಺ʹ͸Googleݕࡧ΍Gmail΍YouTubeͳ
    ͲίϯϐϡʔλʔϦιʔε͕ڧ྽ʹඞཁͳΦϯϥΠϯαʔϏε
    Λ࠷దʹఏڙ͢ΔͨΊʹίϯςφͰӡ༻͍ͨ͠ͱ͍͏ڧྗͳχ
    ʔζ͕͋ͬͨ
    22

    View full-size slide

  14. Կނkubernetes͔ʁ
    • Google͸ίϯςφϕʔεͰ10೥ӡ༻࣮ͯͨ͠੷͕͋Δ
    • googleͷΠϯϑϥ͸99%ίϯςφͰಈ͔ͯ͠ར༻Ͱ͖ΔΑ͏ʹͨ͠తͳ࿦จ
    • Large-scale cluster management at Google with Borg
    • Borg, Omega, and Kubernetes - ACM Queue
    • Googleࣾ಺ʹఏڙ͢ΔΠϯϑϥ͸ίϯςφϕʔεɻσʔληϯλʔͷϓϥΠϚϦʔΩʔΛα
    ʔό͔ΒΞϓϦέʔγϣϯ΁มԽͤͨ͞
    • borg -> omegaΛ࡞ͬͯͨਓ͕ͨͪΦʔϓϯιʔεͱͯ͠࠶࣮૷ͨ͠΋ͷ͕kubernetes
    • KubernetesΛmanagedͳΫϥ΢υαʔϏεͱͯ͠ఏڙ͍ͯ͠Δͷ͕GKE
    • Kubernetes ͸GoogleΫϥ΢υઓུͷத֩ͱͷ͜ͱ
    23

    View full-size slide

  15. Կނkubernetes͔ʁ
    • kubernetes͸αʔόͷڥքΛҙࣝ͠ͳ͍ΞϓϦέʔγϣϯσϓϩΠΛ
    Մೳʹ͢Δ
    • Data Center as a Computer ͱ͍͏ߟ͕͑ࠜͬ͜ʹ͋Δ
    • ͜ͷߟ͑͸ϚΠΫϩαʔϏεͱ਌࿨ੑ͕ߴ͍
    • ϚΠΫϩαʔϏε୯ҐͰͷϦιʔεͷ૿ڧ/ॖୀ͕ϥΫ
    • Kubernetes1.6Ͱ͸, 5000node, 150000pods·Ͱͷεέʔϧ͕
    Մೳ
    24

    View full-size slide

  16. Կނkubernetes͔ʁ
    • ࣮ࡍKubernetesͷத৺ʹ͋Δߟ͑ํ:Ϣʔβ͕๬Ή΂͖ঢ়ଶΛࢦ
    ఆ͠Kubernetes͕ΫϥελΛͦͷΰʔϧͷঢ়ଶʹҠߦͤ͞Δ͜
    ͱʹ͋Δ
    • GKE͸ෳ਺ͷίϯςφΠϝʔδΛϚωδϝϯτ͢Δʹ͸ɺ͓ͦΒ͘
    ݱஈ֊Ͱ͍ͪ͹ΜखܰͰݎ࿚ͳํ๏
    • ϝϦοτɿσϓϩΠͷ؆୯͞/ਝ଎͞ɺϦιʔε࢖༻ͷޮ཰ੑɺϙ
    ʔλϏϦςΟ
    25

    View full-size slide

  17. Կނkubernetes͔ʁ
    • Opsࢹ఺ͰݟΔͱ͜Ε·Ͱconfiguration(chef, ansible)͠
    ͯ࢖ͬͯͨVM͸managedͳnodeʹͳͬͨͷͰ, configuration
    ͸statelessͳίϯϙʔωϯτʹؔͯ͠͸࢖Θͳ͘ͳͬͨ
    • orchestration(fabric, capistrano)Ͱ΍ͬͯͨࣄ͸
    Docker build΍kubernetesʹ·͔ͤΔͷͰɺpodͷ͋Δ΂͖
    ঢ়ଶΛهड़͢Δ͚ͩͰΑ͘ͳͬͨ
    • bootstrap(terraform)෦෼͚ͩ࢒ͬͨ
    26

    View full-size slide

  18. Կނkubernetes͔ʁ
    • ࢖͍ࣺͯՄೳͳΠϛϡʔλϒϧͳProvisioning͸, ҆৺ײ͕
    ߴ͍
    27

    View full-size slide

  19. Design
    • શͯΛKubernetesͰಈ͔͢ඞཁ͸ͳ͍ɻstateful or stateless ?
    • statelessͳΞϓϦέʔγϣϯ͸ίϯςφͱ૬ੑ͕ྑ͍
    • ͱ͸͍͑statefulsetsΛ࢖ͬͯɺঢ়ଶΛ࣋ͬͨίϯςφΛӡ༻͢Δ͜ͱ΋Ͱ͖
    Δ
    • ઃܭ࣍ୈͩͱࢥ͍·͢
    • AbemaTV͸ঢ়ଶΛ࣋ͭΩϟογϡ΋σʔλϕʔε΋ׂΓ੾ΓͰVMͰӡ༻ͯ͠·͢
    • ΋͘͠͸managed ͳσʔλετΞΛ࠾༻͢Δͷ΋ྑ͍Ͱ͢Ͷ
    29

    View full-size slide

  20. Overview: ίϯϙʔωϯτ֓ཁ
    31

    View full-size slide

  21. Overview: ίϯϙʔωϯτ֓ཁ
    • ReplicaSet: Pod(ίϯςφΠϝʔδ܈)ͷՔಈʹඞཁͳ৘ใΛදݱ͢ΔϦιʔε
    • Service: ֎෦/಺෦ͱͲͷΑ͏ʹ௨৴͢Δ͔(Endpoint)Λදݱ͢ΔϦιʔε
    • NodePool : Pod͕Քಇ͢ΔVMΛάϧʔϐϯάԽ͢ΔϦιʔε
    • Node : Pod͕Քಇ͢ΔVM
    • Pod : ίϯςφΠϝʔδ܈
    • Master : kubernetesͷ༷ʑͳ؅ཧΛதԝूݖతʹ؅ཧ͢ΔVMɻmanagedαʔ
    ϏεͳͷͰීஈҙࣝ͢Δ͜ͱ͸͋·Γͳ͍
    33

    View full-size slide

  22. Overview: ίϯϙʔωϯτৄࡉ
    34

    View full-size slide

  23. Overview: ίϯϙʔωϯτৄࡉ
    • master/api: kubernetesΛૢ࡞͢ΔͨΊͷAPIΛఏڙ͢Δ
    • master/controller-manager: ίϯςφͷঢ়ଶ؅ཧ΍ϊʔυͷ؅ཧͱݴ֤ͬͨछ
    ؅ཧ࡞ۀΛߦ͏
    • master/scheduler: ֤ϊʔυʹର͠ίϯςφͷׂΓ౰ͯͳͲΛߦ͏
    • master/etcd: ϦιʔεͷӬଓԽ૚ɻ෼ࢄKVS
    • node/proxy: ίϯςφ΁ͷωοτϫʔΫϧʔςΟϯά͓Αͼෛՙ෼ࢄΛߦ͏
    • node/kubelet: ֤ϊʔυ্Ͱͷίϯςφ࡞੒/࡟আ΍ϘϦϡʔϜͷׂΓ౰ͯͳͲΛ
    ߦ͏
    37

    View full-size slide

  24. Overview: ίϯϙʔωϯτৄࡉ
    • node/kube-dns: ServiceΛఆٛ͢Δ͜ͱͰPodؒͰ௨৴͢
    Δࡍͷ໊લΛղܾͯ͘͠Ε·͢
    • node/fluentd-cloud-logging: Podͷඪ४ग़ྗΛ
    StackdriverLoggingʹૹ৴͠·͢
    • node/heapstar: Monitoring and Performance
    Analysis for Container Cluster
    38

    View full-size slide

  25. Overview: ίϯϙʔωϯτৄࡉ
    • ͍Ζ͍Ζॻ͖·͕ͨ͠ɺ௨ৗӡ༻ঢ়ଶʹ͓͍ͯ͸kubectl΍
    StackdriverLogging΍StackdriverΛ௨ͯ͡PodΛ؍࡯͠
    ͨΓ੍ޚ͢ΔܗͰ͢
    • ӡ༻໘Ͱ͸ͦ͜·ͰෳࡶͰ͸ͳ͍Ͱ͢
    • node͸୆਺ͷௐ੔Λ͢Δ৔߹΍ΞοϓάϨʔυ࣌ʹ͸ҙࣝ͢Δ
    • ໰୊͕͋ͬͨ৔߹ɺnodeʹೖͬͯ৘ใऩू͢Δ͜ͱ͸͋Δ
    39

    View full-size slide

  26. ىಈ͢Δ·ͰͷྲྀΕ
    40

    View full-size slide

  27. • masterͱnodeͰ͍Ζ͍Ζ΍ΓऔΓͯ͠·͕͢ɺReadinessʹ
    ൓Ԡ͠RunningʹભҠ͢Δ·Ͱ͸ඵ୯Ґͷىಈ଎౓Ͱ͢ʢମ
    ײʣ
    42

    View full-size slide

  28. Deployment
    43

    View full-size slide

  29. • kubernetes v1.2͔Βར༻ՄೳͰ͕ͨ͠Rc͔ΒDeployment
    ʹҠߦ͠·ͨ͠
    • ઌఔͷReplicaSet͸DeploymentʹΑͬͯࣗಈ؅ཧ͞Ε͍ͯ
    ·͢
    • ͢͜͠લ͸ReplicationController࢖ͬͯ·͕ͨ͠ɺϚΠΫ
    ϩαʔϏε෦෼͸DeploymentʹҠߦ͠·ͨ͠
    44

    View full-size slide

  30. rc manifest
    apiVersion: v1
    kind: ReplicationController
    metadata:
    name: nginx
    spec:
    replicas: 3
    selector:
    app: nginx
    template:
    metadata:
    name: nginx
    labels:
    app: nginx
    env: production
    spec:
    containers:
    - name: nginx
    image: nginx:1.7.9
    ports:
    - containerPort: 80
    45

    View full-size slide

  31. deployment manifest
    apiVersion: apps/v1beta1 # for versions before 1.6.0 use extensions/v1beta1
    kind: Deployment
    metadata:
    name: nginx-deployment
    spec:
    replicas: 3
    template:
    metadata:
    name: nginx
    labels:
    app: nginx
    env: production
    spec:
    containers:
    - name: nginx
    image: nginx:1.7.9
    ports:
    - containerPort: 80
    46

    View full-size slide

  32. ͋·ΓมΘΓ͕ແ͋͘Γ
    ͕͍ͨʂ
    47

    View full-size slide

  33. • Rc࣌୅͸reload͸kubectl ͷwrapperπʔϧͰreloadͯ͠
    ·ͨ͠ɻͦΕ΋ෆཁʹͳΓ·ͨ͠
    • ͜ΕΒૢ࡞͸kubectl͔ΒͰ΋ग़དྷ·͕͢ɺAbemaTVͰ͸
    ChatOpsʹΑͬͯdeployͯ͠·͢(΅͘ʹΑΔ࢓ࣄͰ͸ͳ͘
    DevͷํʹΑΔ࢓ࣄͰ͢ɻ͋Γ͕ͨ΍)
    • Ͱ΋ඞཁʹԠͯ͡ɺkubectl΋࢖ͬͨΓ͸͠·͢(ো֐ରԠͳ
    Ͳ)
    49

    View full-size slide

  34. ͱ͋Δ೔ͷdeploy
    • ͋Δϩοτ୯ҐͰrolling update͞Ε͍ͯΔ༷ࢠ͕؍࡯Ͱ͖Δ
    50

    View full-size slide

  35. ϩά
    • GKE͸pod͕ඪ४ग़ྗʹग़ྗͨ݁͠ՌΛfluentd-cloud-
    loggingΛ௨ͯ͠StackdriverLoggingʹࣗಈసૹͯ͘͠Ε
    ͯͱͯ΋ϥΫ
    • ΞΫηεϩάͳͲϢʔβߦಈʹؔ܎͠ผ్ूܭ͍ͨ͠΋ͷ͸
    Stream Insertܦ༝ͰBigqueryʹೖΕͯ·͢
    • ෳ਺ͷςʔϒϧʹ෼ׂ͠ɺQuotaʹ͞͞Βͳ͍Α͏޻෉͕ඞཁ
    53

    View full-size slide

  36. Network : Service
    21:26:26 [0] % kubectl describe svc my-service
    Name: my-service
    Namespace: default
    Labels: name=my-service
    Selector: name=my-service
    Type: NodePort
    IP: xx.xx.xxx.xxx
    Port: http 80/TCP
    NodePort: http 30001/TCP
    Endpoints: xx.xx.x.x:80,xx.xx.xx.x:80,xx.xx.x.x:80
    Session Affinity: None
    • Port: αʔϏεVIP্ͷԾ૝Port
    • NodePort: ϗετຖʹଘࡏ͢ΔNodePortͱPodͷContainerPortΛϚοϐϯά͢ΔPort
    57

    View full-size slide

  37. Network : Service
    • ServiceΛఆٛ͢Δ͜ͱͰɺPod܈ʹର͢Δ઀ଓΛ؆୯ʹઃఆ
    Ͱ͖·͢
    62

    View full-size slide

  38. k8s͞ΘͬͯݟΑ͏ͱࢥͬͨํ΁
    • Ұ൪खܰͳͷ͸GKEͩͱࢥ͍·͢
    • GKEΞΧ΢ϯτͳ͍৔߹͸͜Μͳπʔϧ΋͋Γ·͢
    63

    View full-size slide

  39. Tool: minikube
    64

    View full-size slide

  40. Tool: minikube
    localͷkubernetes؀ڥΛ࡞੒Ͱ͖·͢
    % curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.7.1/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
    % minikube start
    65

    View full-size slide

  41. Tool: minikube
    • kubernetesͷίϯϙʔωϯτͰՔಇ͍ͯ͠ΔΘ͚Ͱ͸ͳ͍ͷ
    Ͱɺຊ֨తͳݕূʹ͸࢖Θͳ͍΄͏͕ྑ͍Ͱ͢
    • VagrantͷΑ͏ͳҐஔ෇͚ͩͱࢥ͍·͢
    16:03:16 [0] % kubectl get pod --all-namespaces
    NAMESPACE NAME READY STATUS RESTARTS AGE
    default nginx-ydej2 1/1 Running 0 58m
    kube-system kube-addon-manager-minikubevm 1/1 Running 0 1d
    kube-system kubernetes-dashboard-xj6g7 1/1 Running 0 1d
    66

    View full-size slide

  42. Tool: minikube
    GKE΍minikubeͳͲෳ਺ͷclusterΛ࢖ͬͯΔ৔߹͸ɺखݩͷ
    minikube clusterʹҎԼͰ੾Γସ͑Ͱ͖·͢
    % kubectl config use-context minikube
    67

    View full-size slide

  43. tutorial:
    GuestBook
    68

    View full-size slide

  44. tutorial: GuestBook
    • PHPΞϓϦέʔγϣϯͱRedisΛ࢖ͬͨ؆୯ͳtutorial
    • GKEͷ৔߹͸্هrepoͷREADME.mdʹैͬͯಈ͔ͯ͠ΈΔͱɺGKE
    ͷײ৮͕Θ͔ͬͯྑ͍ͱࢥ͍·͢
    • minikubeͩͱDeploymentͳͲ৽͍͠Resource͕ಈ͔ͪ͘ΐͬͱ
    ࣗ৴͕ͳ͍
    • umiyosh/kubetutorial : 1೥લ͘Β͍ʹ࡞ͬͨ΍ͭͰ͕͢RC
    ϕʔεͳͷͰminikubeͰ΋ಈ͘ͱࢥ͍·͢
    70

    View full-size slide

  45. ݟͱ͍ͨ΄͏͕͍͍΍ͭ
    71

    View full-size slide

  46. ݟͱ͍ͨ΄͏͕͍͍΍ͭ
    • ໌Β͔ʹkube࢖͍࢝Ίͨͱ͖ʹཉ͔ͬͨ͠΍ͭ
    • kube࢖͏৔߹ɺ֤resourceΛΘ͔Γ΍͘͢આ໌ͯ͘͠ΕͯΔͷͰઈରಡΜͩ΄͏͕͍͍Ͱ͢
    72

    View full-size slide

  47. ݟͱ͍ͨ΄͏͕͍͍΍ͭ
    % kubectl explain pod.spec.nodeSelector
    FIELD: nodeSelector
    DESCRIPTION:
    NodeSelector is a selector which must be true for the pod to fit on a node.
    Selector which must match a node's labels for the pod to be scheduled on
    that node. More info:
    http://releases.k8s.io/release-1.2/docs/user-guide/node-selection/README.md
    • → ؆қͳઆ໌ʴkubernetes.io͔kubernetes/
    kubernetes/docʹ༠ಋͯ͘͠ΕΔ
    73

    View full-size slide

  48. ݟͱ͍ͨ΄͏͕͍͍΍ͭ
    • ެࣜυΩϡϝϯτ
    74

    View full-size slide

  49. ݟͱ͍ͨ΄͏͕͍͍΍ͭ
    • ެࣜϦϙδτϦ
    75

    View full-size slide

  50. ݟͱ͍ͨ΄͏͕͍͍΍ͭ
    • gcpug slack ͷ k8s_gke_ja νϟϯωϧ
    76

    View full-size slide

  51. ݟͱ͍ͨ΄͏͕͍͍΍ͭ
    • kubernetes slackνϟωϧ
    77

    View full-size slide

  52. • Kubernetes - Production-Grade Container
    orchestration
    • kubernetes/kubernetes: Production-Grade
    Container Scheduling and Management
    • k8s Slack
    • gcpug Slack
    78

    View full-size slide

  53. ·ͱΊ
    • DockerͷΦʔέετϨʔγϣϯ͸kubernetes͕ͱͯ΋ϥΫͰ
    ݱ࣮త
    • KubernetesΛ࢖͏ͳΒGKE͕ϥΫͰݱ࣮త
    • ϋΠεϧʔϓοτͷαʔϏεͰ΋҆ఆͯ͠࢖͍͑ͯ·͢
    80

    View full-size slide

  54. σʔληϯλʔͷϓϥΠ
    ϚϦʔΩʔΛαʔό͔Β
    ΞϓϦέʔγϣϯ΁มԽ
    ͤ͞·͠ΐ͏
    81

    View full-size slide