Upgrade to Pro — share decks privately, control downloads, hide ads and more …

GKEで作る動画サービス

umiyosh
June 28, 2017
1.3k

 GKEで作る動画サービス

市ヶ谷Geek★Night#13 GCPはじめました。
https://ichigayageek.connpass.com/event/57962/
Session1:「GKEで作る動画サービス」

umiyosh

June 28, 2017
Tweet

Transcript

  1. GKEͰ࡞ΔಈըαʔϏε
    1

    View Slide

  2. ͸͡Ί·ͯ͠ʂ
    2

    View Slide

  3. 3

    View Slide

  4. AbemaTV αʔϏε঺հ
    4

    View Slide

  5. 5

    View Slide

  6. 6

    View Slide

  7. 7

    View Slide

  8. 8

    View Slide

  9. 9

    View Slide

  10. ಈըͱ୊͍ͯ͠·͕͢ɺ
    ಈըٕज़ͷ࿩͸ͳ͍Ͱ͢
    10

    View Slide

  11. ΄΅GKEͱkubernetes
    ͷ࿩Λͤͯ͞௖͖·͢
    11

    View Slide

  12. AbemaTV Overview
    12

    View Slide

  13. 13

    View Slide

  14. GKE
    14

    View Slide

  15. GKE͸managed
    Kubernetes
    15

    View Slide

  16. ࠓ೔ͷ࿩͸GKE্࢖͏
    kubernetesͷ࿩Ͱ͢
    16

    View Slide

  17. kubernetes
    17

    View Slide

  18. ൃԻ
    18

    View Slide

  19. ΫʔόΝωΠςΟε
    19

    View Slide

  20. Ͱ΋Googleͷਓ͸Ϋϕ
    ͱ͔Ͱ΋͍͍Αͱݴͬͯ
    ͨͷͰ͓޷͖ͳײ͡Ͱ
    20

    View Slide

  21. Կނkubernetes͔ʁ
    21

    View Slide

  22. Կނkubernetes͔ʁ
    • ΋ͱ΋ͱ͸Googleࣾ಺ʹ͸Googleݕࡧ΍Gmail΍YouTubeͳ
    ͲίϯϐϡʔλʔϦιʔε͕ڧ྽ʹඞཁͳΦϯϥΠϯαʔϏε
    Λ࠷దʹఏڙ͢ΔͨΊʹίϯςφͰӡ༻͍ͨ͠ͱ͍͏ڧྗͳχ
    ʔζ͕͋ͬͨ
    22

    View Slide

  23. Կނkubernetes͔ʁ
    • Google͸ίϯςφϕʔεͰ10೥ӡ༻࣮ͯͨ͠੷͕͋Δ
    • googleͷΠϯϑϥ͸99%ίϯςφͰಈ͔ͯ͠ར༻Ͱ͖ΔΑ͏ʹͨ͠తͳ࿦จ
    • Large-scale cluster management at Google with Borg
    • Borg, Omega, and Kubernetes - ACM Queue
    • Googleࣾ಺ʹఏڙ͢ΔΠϯϑϥ͸ίϯςφϕʔεɻσʔληϯλʔͷϓϥΠϚϦʔΩʔΛα
    ʔό͔ΒΞϓϦέʔγϣϯ΁มԽͤͨ͞
    • borg -> omegaΛ࡞ͬͯͨਓ͕ͨͪΦʔϓϯιʔεͱͯ͠࠶࣮૷ͨ͠΋ͷ͕kubernetes
    • KubernetesΛmanagedͳΫϥ΢υαʔϏεͱͯ͠ఏڙ͍ͯ͠Δͷ͕GKE
    • Kubernetes ͸GoogleΫϥ΢υઓུͷத֩ͱͷ͜ͱ
    23

    View Slide

  24. Կނkubernetes͔ʁ
    • kubernetes͸αʔόͷڥքΛҙࣝ͠ͳ͍ΞϓϦέʔγϣϯσϓϩΠΛ
    Մೳʹ͢Δ
    • Data Center as a Computer ͱ͍͏ߟ͕͑ࠜͬ͜ʹ͋Δ
    • ͜ͷߟ͑͸ϚΠΫϩαʔϏεͱ਌࿨ੑ͕ߴ͍
    • ϚΠΫϩαʔϏε୯ҐͰͷϦιʔεͷ૿ڧ/ॖୀ͕ϥΫ
    • Kubernetes1.6Ͱ͸, 5000node, 150000pods·Ͱͷεέʔϧ͕
    Մೳ
    24

    View Slide

  25. Կނkubernetes͔ʁ
    • ࣮ࡍKubernetesͷத৺ʹ͋Δߟ͑ํ:Ϣʔβ͕๬Ή΂͖ঢ়ଶΛࢦ
    ఆ͠Kubernetes͕ΫϥελΛͦͷΰʔϧͷঢ়ଶʹҠߦͤ͞Δ͜
    ͱʹ͋Δ
    • GKE͸ෳ਺ͷίϯςφΠϝʔδΛϚωδϝϯτ͢Δʹ͸ɺ͓ͦΒ͘
    ݱஈ֊Ͱ͍ͪ͹ΜखܰͰݎ࿚ͳํ๏
    • ϝϦοτɿσϓϩΠͷ؆୯͞/ਝ଎͞ɺϦιʔε࢖༻ͷޮ཰ੑɺϙ
    ʔλϏϦςΟ
    25

    View Slide

  26. Կނkubernetes͔ʁ
    • Opsࢹ఺ͰݟΔͱ͜Ε·Ͱconfiguration(chef, ansible)͠
    ͯ࢖ͬͯͨVM͸managedͳnodeʹͳͬͨͷͰ, configuration
    ͸statelessͳίϯϙʔωϯτʹؔͯ͠͸࢖Θͳ͘ͳͬͨ
    • orchestration(fabric, capistrano)Ͱ΍ͬͯͨࣄ͸
    Docker build΍kubernetesʹ·͔ͤΔͷͰɺpodͷ͋Δ΂͖
    ঢ়ଶΛهड़͢Δ͚ͩͰΑ͘ͳͬͨ
    • bootstrap(terraform)෦෼͚ͩ࢒ͬͨ
    26

    View Slide

  27. Կނkubernetes͔ʁ
    • ࢖͍ࣺͯՄೳͳΠϛϡʔλϒϧͳProvisioning͸, ҆৺ײ͕
    ߴ͍
    27

    View Slide

  28. Design
    28

    View Slide

  29. Design
    • શͯΛKubernetesͰಈ͔͢ඞཁ͸ͳ͍ɻstateful or stateless ?
    • statelessͳΞϓϦέʔγϣϯ͸ίϯςφͱ૬ੑ͕ྑ͍
    • ͱ͸͍͑statefulsetsΛ࢖ͬͯɺঢ়ଶΛ࣋ͬͨίϯςφΛӡ༻͢Δ͜ͱ΋Ͱ͖
    Δ
    • ઃܭ࣍ୈͩͱࢥ͍·͢
    • AbemaTV͸ঢ়ଶΛ࣋ͭΩϟογϡ΋σʔλϕʔε΋ׂΓ੾ΓͰVMͰӡ༻ͯ͠·͢
    • ΋͘͠͸managed ͳσʔλετΞΛ࠾༻͢Δͷ΋ྑ͍Ͱ͢Ͷ
    29

    View Slide

  30. Overview
    30

    View Slide

  31. Overview: ίϯϙʔωϯτ֓ཁ
    31

    View Slide

  32. 32

    View Slide

  33. Overview: ίϯϙʔωϯτ֓ཁ
    • ReplicaSet: Pod(ίϯςφΠϝʔδ܈)ͷՔಈʹඞཁͳ৘ใΛදݱ͢ΔϦιʔε
    • Service: ֎෦/಺෦ͱͲͷΑ͏ʹ௨৴͢Δ͔(Endpoint)Λදݱ͢ΔϦιʔε
    • NodePool : Pod͕Քಇ͢ΔVMΛάϧʔϐϯάԽ͢ΔϦιʔε
    • Node : Pod͕Քಇ͢ΔVM
    • Pod : ίϯςφΠϝʔδ܈
    • Master : kubernetesͷ༷ʑͳ؅ཧΛதԝूݖతʹ؅ཧ͢ΔVMɻmanagedαʔ
    ϏεͳͷͰීஈҙࣝ͢Δ͜ͱ͸͋·Γͳ͍
    33

    View Slide

  34. Overview: ίϯϙʔωϯτৄࡉ
    34

    View Slide

  35. 35

    View Slide

  36. 36

    View Slide

  37. Overview: ίϯϙʔωϯτৄࡉ
    • master/api: kubernetesΛૢ࡞͢ΔͨΊͷAPIΛఏڙ͢Δ
    • master/controller-manager: ίϯςφͷঢ়ଶ؅ཧ΍ϊʔυͷ؅ཧͱݴ֤ͬͨछ
    ؅ཧ࡞ۀΛߦ͏
    • master/scheduler: ֤ϊʔυʹର͠ίϯςφͷׂΓ౰ͯͳͲΛߦ͏
    • master/etcd: ϦιʔεͷӬଓԽ૚ɻ෼ࢄKVS
    • node/proxy: ίϯςφ΁ͷωοτϫʔΫϧʔςΟϯά͓Αͼෛՙ෼ࢄΛߦ͏
    • node/kubelet: ֤ϊʔυ্Ͱͷίϯςφ࡞੒/࡟আ΍ϘϦϡʔϜͷׂΓ౰ͯͳͲΛ
    ߦ͏
    37

    View Slide

  38. Overview: ίϯϙʔωϯτৄࡉ
    • node/kube-dns: ServiceΛఆٛ͢Δ͜ͱͰPodؒͰ௨৴͢
    Δࡍͷ໊લΛղܾͯ͘͠Ε·͢
    • node/fluentd-cloud-logging: Podͷඪ४ग़ྗΛ
    StackdriverLoggingʹૹ৴͠·͢
    • node/heapstar: Monitoring and Performance
    Analysis for Container Cluster
    38

    View Slide

  39. Overview: ίϯϙʔωϯτৄࡉ
    • ͍Ζ͍Ζॻ͖·͕ͨ͠ɺ௨ৗӡ༻ঢ়ଶʹ͓͍ͯ͸kubectl΍
    StackdriverLogging΍StackdriverΛ௨ͯ͡PodΛ؍࡯͠
    ͨΓ੍ޚ͢ΔܗͰ͢
    • ӡ༻໘Ͱ͸ͦ͜·ͰෳࡶͰ͸ͳ͍Ͱ͢
    • node͸୆਺ͷௐ੔Λ͢Δ৔߹΍ΞοϓάϨʔυ࣌ʹ͸ҙࣝ͢Δ
    • ໰୊͕͋ͬͨ৔߹ɺnodeʹೖͬͯ৘ใऩू͢Δ͜ͱ͸͋Δ
    39

    View Slide

  40. ىಈ͢Δ·ͰͷྲྀΕ
    40

    View Slide

  41. 41

    View Slide

  42. • masterͱnodeͰ͍Ζ͍Ζ΍ΓऔΓͯ͠·͕͢ɺReadinessʹ
    ൓Ԡ͠RunningʹભҠ͢Δ·Ͱ͸ඵ୯Ґͷىಈ଎౓Ͱ͢ʢମ
    ײʣ
    42

    View Slide

  43. Deployment
    43

    View Slide

  44. • kubernetes v1.2͔Βར༻ՄೳͰ͕ͨ͠Rc͔ΒDeployment
    ʹҠߦ͠·ͨ͠
    • ઌఔͷReplicaSet͸DeploymentʹΑͬͯࣗಈ؅ཧ͞Ε͍ͯ
    ·͢
    • ͢͜͠લ͸ReplicationController࢖ͬͯ·͕ͨ͠ɺϚΠΫ
    ϩαʔϏε෦෼͸DeploymentʹҠߦ͠·ͨ͠
    44

    View Slide

  45. rc manifest
    apiVersion: v1
    kind: ReplicationController
    metadata:
    name: nginx
    spec:
    replicas: 3
    selector:
    app: nginx
    template:
    metadata:
    name: nginx
    labels:
    app: nginx
    env: production
    spec:
    containers:
    - name: nginx
    image: nginx:1.7.9
    ports:
    - containerPort: 80
    45

    View Slide

  46. deployment manifest
    apiVersion: apps/v1beta1 # for versions before 1.6.0 use extensions/v1beta1
    kind: Deployment
    metadata:
    name: nginx-deployment
    spec:
    replicas: 3
    template:
    metadata:
    name: nginx
    labels:
    app: nginx
    env: production
    spec:
    containers:
    - name: nginx
    image: nginx:1.7.9
    ports:
    - containerPort: 80
    46

    View Slide

  47. ͋·ΓมΘΓ͕ແ͋͘Γ
    ͕͍ͨʂ
    47

    View Slide

  48. 48

    View Slide

  49. • Rc࣌୅͸reload͸kubectl ͷwrapperπʔϧͰreloadͯ͠
    ·ͨ͠ɻͦΕ΋ෆཁʹͳΓ·ͨ͠
    • ͜ΕΒૢ࡞͸kubectl͔ΒͰ΋ग़དྷ·͕͢ɺAbemaTVͰ͸
    ChatOpsʹΑͬͯdeployͯ͠·͢(΅͘ʹΑΔ࢓ࣄͰ͸ͳ͘
    DevͷํʹΑΔ࢓ࣄͰ͢ɻ͋Γ͕ͨ΍)
    • Ͱ΋ඞཁʹԠͯ͡ɺkubectl΋࢖ͬͨΓ͸͠·͢(ো֐ରԠͳ
    Ͳ)
    49

    View Slide

  50. ͱ͋Δ೔ͷdeploy
    • ͋Δϩοτ୯ҐͰrolling update͞Ε͍ͯΔ༷ࢠ͕؍࡯Ͱ͖Δ
    50

    View Slide

  51. ϩά
    51

    View Slide

  52. 52

    View Slide

  53. ϩά
    • GKE͸pod͕ඪ४ग़ྗʹग़ྗͨ݁͠ՌΛfluentd-cloud-
    loggingΛ௨ͯ͠StackdriverLoggingʹࣗಈసૹͯ͘͠Ε
    ͯͱͯ΋ϥΫ
    • ΞΫηεϩάͳͲϢʔβߦಈʹؔ܎͠ผ్ूܭ͍ͨ͠΋ͷ͸
    Stream Insertܦ༝ͰBigqueryʹೖΕͯ·͢
    • ෳ਺ͷςʔϒϧʹ෼ׂ͠ɺQuotaʹ͞͞Βͳ͍Α͏޻෉͕ඞཁ
    53

    View Slide

  54. Network
    54

    View Slide

  55. 55

    View Slide

  56. 56

    View Slide

  57. Network : Service
    21:26:26 [0] % kubectl describe svc my-service
    Name: my-service
    Namespace: default
    Labels: name=my-service
    Selector: name=my-service
    Type: NodePort
    IP: xx.xx.xxx.xxx
    Port: http 80/TCP
    NodePort: http 30001/TCP
    Endpoints: xx.xx.x.x:80,xx.xx.xx.x:80,xx.xx.x.x:80
    Session Affinity: None
    • Port: αʔϏεVIP্ͷԾ૝Port
    • NodePort: ϗετຖʹଘࡏ͢ΔNodePortͱPodͷContainerPortΛϚοϐϯά͢ΔPort
    57

    View Slide

  58. 58

    View Slide

  59. 59

    View Slide

  60. 60

    View Slide

  61. 61

    View Slide

  62. Network : Service
    • ServiceΛఆٛ͢Δ͜ͱͰɺPod܈ʹର͢Δ઀ଓΛ؆୯ʹઃఆ
    Ͱ͖·͢
    62

    View Slide

  63. k8s͞ΘͬͯݟΑ͏ͱࢥͬͨํ΁
    • Ұ൪खܰͳͷ͸GKEͩͱࢥ͍·͢
    • GKEΞΧ΢ϯτͳ͍৔߹͸͜Μͳπʔϧ΋͋Γ·͢
    63

    View Slide

  64. Tool: minikube
    64

    View Slide

  65. Tool: minikube
    localͷkubernetes؀ڥΛ࡞੒Ͱ͖·͢
    % curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.7.1/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
    % minikube start
    65

    View Slide

  66. Tool: minikube
    • kubernetesͷίϯϙʔωϯτͰՔಇ͍ͯ͠ΔΘ͚Ͱ͸ͳ͍ͷ
    Ͱɺຊ֨తͳݕূʹ͸࢖Θͳ͍΄͏͕ྑ͍Ͱ͢
    • VagrantͷΑ͏ͳҐஔ෇͚ͩͱࢥ͍·͢
    16:03:16 [0] % kubectl get pod --all-namespaces
    NAMESPACE NAME READY STATUS RESTARTS AGE
    default nginx-ydej2 1/1 Running 0 58m
    kube-system kube-addon-manager-minikubevm 1/1 Running 0 1d
    kube-system kubernetes-dashboard-xj6g7 1/1 Running 0 1d
    66

    View Slide

  67. Tool: minikube
    GKE΍minikubeͳͲෳ਺ͷclusterΛ࢖ͬͯΔ৔߹͸ɺखݩͷ
    minikube clusterʹҎԼͰ੾Γସ͑Ͱ͖·͢
    % kubectl config use-context minikube
    67

    View Slide

  68. tutorial:
    GuestBook
    68

    View Slide

  69. 69

    View Slide

  70. tutorial: GuestBook
    • PHPΞϓϦέʔγϣϯͱRedisΛ࢖ͬͨ؆୯ͳtutorial
    • GKEͷ৔߹͸্هrepoͷREADME.mdʹैͬͯಈ͔ͯ͠ΈΔͱɺGKE
    ͷײ৮͕Θ͔ͬͯྑ͍ͱࢥ͍·͢
    • minikubeͩͱDeploymentͳͲ৽͍͠Resource͕ಈ͔ͪ͘ΐͬͱ
    ࣗ৴͕ͳ͍
    • umiyosh/kubetutorial : 1೥લ͘Β͍ʹ࡞ͬͨ΍ͭͰ͕͢RC
    ϕʔεͳͷͰminikubeͰ΋ಈ͘ͱࢥ͍·͢
    70

    View Slide

  71. ݟͱ͍ͨ΄͏͕͍͍΍ͭ
    71

    View Slide

  72. ݟͱ͍ͨ΄͏͕͍͍΍ͭ
    • ໌Β͔ʹkube࢖͍࢝Ίͨͱ͖ʹཉ͔ͬͨ͠΍ͭ
    • kube࢖͏৔߹ɺ֤resourceΛΘ͔Γ΍͘͢આ໌ͯ͘͠ΕͯΔͷͰઈରಡΜͩ΄͏͕͍͍Ͱ͢
    72

    View Slide

  73. ݟͱ͍ͨ΄͏͕͍͍΍ͭ
    % kubectl explain pod.spec.nodeSelector
    FIELD: nodeSelector
    DESCRIPTION:
    NodeSelector is a selector which must be true for the pod to fit on a node.
    Selector which must match a node's labels for the pod to be scheduled on
    that node. More info:
    http://releases.k8s.io/release-1.2/docs/user-guide/node-selection/README.md
    • → ؆қͳઆ໌ʴkubernetes.io͔kubernetes/
    kubernetes/docʹ༠ಋͯ͘͠ΕΔ
    73

    View Slide

  74. ݟͱ͍ͨ΄͏͕͍͍΍ͭ
    • ެࣜυΩϡϝϯτ
    74

    View Slide

  75. ݟͱ͍ͨ΄͏͕͍͍΍ͭ
    • ެࣜϦϙδτϦ
    75

    View Slide

  76. ݟͱ͍ͨ΄͏͕͍͍΍ͭ
    • gcpug slack ͷ k8s_gke_ja νϟϯωϧ
    76

    View Slide

  77. ݟͱ͍ͨ΄͏͕͍͍΍ͭ
    • kubernetes slackνϟωϧ
    77

    View Slide

  78. • Kubernetes - Production-Grade Container
    orchestration
    • kubernetes/kubernetes: Production-Grade
    Container Scheduling and Management
    • k8s Slack
    • gcpug Slack
    78

    View Slide

  79. ·ͱΊ
    79

    View Slide

  80. ·ͱΊ
    • DockerͷΦʔέετϨʔγϣϯ͸kubernetes͕ͱͯ΋ϥΫͰ
    ݱ࣮త
    • KubernetesΛ࢖͏ͳΒGKE͕ϥΫͰݱ࣮త
    • ϋΠεϧʔϓοτͷαʔϏεͰ΋҆ఆͯ͠࢖͍͑ͯ·͢
    80

    View Slide

  81. σʔληϯλʔͷϓϥΠ
    ϚϦʔΩʔΛαʔό͔Β
    ΞϓϦέʔγϣϯ΁มԽ
    ͤ͞·͠ΐ͏
    81

    View Slide

  82. ͓͠·͍
    82

    View Slide