Shakr - Google Container Engine을 이용한 빠르고 안정적인 서비스 개발 및 배포

Shakr - Google Container Engine을 이용한 빠르고 안정적인 서비스 개발 및 배포

Shakr에서 기존 인프라스트럭쳐를 어떻게 Container Engine으로 이전했는지, 새로운 서비스를 빠르게 Container Engine에 배포할 수 있었는지를 소개합니다.

91b2bf7b681403e98e6846677d9ca029?s=128

Minku Lee

July 06, 2017
Tweet

Transcript

  1. 2.

    v ⿏࠺٣য়ҟҊ Container icon by Dominic Plag from the Noun

    Project ֥ۄ਎੿ب۽рױೠ࠺٣য়ઁ੘ݧ૓࠺٣য়٣੗ੋ੉ೠоٙ
  2. 9.
  3. 12.

    Kubernetes ҳࢿਃࣗ ⿏1PE ⿏%FQMPZNFOU ⿏4FSWJDF ⿏*OHSFTT ⿏%BFNPO4FU ⿏4FDSFU ⿏$POGJH.BQ ⿏4UBUFGVM4FU

    ⿏3FQMJDB4FU ⿏$SPO+PC ⿏+PC ⿏1FSTJTUFOU7PMVNF ⿏1FSTJTUFOU7PMVNF$MBJN ⿏FUDj
  4. 13.

    Kubernetes ҳࢿਃࣗ ⿏1PE ⿏%FQMPZNFOU ⿏4FSWJDF ⿏*OHSFTT ⿏%BFNPO4FU ⿏4FDSFU ⿏$POGJH.BQ ⿏4UBUFGVM4FU

    ⿏3FQMJDB4FU ⿏$SPO+PC ⿏+PC ⿏1FSTJTUFOU7PMVNF ⿏1FSTJTUFOU7PMVNF$MBJN ⿏FUDj
  5. 15.

    ӝઓইఃఫ୊ Redis Task Queue Fastly CDN Original Media Files Cloud

    Storage Processed Media Files Cloud Storage Storage User Uploads Cloud Storage Rendered Segments Cloud Storage CDN Interconnect Upload Processing Worker Compute Engine Multiple Instances Video Processing Worker Compute Engine Multiple Instances Image Processing Worker Compute Engine Multiple Instances Video Rendering Worker Compute Engine Multiple Instances Video Design Processing Worker Compute Engine Multiple Instances Compute Papertrail Logging Datadog Monitoring
  6. 22.

    ௿۞झఠࢤࢿ $ gcloud container clusters create my-awesome-cluster \ --zone asia-northeast1-a

    \ --enable-cloud-logging --enable-cloud-monitoring \ --machine-type n1-standard-4 \ —-num-nodes 3 Bash
  7. 23.

    $ gcloud container clusters get-credentials \ my-awesome-cluster --zone asia-northeast1-a Fetching

    cluster endpoint and auth data. kubeconfig entry generated for my-awesome-cluster. $ kubectl proxy Starting to serve on 127.0.0.1:8001 Bash Kubernetes Dashboard
  8. 30.

    yaml deployment.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: myapp-deployment spec:

    replicas: 3 template: metadata: labels: app: myapp spec: containers: - name: myapp image: gcr.io/my-project/myapp:3.0.0 resources: requests: cpu: 200m memory: 100Mi limits: cpu: "4" memory: 8Gi
  9. 31.

    yaml deployment.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: myapp-deployment spec:

    replicas: 3 template: metadata: labels: app: myapp spec: containers: - name: myapp image: gcr.io/my-project/myapp:3.0.0 resources: requests: cpu: 200m memory: 100Mi limits: cpu: "4" memory: 8Gi livenessProbe: httpGet: path: / port: 80 readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 periodSeconds: 3
  10. 32.

    yaml deployment.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: myapp-deployment spec:

    replicas: 3 template: metadata: labels: app: myapp spec: containers: - name: myapp image: gcr.io/my-project/myapp:3.0.0 resources: requests: cpu: 200m memory: 100Mi limits: cpu: "4" memory: 8Gi livenessProbe: httpGet: path: / port: 80 readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 periodSeconds: 3 env: - name: RACK_ENV value: production - name: PORT value: 80 volumeMounts: - name: tmp mountPath: /tmp volumes: - name: tmp emptyDir: {}
  11. 33.
  12. 34.

    Deployment ߓನೞӝ Bash # Get credentials $ gcloud container clusters

    get-credentials my-awesome-cluster Fetching cluster endpoint and auth data. kubeconfig entry generated for my-awesome-cluster.
  13. 35.

    Deployment ߓನೞӝ Bash # Get credentials $ gcloud container clusters

    get-credentials my-awesome-cluster Fetching cluster endpoint and auth data. kubeconfig entry generated for my-awesome-cluster. # Deploy! $ kubectl apply -f path/to/deployment.yml deployment "myapp-deployment" created
  14. 36.

    yaml deployment.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: myapp-deployment spec:

    replicas: 3 template: metadata: labels: app: myapp spec: containers: - name: myapp image: gcr.io/my-project/myapp:3.0.0 resources: requests: cpu: 200m memory: 100Mi limits: cpu: "4" memory: 8Gi livenessProbe: httpGet: path: / port: 80 readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 periodSeconds: 3 env: - name: RACK_ENV value: production - name: PORT value: 80 volumeMounts: - name: tmp mountPath: /tmp volumes: - name: tmp emptyDir: {}
  15. 37.

    yaml deployment.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: myapp-deployment spec:

    replicas: 30 template: metadata: labels: app: myapp spec: containers: - name: myapp image: gcr.io/my-project/myapp:3.0.0 resources: requests: cpu: 200m memory: 100Mi limits: cpu: "4" memory: 8Gi livenessProbe: httpGet: path: / port: 80 readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 periodSeconds: 3 env: - name: RACK_ENV value: production - name: PORT value: 80 volumeMounts: - name: tmp mountPath: /tmp volumes: - name: tmp emptyDir: {}
  16. 38.

    yaml deployment.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: myapp-deployment spec:

    replicas: 30 template: metadata: labels: app: myapp spec: containers: - name: myapp image: gcr.io/my-project/myapp:3.1.0 resources: requests: cpu: 200m memory: 100Mi limits: cpu: "4" memory: 8Gi livenessProbe: httpGet: path: / port: 80 readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 periodSeconds: 3 env: - name: RACK_ENV value: production - name: PORT value: 80 volumeMounts: - name: tmp mountPath: /tmp volumes: - name: tmp emptyDir: {}
  17. 42.

    Node Pool୶о߂࢏ઁ $ gcloud container node-pools create my-new-nodepool \ --cluster

    my-awesome-cluster \ --machine-type n1-standard-16 \ --num-nodes 5 Bash
  18. 46.

    Preemptible Node Pool $ gcloud beta container node-pools create my-new-nodepool

    \ --cluster my-awesome-cluster \ --machine-type n1-standard-16 \ --num-nodes 5 \ --preemptible Bash BETA
  19. 48.

    PVM with nodeAffinity apiVersion: v1 kind: Pod spec: affinity: nodeAffinity:

    preferredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-preemptible operator: DoesNotExist yaml BETA
  20. 50.
  21. 54.

    ௿۞झఠࢸ੿ $ gcloud container clusters create my-awesome-cluster \ --zone asia-northeast1-a

    \ --enable-cloud-logging --enable-cloud-monitoring \ --machine-type n1-standard-4 \ —-num-nodes 3 Bash
  22. 62.

    Schedule April 2017 2 9 16 23 30 3 10

    17 24 4 11 18 25 5 12 19 26 6 13 20 27 7 14 21 28 8 15 22 29
  23. 63.

    Schedule April 2017 2 9 16 23 30 3 10

    17 24 4 11 18 25 5 12 19 26 6 13 20 27 7 14 21 28 8 15 22 29 ൞ݎ۠டੌ
  24. 64.

    Schedule April 2017 2 9 16 23 30 3 10

    17 24 4 11 18 25 5 12 19 26 6 13 20 27 7 14 21 28 8 15 22 29 ೐۽షఋ੉ೝ ೐۽ં౟ࣇস ൞ݎ۠டੌ
  25. 65.

    Schedule April 2017 2 9 16 23 30 3 10

    17 24 4 11 18 25 5 12 19 26 6 13 20 27 7 14 21 28 8 15 22 29 ೐۽షఋ੉ೝ ೐۽ં౟ࣇস ൞ݎ۠டੌ ߔূ٘೐ۿ౟ূ٘ѐߊ $*ജ҃ࣇস
  26. 66.

    Schedule April 2017 2 9 16 23 30 3 10

    17 24 4 11 18 25 5 12 19 26 6 13 20 27 7 14 21 28 8 15 22 29 ೐۽షఋ੉ೝ ೐۽ં౟ࣇস ൞ݎ۠டੌ ߔূ٘೐ۿ౟ূ٘ѐߊ $*ജ҃ࣇস ೐۽؋࣌ജ҃ҳ୷
  27. 69.

    ইఃఫ୊ Container Engine Redis Task Queue Stackdriver Logging Cloud CDN

    Container Builder Docker image builds Frontend k8s Deployment Multiple Pods Backend k8s Deployment Multiple Pods Worker k8s Deployment Multiple Pods Datadog Monitoring Cloud Datastore Frontend k8s Service Backend k8s Service StoriesAds k8s Ingress GitHub Source Repository Travis CI Continuous Integration Cloud Load Balancing
  28. 70.

    Container Builder Container Engine Redis Task Queue Stackdriver Logging Cloud

    CDN Container Builder Docker image builds Frontend k8s Deployment Multiple Pods Backend k8s Deployment Multiple Pods Worker k8s Deployment Multiple Pods Datadog Monitoring Cloud Datastore Frontend k8s Service Backend k8s Service StoriesAds k8s Ingress GitHub Source Repository Travis CI Continuous Integration Cloud Load Balancing
  29. 73.

    yaml cloudbuild.yaml steps: - name: 'gcr.io/keynote-demo-project/node:7.9.0' args: ['yarn'] id: 'install-packages'

    - name: 'gcr.io/keynote-demo-project/node:7.9.0' args: ['yarn', 'run', 'build'] env: - 'BACKEND=staging' - 'COMMIT_SHA=$COMMIT_SHA' waitFor: ['install-packages'] id: 'staging' - name: 'gcr.io/keynote-demo-project/node:7.9.0' args: ['yarn', 'run', 'build'] env: - 'BACKEND=production' - 'COMMIT_SHA=$COMMIT_SHA' waitFor: ['install-packages'] id: 'production'
  30. 74.

    yaml cloudbuild.yaml steps: - name: 'gcr.io/keynote-demo-project/node:7.9.0' args: ['yarn'] id: 'install-packages'

    - name: 'gcr.io/keynote-demo-project/node:7.9.0' args: ['yarn', 'run', 'build'] env: - 'BACKEND=staging' - 'COMMIT_SHA=$COMMIT_SHA' waitFor: ['install-packages'] id: 'staging' - name: 'gcr.io/keynote-demo-project/node:7.9.0' args: ['yarn', 'run', 'build'] env: - 'BACKEND=production' - 'COMMIT_SHA=$COMMIT_SHA' waitFor: ['install-packages'] id: 'production' - name: 'gcr.io/cloud-builders/docker' args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/frontend:staging-$COMMIT_SHA', '--build-arg', 'backend=staging', '.' ] waitFor: ['staging'] - name: 'gcr.io/cloud-builders/docker' args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/frontend:production-$COMMIT_SHA', '--build-arg', 'backend=production', '.' ] waitFor: ['production']
  31. 75.

    yaml cloudbuild.yaml steps: - name: 'gcr.io/keynote-demo-project/node:7.9.0' args: ['yarn'] id: 'install-packages'

    - name: 'gcr.io/keynote-demo-project/node:7.9.0' args: ['yarn', 'run', 'build'] env: - 'BACKEND=staging' - 'COMMIT_SHA=$COMMIT_SHA' waitFor: ['install-packages'] id: 'staging' - name: 'gcr.io/keynote-demo-project/node:7.9.0' args: ['yarn', 'run', 'build'] env: - 'BACKEND=production' - 'COMMIT_SHA=$COMMIT_SHA' waitFor: ['install-packages'] id: 'production' - name: 'gcr.io/cloud-builders/docker' args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/frontend:staging-$COMMIT_SHA', '--build-arg', 'backend=staging', '.' ] waitFor: ['staging'] - name: 'gcr.io/cloud-builders/docker' args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/frontend:production-$COMMIT_SHA', '--build-arg', 'backend=production', '.' ] waitFor: ['production'] images: [ 'gcr.io/$PROJECT_ID/frontend:staging-$COMMIT_SHA', 'gcr.io/$PROJECT_ID/frontend:production-$COMMIT_SHA' ]
  32. 76.

    Deployment Container Engine Redis Task Queue Stackdriver Logging Cloud CDN

    Container Builder Docker image builds Frontend k8s Deployment Multiple Pods Backend k8s Deployment Multiple Pods Worker k8s Deployment Multiple Pods Datadog Monitoring Cloud Datastore Frontend k8s Service Backend k8s Service StoriesAds k8s Ingress GitHub Source Repository Travis CI Continuous Integration Cloud Load Balancing
  33. 77.

    yaml deployment-frontend.yaml apiVersion: apps/v1beta1 kind: Deployment metadata: name: frontend-deployment spec:

    replicas: 2 template: metadata: labels: app: frontend spec: containers: - name: frontend image: gcr.io/project/frontend:staging-v01 resources: requests: memory: 100Mi ports: - containerPort: 80
  34. 78.

    yaml deployment-api.yaml apiVersion: apps/v1beta1 kind: Deployment metadata: name: api-deployment spec:

    replicas: 3 template: metadata: labels: app: api spec: containers: - name: api image: gcr.io/project/api:v01 resources: requests: memory: 100Mi ports: - containerPort: 80 env: - name: RAILS_ENV value: production - name: RAILS_LOG_TO_STDOUT value: "true" - name: CONFIG_KEY valueFrom: configMapKeyRef: name: my-configs key: CONFIG_KEY - name: SECRET_KEY valueFrom: secretKeyRef: name: my-secrets key: SECRET_KEY
  35. 79.

    yaml deployment-api.yaml apiVersion: apps/v1beta1 kind: Deployment metadata: name: api-deployment spec:

    replicas: 3 template: metadata: labels: app: api spec: containers: - name: api image: gcr.io/project/api:v01 resources: requests: memory: 100Mi ports: - containerPort: 80 env: - name: RAILS_ENV value: production - name: RAILS_LOG_TO_STDOUT value: "true" - name: CONFIG_KEY valueFrom: configMapKeyRef: name: my-configs key: CONFIG_KEY - name: SECRET_KEY valueFrom: secretKeyRef: name: my-secrets key: SECRET_KEY
  36. 80.

    secret.yaml apiVersion: v1 kind: Secret metadata: name: my-secrets type: Opaque

    data: SECRET_KEY: <base64_string> config_map.yaml apiVersion: v1 kind: ConfigMap metadata: name: my-configs type: Opaque data: CONFIG_KEY: <base64_string>
  37. 81.

    Container Engine Redis Task Queue Stackdriver Logging Cloud CDN Container

    Builder Docker image builds Frontend k8s Deployment Multiple Pods Backend k8s Deployment Multiple Pods Worker k8s Deployment Multiple Pods Datadog Monitoring Cloud Datastore Frontend k8s Service Backend k8s Service StoriesAds k8s Ingress GitHub Source Repository Travis CI Continuous Integration Cloud Load Balancing Cloud Datastore
  38. 82.

    0 6 / 0 7/ 20 1 7, 1 4

    *4 5 P ag e 1 of 1 https:/ /cloud.google.com/storage-options/
  39. 83.

    Cloud Datastore ⿏#JH5BCMFীӝ߈ਸك/P42-ؘ੉ఠ߬੉झ ⿏ࢥ٬ TIBSEJOH җࠂઁ SFQMJDBUJPO ਸ੗زਵ۽ҙܻೞৈਬোೠഛ੢оמ ⿏ੋؙझ૑ਗ ੗زࢤࢿ߂ࣻزࢤࢿ

     ⿏ੋझఢझ׼җӘ੉ইצ੷੢۝җয়ಌۨ੉࣌׼җӘ ⿏ޖܐ౭যઁҕ ⿏/JBOUJD 1PL䟖NPO(Pѐߊࢎ ীࢲࢎਊ઺
  40. 85.

    Service Container Engine Redis Task Queue Stackdriver Logging Cloud CDN

    Container Builder Docker image builds Frontend k8s Deployment Multiple Pods Backend k8s Deployment Multiple Pods Worker k8s Deployment Multiple Pods Datadog Monitoring Cloud Datastore Frontend k8s Service Backend k8s Service StoriesAds k8s Ingress GitHub Source Repository Travis CI Continuous Integration Cloud Load Balancing
  41. 87.

    Service Discovery $ kubectl exec -ti debug-pod -- bash root@debug-pod:/#

    curl -I http://api-service HTTP/1.1 200 OK X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff Content-Type: text/html X-Request-Id: 7b194765-ixz8-1qke-9572-0fa67509e500 X-Runtime: 0.001207 Bash
  42. 88.

    service-frontend.yaml apiVersion: v1 kind: Service metadata: name: frontend-service labels: app:

    frontend spec: ports: - port: 80 protocol: TCP name: http selector: app: frontend type: NodePort service-api.yaml apiVersion: v1 kind: Service metadata: name: api-service labels: app: api spec: ports: - port: 80 protocol: TCP name: http selector: app: api type: NodePort
  43. 89.

    Ingress Container Engine Redis Task Queue Stackdriver Logging Cloud CDN

    Container Builder Docker image builds Frontend k8s Deployment Multiple Pods Backend k8s Deployment Multiple Pods Worker k8s Deployment Multiple Pods Datadog Monitoring Cloud Datastore Frontend k8s Service Backend k8s Service StoriesAds k8s Ingress GitHub Source Repository Travis CI Continuous Integration Cloud Load Balancing
  44. 91.

    yaml ingress.yml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: storiesads-com-ingress annotations:

    kubernetes.io/ingress.global-static-ip-name: "storiesads-com" spec: tls: - secretName: tls-certificate rules: - http: paths: - path: /api/* backend: serviceName: api-service servicePort: 80 - path: /* backend: serviceName: frontend-service servicePort: 80
  45. 92.

    Cloud Load Balancing & CDN Container Engine Redis Task Queue

    Stackdriver Logging Cloud CDN Container Builder Docker image builds Frontend k8s Deployment Multiple Pods Backend k8s Deployment Multiple Pods Worker k8s Deployment Multiple Pods Datadog Monitoring Cloud Datastore Frontend k8s Service Backend k8s Service StoriesAds k8s Ingress GitHub Source Repository Travis CI Continuous Integration Cloud Load Balancing
  46. 93.

    ইఃఫ୊ Container Engine Redis Task Queue Stackdriver Logging Cloud CDN

    Container Builder Docker image builds Frontend k8s Deployment Multiple Pods Backend k8s Deployment Multiple Pods Worker k8s Deployment Multiple Pods Datadog Monitoring Cloud Datastore Frontend k8s Service Backend k8s Service StoriesAds k8s Ingress GitHub Source Repository Travis CI Continuous Integration Cloud Load Balancing