Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Shakr - Google Container Engine을 이용한 빠르고 안정적인 서...

Shakr - Google Container Engine을 이용한 빠르고 안정적인 서비스 개발 및 배포

Shakr에서 기존 인프라스트럭쳐를 어떻게 Container Engine으로 이전했는지, 새로운 서비스를 빠르게 Container Engine에 배포할 수 있었는지를 소개합니다.

Minku Lee

July 06, 2017
Tweet

More Decks by Minku Lee

Other Decks in Technology

Transcript

  1. v ⿏࠺٣য়ҟҊ Container icon by Dominic Plag from the Noun

    Project ֥ۄ਎੿ب۽рױೠ࠺٣য়ઁ੘ݧ૓࠺٣য়٣੗ੋ੉ೠоٙ
  2. Kubernetes ҳࢿਃࣗ ⿏1PE ⿏%FQMPZNFOU ⿏4FSWJDF ⿏*OHSFTT ⿏%BFNPO4FU ⿏4FDSFU ⿏$POGJH.BQ ⿏4UBUFGVM4FU

    ⿏3FQMJDB4FU ⿏$SPO+PC ⿏+PC ⿏1FSTJTUFOU7PMVNF ⿏1FSTJTUFOU7PMVNF$MBJN ⿏FUDj
  3. Kubernetes ҳࢿਃࣗ ⿏1PE ⿏%FQMPZNFOU ⿏4FSWJDF ⿏*OHSFTT ⿏%BFNPO4FU ⿏4FDSFU ⿏$POGJH.BQ ⿏4UBUFGVM4FU

    ⿏3FQMJDB4FU ⿏$SPO+PC ⿏+PC ⿏1FSTJTUFOU7PMVNF ⿏1FSTJTUFOU7PMVNF$MBJN ⿏FUDj
  4. ӝઓইఃఫ୊ Redis Task Queue Fastly CDN Original Media Files Cloud

    Storage Processed Media Files Cloud Storage Storage User Uploads Cloud Storage Rendered Segments Cloud Storage CDN Interconnect Upload Processing Worker Compute Engine Multiple Instances Video Processing Worker Compute Engine Multiple Instances Image Processing Worker Compute Engine Multiple Instances Video Rendering Worker Compute Engine Multiple Instances Video Design Processing Worker Compute Engine Multiple Instances Compute Papertrail Logging Datadog Monitoring
  5. ௿۞झఠࢤࢿ $ gcloud container clusters create my-awesome-cluster \ --zone asia-northeast1-a

    \ --enable-cloud-logging --enable-cloud-monitoring \ --machine-type n1-standard-4 \ —-num-nodes 3 Bash
  6. $ gcloud container clusters get-credentials \ my-awesome-cluster --zone asia-northeast1-a Fetching

    cluster endpoint and auth data. kubeconfig entry generated for my-awesome-cluster. $ kubectl proxy Starting to serve on 127.0.0.1:8001 Bash Kubernetes Dashboard
  7. yaml deployment.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: myapp-deployment spec:

    replicas: 3 template: metadata: labels: app: myapp spec: containers: - name: myapp image: gcr.io/my-project/myapp:3.0.0 resources: requests: cpu: 200m memory: 100Mi limits: cpu: "4" memory: 8Gi
  8. yaml deployment.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: myapp-deployment spec:

    replicas: 3 template: metadata: labels: app: myapp spec: containers: - name: myapp image: gcr.io/my-project/myapp:3.0.0 resources: requests: cpu: 200m memory: 100Mi limits: cpu: "4" memory: 8Gi livenessProbe: httpGet: path: / port: 80 readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 periodSeconds: 3
  9. yaml deployment.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: myapp-deployment spec:

    replicas: 3 template: metadata: labels: app: myapp spec: containers: - name: myapp image: gcr.io/my-project/myapp:3.0.0 resources: requests: cpu: 200m memory: 100Mi limits: cpu: "4" memory: 8Gi livenessProbe: httpGet: path: / port: 80 readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 periodSeconds: 3 env: - name: RACK_ENV value: production - name: PORT value: 80 volumeMounts: - name: tmp mountPath: /tmp volumes: - name: tmp emptyDir: {}
  10. Deployment ߓನೞӝ Bash # Get credentials $ gcloud container clusters

    get-credentials my-awesome-cluster Fetching cluster endpoint and auth data. kubeconfig entry generated for my-awesome-cluster.
  11. Deployment ߓನೞӝ Bash # Get credentials $ gcloud container clusters

    get-credentials my-awesome-cluster Fetching cluster endpoint and auth data. kubeconfig entry generated for my-awesome-cluster. # Deploy! $ kubectl apply -f path/to/deployment.yml deployment "myapp-deployment" created
  12. yaml deployment.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: myapp-deployment spec:

    replicas: 3 template: metadata: labels: app: myapp spec: containers: - name: myapp image: gcr.io/my-project/myapp:3.0.0 resources: requests: cpu: 200m memory: 100Mi limits: cpu: "4" memory: 8Gi livenessProbe: httpGet: path: / port: 80 readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 periodSeconds: 3 env: - name: RACK_ENV value: production - name: PORT value: 80 volumeMounts: - name: tmp mountPath: /tmp volumes: - name: tmp emptyDir: {}
  13. yaml deployment.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: myapp-deployment spec:

    replicas: 30 template: metadata: labels: app: myapp spec: containers: - name: myapp image: gcr.io/my-project/myapp:3.0.0 resources: requests: cpu: 200m memory: 100Mi limits: cpu: "4" memory: 8Gi livenessProbe: httpGet: path: / port: 80 readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 periodSeconds: 3 env: - name: RACK_ENV value: production - name: PORT value: 80 volumeMounts: - name: tmp mountPath: /tmp volumes: - name: tmp emptyDir: {}
  14. yaml deployment.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: myapp-deployment spec:

    replicas: 30 template: metadata: labels: app: myapp spec: containers: - name: myapp image: gcr.io/my-project/myapp:3.1.0 resources: requests: cpu: 200m memory: 100Mi limits: cpu: "4" memory: 8Gi livenessProbe: httpGet: path: / port: 80 readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 periodSeconds: 3 env: - name: RACK_ENV value: production - name: PORT value: 80 volumeMounts: - name: tmp mountPath: /tmp volumes: - name: tmp emptyDir: {}
  15. Node Pool୶о߂࢏ઁ $ gcloud container node-pools create my-new-nodepool \ --cluster

    my-awesome-cluster \ --machine-type n1-standard-16 \ --num-nodes 5 Bash
  16. Preemptible Node Pool $ gcloud beta container node-pools create my-new-nodepool

    \ --cluster my-awesome-cluster \ --machine-type n1-standard-16 \ --num-nodes 5 \ --preemptible Bash BETA
  17. PVM with nodeAffinity apiVersion: v1 kind: Pod spec: affinity: nodeAffinity:

    preferredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-preemptible operator: DoesNotExist yaml BETA
  18. ௿۞झఠࢸ੿ $ gcloud container clusters create my-awesome-cluster \ --zone asia-northeast1-a

    \ --enable-cloud-logging --enable-cloud-monitoring \ --machine-type n1-standard-4 \ —-num-nodes 3 Bash
  19. Schedule April 2017 2 9 16 23 30 3 10

    17 24 4 11 18 25 5 12 19 26 6 13 20 27 7 14 21 28 8 15 22 29
  20. Schedule April 2017 2 9 16 23 30 3 10

    17 24 4 11 18 25 5 12 19 26 6 13 20 27 7 14 21 28 8 15 22 29 ൞ݎ۠டੌ
  21. Schedule April 2017 2 9 16 23 30 3 10

    17 24 4 11 18 25 5 12 19 26 6 13 20 27 7 14 21 28 8 15 22 29 ೐۽షఋ੉ೝ ೐۽ં౟ࣇস ൞ݎ۠டੌ
  22. Schedule April 2017 2 9 16 23 30 3 10

    17 24 4 11 18 25 5 12 19 26 6 13 20 27 7 14 21 28 8 15 22 29 ೐۽షఋ੉ೝ ೐۽ં౟ࣇস ൞ݎ۠டੌ ߔূ٘೐ۿ౟ূ٘ѐߊ $*ജ҃ࣇস
  23. Schedule April 2017 2 9 16 23 30 3 10

    17 24 4 11 18 25 5 12 19 26 6 13 20 27 7 14 21 28 8 15 22 29 ೐۽షఋ੉ೝ ೐۽ં౟ࣇস ൞ݎ۠டੌ ߔূ٘೐ۿ౟ূ٘ѐߊ $*ജ҃ࣇস ೐۽؋࣌ജ҃ҳ୷
  24. ইఃఫ୊ Container Engine Redis Task Queue Stackdriver Logging Cloud CDN

    Container Builder Docker image builds Frontend k8s Deployment Multiple Pods Backend k8s Deployment Multiple Pods Worker k8s Deployment Multiple Pods Datadog Monitoring Cloud Datastore Frontend k8s Service Backend k8s Service StoriesAds k8s Ingress GitHub Source Repository Travis CI Continuous Integration Cloud Load Balancing
  25. Container Builder Container Engine Redis Task Queue Stackdriver Logging Cloud

    CDN Container Builder Docker image builds Frontend k8s Deployment Multiple Pods Backend k8s Deployment Multiple Pods Worker k8s Deployment Multiple Pods Datadog Monitoring Cloud Datastore Frontend k8s Service Backend k8s Service StoriesAds k8s Ingress GitHub Source Repository Travis CI Continuous Integration Cloud Load Balancing
  26. yaml cloudbuild.yaml steps: - name: 'gcr.io/keynote-demo-project/node:7.9.0' args: ['yarn'] id: 'install-packages'

    - name: 'gcr.io/keynote-demo-project/node:7.9.0' args: ['yarn', 'run', 'build'] env: - 'BACKEND=staging' - 'COMMIT_SHA=$COMMIT_SHA' waitFor: ['install-packages'] id: 'staging' - name: 'gcr.io/keynote-demo-project/node:7.9.0' args: ['yarn', 'run', 'build'] env: - 'BACKEND=production' - 'COMMIT_SHA=$COMMIT_SHA' waitFor: ['install-packages'] id: 'production'
  27. yaml cloudbuild.yaml steps: - name: 'gcr.io/keynote-demo-project/node:7.9.0' args: ['yarn'] id: 'install-packages'

    - name: 'gcr.io/keynote-demo-project/node:7.9.0' args: ['yarn', 'run', 'build'] env: - 'BACKEND=staging' - 'COMMIT_SHA=$COMMIT_SHA' waitFor: ['install-packages'] id: 'staging' - name: 'gcr.io/keynote-demo-project/node:7.9.0' args: ['yarn', 'run', 'build'] env: - 'BACKEND=production' - 'COMMIT_SHA=$COMMIT_SHA' waitFor: ['install-packages'] id: 'production' - name: 'gcr.io/cloud-builders/docker' args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/frontend:staging-$COMMIT_SHA', '--build-arg', 'backend=staging', '.' ] waitFor: ['staging'] - name: 'gcr.io/cloud-builders/docker' args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/frontend:production-$COMMIT_SHA', '--build-arg', 'backend=production', '.' ] waitFor: ['production']
  28. yaml cloudbuild.yaml steps: - name: 'gcr.io/keynote-demo-project/node:7.9.0' args: ['yarn'] id: 'install-packages'

    - name: 'gcr.io/keynote-demo-project/node:7.9.0' args: ['yarn', 'run', 'build'] env: - 'BACKEND=staging' - 'COMMIT_SHA=$COMMIT_SHA' waitFor: ['install-packages'] id: 'staging' - name: 'gcr.io/keynote-demo-project/node:7.9.0' args: ['yarn', 'run', 'build'] env: - 'BACKEND=production' - 'COMMIT_SHA=$COMMIT_SHA' waitFor: ['install-packages'] id: 'production' - name: 'gcr.io/cloud-builders/docker' args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/frontend:staging-$COMMIT_SHA', '--build-arg', 'backend=staging', '.' ] waitFor: ['staging'] - name: 'gcr.io/cloud-builders/docker' args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/frontend:production-$COMMIT_SHA', '--build-arg', 'backend=production', '.' ] waitFor: ['production'] images: [ 'gcr.io/$PROJECT_ID/frontend:staging-$COMMIT_SHA', 'gcr.io/$PROJECT_ID/frontend:production-$COMMIT_SHA' ]
  29. Deployment Container Engine Redis Task Queue Stackdriver Logging Cloud CDN

    Container Builder Docker image builds Frontend k8s Deployment Multiple Pods Backend k8s Deployment Multiple Pods Worker k8s Deployment Multiple Pods Datadog Monitoring Cloud Datastore Frontend k8s Service Backend k8s Service StoriesAds k8s Ingress GitHub Source Repository Travis CI Continuous Integration Cloud Load Balancing
  30. yaml deployment-frontend.yaml apiVersion: apps/v1beta1 kind: Deployment metadata: name: frontend-deployment spec:

    replicas: 2 template: metadata: labels: app: frontend spec: containers: - name: frontend image: gcr.io/project/frontend:staging-v01 resources: requests: memory: 100Mi ports: - containerPort: 80
  31. yaml deployment-api.yaml apiVersion: apps/v1beta1 kind: Deployment metadata: name: api-deployment spec:

    replicas: 3 template: metadata: labels: app: api spec: containers: - name: api image: gcr.io/project/api:v01 resources: requests: memory: 100Mi ports: - containerPort: 80 env: - name: RAILS_ENV value: production - name: RAILS_LOG_TO_STDOUT value: "true" - name: CONFIG_KEY valueFrom: configMapKeyRef: name: my-configs key: CONFIG_KEY - name: SECRET_KEY valueFrom: secretKeyRef: name: my-secrets key: SECRET_KEY
  32. yaml deployment-api.yaml apiVersion: apps/v1beta1 kind: Deployment metadata: name: api-deployment spec:

    replicas: 3 template: metadata: labels: app: api spec: containers: - name: api image: gcr.io/project/api:v01 resources: requests: memory: 100Mi ports: - containerPort: 80 env: - name: RAILS_ENV value: production - name: RAILS_LOG_TO_STDOUT value: "true" - name: CONFIG_KEY valueFrom: configMapKeyRef: name: my-configs key: CONFIG_KEY - name: SECRET_KEY valueFrom: secretKeyRef: name: my-secrets key: SECRET_KEY
  33. secret.yaml apiVersion: v1 kind: Secret metadata: name: my-secrets type: Opaque

    data: SECRET_KEY: <base64_string> config_map.yaml apiVersion: v1 kind: ConfigMap metadata: name: my-configs type: Opaque data: CONFIG_KEY: <base64_string>
  34. Container Engine Redis Task Queue Stackdriver Logging Cloud CDN Container

    Builder Docker image builds Frontend k8s Deployment Multiple Pods Backend k8s Deployment Multiple Pods Worker k8s Deployment Multiple Pods Datadog Monitoring Cloud Datastore Frontend k8s Service Backend k8s Service StoriesAds k8s Ingress GitHub Source Repository Travis CI Continuous Integration Cloud Load Balancing Cloud Datastore
  35. 0 6 / 0 7/ 20 1 7, 1 4

    *4 5 P ag e 1 of 1 https:/ /cloud.google.com/storage-options/
  36. Cloud Datastore ⿏#JH5BCMFীӝ߈ਸك/P42-ؘ੉ఠ߬੉झ ⿏ࢥ٬ TIBSEJOH җࠂઁ SFQMJDBUJPO ਸ੗زਵ۽ҙܻೞৈਬোೠഛ੢оמ ⿏ੋؙझ૑ਗ ੗زࢤࢿ߂ࣻزࢤࢿ

     ⿏ੋझఢझ׼җӘ੉ইצ੷੢۝җয়ಌۨ੉࣌׼җӘ ⿏ޖܐ౭যઁҕ ⿏/JBOUJD 1PL䟖NPO(Pѐߊࢎ ীࢲࢎਊ઺
  37. Service Container Engine Redis Task Queue Stackdriver Logging Cloud CDN

    Container Builder Docker image builds Frontend k8s Deployment Multiple Pods Backend k8s Deployment Multiple Pods Worker k8s Deployment Multiple Pods Datadog Monitoring Cloud Datastore Frontend k8s Service Backend k8s Service StoriesAds k8s Ingress GitHub Source Repository Travis CI Continuous Integration Cloud Load Balancing
  38. Service Discovery $ kubectl exec -ti debug-pod -- bash root@debug-pod:/#

    curl -I http://api-service HTTP/1.1 200 OK X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff Content-Type: text/html X-Request-Id: 7b194765-ixz8-1qke-9572-0fa67509e500 X-Runtime: 0.001207 Bash
  39. service-frontend.yaml apiVersion: v1 kind: Service metadata: name: frontend-service labels: app:

    frontend spec: ports: - port: 80 protocol: TCP name: http selector: app: frontend type: NodePort service-api.yaml apiVersion: v1 kind: Service metadata: name: api-service labels: app: api spec: ports: - port: 80 protocol: TCP name: http selector: app: api type: NodePort
  40. Ingress Container Engine Redis Task Queue Stackdriver Logging Cloud CDN

    Container Builder Docker image builds Frontend k8s Deployment Multiple Pods Backend k8s Deployment Multiple Pods Worker k8s Deployment Multiple Pods Datadog Monitoring Cloud Datastore Frontend k8s Service Backend k8s Service StoriesAds k8s Ingress GitHub Source Repository Travis CI Continuous Integration Cloud Load Balancing
  41. yaml ingress.yml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: storiesads-com-ingress annotations:

    kubernetes.io/ingress.global-static-ip-name: "storiesads-com" spec: tls: - secretName: tls-certificate rules: - http: paths: - path: /api/* backend: serviceName: api-service servicePort: 80 - path: /* backend: serviceName: frontend-service servicePort: 80
  42. Cloud Load Balancing & CDN Container Engine Redis Task Queue

    Stackdriver Logging Cloud CDN Container Builder Docker image builds Frontend k8s Deployment Multiple Pods Backend k8s Deployment Multiple Pods Worker k8s Deployment Multiple Pods Datadog Monitoring Cloud Datastore Frontend k8s Service Backend k8s Service StoriesAds k8s Ingress GitHub Source Repository Travis CI Continuous Integration Cloud Load Balancing
  43. ইఃఫ୊ Container Engine Redis Task Queue Stackdriver Logging Cloud CDN

    Container Builder Docker image builds Frontend k8s Deployment Multiple Pods Backend k8s Deployment Multiple Pods Worker k8s Deployment Multiple Pods Datadog Monitoring Cloud Datastore Frontend k8s Service Backend k8s Service StoriesAds k8s Ingress GitHub Source Repository Travis CI Continuous Integration Cloud Load Balancing