GCP+GKE Deep Dive Part 2: Advanced Cluster Management

GCP+GKE Deep Dive Part 2: Advanced Cluster Management

Part 1 바로가기: https://speakerdeck.com/premist/gcp-plus-gke-deep-dive-part-1-initial-app-development/

처음 GKE를 사용하기 시작하면 난해할 수 있는 서비스 배포, GKE 클러스터를 생성하고 여러 GCP의 서비스를 이용하여 첫 애플리케이션을 배포하는 과정까지 자세하게 살펴봅니다. 또한 애플리케이션을 배포한 이후 안정적인 서비스 운영을 위해 활용할 수 있는 클러스터 관리 테크닉을 소개합니다.

Part 2: Advanced Cluster Management
Kubernetes에는 분산 시스템을 구축하고 관리하는 것을 도와주는 다양한 기능이 있지만, 워낙 많은 사용 사례에 대비하다보니 어떤 기능이 있는지를 쉽게 간과하고 넘어가는 경우가 많습니다.

두 번째 파트에서는 클러스터와 애플리케이션을 보다 효율적으로 관리할 수 있도록 도와주는 여러 가지 리소스와 기능을 소개합니다. Google Cloud 서비스와 연동되어 Service Account 프로비저닝을 GKE 내에서 관리할 수 있게 도와주는 Service Broker, 한 번 실행되거나 시간대에 맞춰 자동으로 실행되는 CronJob, 애플리케이션의 SLO(Service Level Objective)를 유지하기 위한 Affinity 및 Pod Disruption Budget에 대해 알아봅니다.

필요 이해도: Kubernetes에 애플리케이션을 배포해 보고 관리해 보신 경험이 있는 분에게 적합합니다. Part 1을 듣고 이어서 들으셔도 무방합니다.

91b2bf7b681403e98e6846677d9ca029?s=128

Minku Lee

June 29, 2018
Tweet

Transcript

  1. PART 2 14:00~ Advanced Cluster Management

  2. 킪핟믾헒 • Kubernetes펞샎믾쫆헏핆힎킫핂핖몮 
 팮읺핂켦픒짾쫆몋픒헒헪옪삖삲 • Google Kubernetes Engine (GKE)

    믾훎픊옪
 컲졓삖삲
  3. CronJob

  4. None
  5. 쭒칾킪큲펞컪팖헣헏픊옪폖퍋핟펓픒쿦쿦핖픒밚 

  6. None
  7. apiVersion: batch/v1beta1 kind: CronJob metadata: name: recurring-job spec: schedule: "*/1

    * * * *" jobTemplate: spec: template: spec: containers: - name: recurringwork image: recurringwork:latest args: - ./do-recurring.sh restartPolicy: OnFailure cronjob.yml
  8. apiVersion: batch/v1beta1 kind: CronJob metadata: name: recurring-job spec: schedule: "*/1

    * * * *" concurrencyPolicy: Replace jobTemplate: spec: template: spec: containers: - name: recurringwork image: recurringwork:latest args: - ./do-recurring.sh restartPolicy: OnFailure cronjob.yml 핟펓킪핟픒킪맒핂쇦펖쁢섾핂헒핟펓핂퐒헒븫빦힎팘팦삲졂 Allow 솧킪킲픒푷 Forbid 솧킪킲픒믖힎 Replace 믾홂핟펓픒홓욚몮킲
  9. $ kubectl apply -f cronjob.yml kubectl옪CronJob캫컿

  10. CronJob • Cron syntax읊믆샎옪칺푷펺얺큲픦쭒칾핟펓픒묺솧 쿦핖삲 • ⚠ 콚쩖킲(At Least Once)픒쫂핳믾쌚줆펞펺

    빦펺얺쩖킲쇮캏펞샎샎찒많푢
 (de-duplication옪힏슿)
  11. Affinity

  12. None
  13. None
  14. None
  15. "핊헣혾멂픦NodePool펞잚
 Pod핂큲흂쇦솒옫쿦펔픒밚?"

  16. apiVersion: apps/v1 kind: Deployment metadata: name: gitlab labels: app: gitlab

    spec: replicas: 1 selector: matchLabels: app: gitlab template: metadata: labels: apps: gitlab deployment.yml spec: nodeSelector: cloud.google.com/gke-preemptible: "true" containers: - name: gitlab image: gitlab/gitlab-ce:latest resources: requests: cpu: "0.5" memory: 1Gi env: - name: GITLAB_OMNIBUS_CONFIG value: ...
  17. "핊헣혾멂픦NodePool펞
 Pod핂큲흂쇦쁢멆컮힎잚 쭖많몋푾펞쁢
 삲읆NodePool펞큲흂쇦솒옫쿦펔픒밚?"

  18. Affinity • 쫂삲퓮펾몮삲퍟혾멂슲옪Pod핂펂쎉멚큲흂쇮힎
 헣픦쿦핖픚 • 펺얺찒묞펾칾핞칺푷많쁳
 In, NotIn, Exists, DoesNotExist,

    Gt(>), Lt(<) • Node믾훎픎Pod믾훎픊옪힎헣많쁳 BETA
  19. Examples

  20. "빦쁢GPU많핖쁢Node펞줂혾멂큲흂쇦펂퍊"

  21. spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-accelerator

    operator: In values: - nvidia-tesla-p100 - nvidia-tesla-v100 pod-gpuonly.yml
  22. "빦쁢GPU많핖쁢Node펞큲흂쇦쁢멆컮힎잚
 GPU많핖쁢Node많펔펂솒펂싢펢많쁢큲흂쇦펂퍊"

  23. spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-accelerator

    operator: In values: - nvidia-tesla-p100 - nvidia-tesla-v100 pod-gpu-preferred.yml
  24. "빦쁢GPU많핖쁢Node펞줂혾멂큲흂쇦펂퍊몮 
 pool-a NodePool펞큲흂쇦졂홙멮힎잚
 팖쇦펂솒멚캏뫎펔펂"

  25. spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-accelerator

    operator: In values: - nvidia-tesla-p100 - nvidia-tesla-v100 preferredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-nodepool operator: In values: - pool-a pod-gpu-and-nodepool.yml
  26. "빦쁢app=gitlab핆섾 
 쇦솒옫빦퐎맧픎app펞콚콛쇪Pod뫊쁢삲읆
 Availability Zone펞큲흂쇦몮탄펂

  27. spec: affinity: podAffinity: preferredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app

    operator: In values: - gitlab topologyKey: failure-domain.beta.kubernetes.io/zone different-zone-preferred.yml
  28. Affinity • 삶핊얺큲옪삲퍟풚옪슪읊힎풞쌚솒풎핂잜핂쇦쁢믾쁳 • ⚠ requiredDuringSchedulingIgnoredDuringExecution픒 잜핂칺푷컪큲흂쇮쿦펔쁢Pod핂
 잜팒힎쁢멑픒핦뫎읺퍊 BETA

  29. Pod Disruption Budgets

  30. None
  31. node-1 web api job node-2 web api job node-3 web

    api job
  32. web api job node-2 web api job node-3 web api

    job
  33. web api job node-2 web api job node-3 web api

    job #
  34. "짆켦읺팮읺핂켦픦훟삶펔핂
 Node읊펔팮몮탄픎섾펂쎉멚힎 

  35. Pod Disruption Budget • 혾멂펞재쁢Pod핂캏헣힒맽쿦옪퓮힎쇮멑픒
 맣헪쿦핖쁢믾쁳 • minAvailable짝maxUnavailable 퐃켦픊옪컲헣 •

    ⚠ 핞짪헏핆disruption픦몋푾펞잚PDB 훎쿦많쇦즎옪 
 폖믾팘픎/PEF픦핳팮짪캫킪펞쁢PDB많힎힎힎팘픒쿦핖픚 BETA
  36. apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: gitlab-pdb spec: minAvailable: 2

    selector: matchLabels: app: gitlab pod-disruption-budget.yml
  37. Pod Disruption Budget • 혾멂펞재쁢Pod핂캏헣힒맽쿦옪퓮힎쇮멑픒
 맣헪쿦핖쁢믾쁳 • minAvailable짝maxUnavailable 퐃켦픊옪컲헣 •

    ⚠ 핞짪헏핆disruption픦몋푾펞잚PDB 훎쿦많쇦즎옪 
 폖믾팘픎/PEF픦핳팮짪캫킪펞쁢PDB많힎힎힎팘픒쿦핖픚 BETA
  38. Node Draining $ kubectl cordon NODE $ kubectl drain NODE

    훊펂힒Node읊큲흂핂쇦힎팘솒옫힎헣삲 훊펂힒Node읊큲흂핂쇦힎팘솒옫힎헣몮 
 맏Pod읊칻헪삲
  39. $ kubectl cordon gke-my-cluster-my-pool-592cda94-2w25 node "gke-my-cluster-my-pool-592cda94-2w25" cordoned $ kubectl describe

    gke-my-cluster-my-pool-592cda94-2w25 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal NodeNotSchedulable 45s kubelet, ... Node status is now: NodeNotSchedulable kubectl cordon
  40. Google Kubernetes Engine픎Node쪎몋킪drain픒팚팒컪읺삲

  41. 핞솧Node펓믆엖핂슪빦Autoscale 칺푷킪Pod Disruption Budget픎쿦

  42. Service Catalog and Broker

  43. Pod Secret serviceaccount.json: ... Cloud SQL Instance Kubernetes(GKE) 짤펞컪 쿦솧픊옪캫컿퍊

  44. $ gcloud iam service-accounts create gitlab \ --display-name="GitLab Service Account"

    Service Account캫컿
  45. $ gcloud projects add-iam-policy-binding $PROJECT \ --member="serviceAccount:$EMAIL"\ --role="roles/cloudsql.client" 뭚쭎펺

  46. $ gcloud iam service-accounts keys create \ ./artifacts/serviceaccount.json \ --iam-account

    $EMAIL 슫
  47. $ kubectl create secret generic gitlab-config \ ... --from-file=./artifacts/serviceaccount.json kubectl픒핂푷Secret캫컿

    $
  48. "Kubernetes뺂펞컪
 푆쭎컪찒큲펞샎컲헣픒쿦쁢펔픒밚?"

  49. Service CatalogBETA

  50. None
  51. %

  52. Pod Secret serviceaccount.json: ... Cloud SQL Instance Kubernetes(GKE) 짤펞컪 쿦솧픊옪캫컿퍊

  53. Cloud SQL Instance Service
 Account Service Instance Service Instance

  54. 핺힎풞쇦쁢Service Instance홓윦 Service Account Cloud Spanner Cloud Pub/Sub Cloud SQL

    (MySQL) BigQuery Cloud BigTable Cloud Storage
  55. $ kubectl create clusterrolebinding cluster-admin-binding \ --clusterrole=cluster-admin \ --user=$(gcloud config

    get-value account) clusterrolebinding "cluster-admin-binding" created Service Catalog컲 https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog
  56. None
  57. $ sc install account: account@example.com project: shakr-openinfra-demo zone: generated service

    catalog deployment config in dir: /tmp/service- catalog544428136 Service Catalog installed successfully. Service Catalog컲 https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog
  58. $ sc add-gcp-broker using project: shakr-openinfra-demo enabling a GCP API:

    servicebroker.googleapis.com enabling a GCP API: bigtableadmin.googleapis.com enabling a GCP API: ml.googleapis.com ... The Service Broker has been added successfully. Service Catalog컲 https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog
  59. $ gcloud projects add-iam-policy-binding $PROJECT_ID \ --member serviceAccount:$EMAIL \ --role=roles/owner

    Service Catalog컲 https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog
  60. $ kubectl -o "custom- columns=NAME:.spec.externalName,DESCRIPTION:.spec.description" \ get clusterserviceclasses NAME DESCRIPTION

    cloud-spanner The first horizontally scalable... cloud-iam-service-account Specialized service which provisions... cloud-pubsub Ingest event streams from anywhere... cloud-sql-mysql A fully-managed MySQL database service bigquery A fast, highly scalable, cost-effective cloud-bigtable A high performance NoSQL database Service Catalog핆
  61. apiVersion: servicecatalog.k8s.io/v1beta1 kind: ServiceInstance metadata: name: test-storage namespace: default spec:

    clusterServiceClassExternalName: cloud-storage clusterServicePlanExternalName: beta parameters: bucketId: shakr-openinfra-demo-test-storage location: US storageClass: STANDARD serviceinstance.yml
  62. $ kubectl apply -f serviceinstance.yml kubectl옪Service Instance캫컿

  63. None
  64. &&&&

  65. apiVersion: servicecatalog.k8s.io/v1beta1 kind: ServiceInstance metadata: name: test-storage namespace: default spec:

    clusterServiceClassExternalName: cloud-storage clusterServicePlanExternalName: beta parameters: bucketId: shakr-openinfra-demo-test-storage location: US storageClass: STANDARD serviceinstance.yml
  66. https://twitter.com/tenderlove/status/988887936128040960

  67. 콢픒칺푷펺Service Instance캫컿

  68. 콢픒칺푷펺Service Instance캫컿

  69. $ svcat provision test-storage \ --class cloud-storage \ --plan beta

    \ --namespace default \ --param bucketId=shakr-openinfra-demo-test-storage \ --param location=US \ --param storageClass=STANDARD svcat픊옪Service Instance캫컿
  70. Service Binding

  71. Cloud Storage Instance Service Instance Service Instance Service Binding Service

    Binding Secret privateKeyData: ... Service Account
  72. Cloud Storage Instance Service Instance Service Binding Secret privateKeyData: ...

    createServiceAccount: true
  73. $ svcat bind test-storage \ --name test-storage-binding \ --params-json \

    '{ "serviceAccount": "test-storage-serviceaccount", "createServiceAccount": true, "roles": [ "roles/storage.objectCreator", "roles/storage.objectViewer" ] }' Service Binding캫컿
  74. $ kubectl get secrets test-storage-binding NAME TYPE DATA AGE test-storage-binding

    Opaque 2 5m Service Account Secret핆
  75. spec: volumes: - name: test-storage-binding secret: secretName: test-storage-binding containers: -

    name: my-app image: shakr/my-app:latest volumeMounts: - name: binding mountPath: /mnt/binding env: - name: GOOGLE_APPLICATION_CREDENTIALS value: /mnt/binding/privateKeyData - name: STORAGE_PROJECT valueFrom: secretKeyRef: name: user-storage-binding key: projectId - name: STORAGE_BUCKET valueFrom: secretKeyRef: name: user-storage-binding key: bucketId deployment.yml (pod spec)
  76. Service Catalog TL;DR • GKE(Kubernetes) 얺큲펞컪 GCP픦컪찒큲읊Service Instance옪 캫컿펺짢옪칺푷쿦핖삲 •

    Service Account JSON Key픦뽆픒
 먿헣힎팘팒솒쇪삲 • ⚠ Service Instance 칻헪킪킲헪읺콚큲 GCS쩒 SQL핆큲큲 솒
 칻헪쇦삖훊픦
  77. https://svc-cat.io

  78. Recap

  79. 쭒칾킪큲펞컪팖헣헏픊옪폖퍋핟펓픒쿦쿦핖픒밚 

  80. CronJob • Cron syntax읊믆샎옪칺푷펺얺큲픦쭒칾핟펓픒묺솧쿦핖삲 • ⚠ 콚쩖킲(At Least Once)픒쫂핳믾쌚줆펞펺빦펺얺쩖킲 쇮캏펞샎샎찒많푢(de-duplication옪힏슿)

  81. "빦쁢GPU많핖쁢Node펞줂혾멂큲흂쇦펂퍊몮 
 pool-a NodePool펞큲흂쇦졂홙멮힎잚
 팖쇦펂솒멚캏뫎펔펂"

  82. Affinity • 쫂삲퓮펾몮삲퍟혾멂슲옪Pod핂펂쎉멚큲흂쇮힎헣픦쿦핖픚 • 펺얺찒묞펾칾핞칺푷많쁳
 In, NotIn, Exists, DoesNotExist, Gt(>),

    Lt(<) • Node믾훎픎Pod믾훎픊옪힎헣많쁳 BETA
  83. "짆켦읺팮읺핂켦픦훟삶펔핂
 Node읊펔팮몮탄픎섾펂쎉멚힎 

  84. Pod Disruption Budget • 혾멂펞재쁢Pod핂캏헣힒맽쿦옪퓮힎쇮멑픒
 맣헪쿦핖쁢믾쁳 • minAvailable짝maxUnavailable 퐃켦픊옪컲헣 •

    ⚠ 핞짪헏핆disruption픦몋푾펞잚PDB 훎쿦많쇦즎옪 
 폖믾팘픎/PEF픦핳팮짪캫킪펞쁢PDB많힎힎힎팘픒쿦핖픚 BETA
  85. "Kubernetes뺂펞컪
 푆쭎컪찒큲펞샎컲헣픒쿦쁢펔픒밚?"

  86. Service Catalog • GKE(Kubernetes) 얺큲펞컪 GCP픦컪찒큲읊Service Instance옪 캫컿펺짢옪칺푷쿦핖삲 • Service

    Account JSON Key픦뽆픒
 먿헣힎팘팒솒쇪삲 • ⚠ Service Instance 칻헪킪킲헪읺콚큲 GCS쩒 SQL핆큲큲 솒
 칻헪쇦삖훊픦
  87. CronJob Affinity Pod Disruption Budget Service Catalog

  88. https://twitter.com/_tr/status/1007619178222665730

  89. 맞칺삖삲 '

  90. GCP+GKE Deep Dive Minku Lee CTO Shakr Shakr펞컪쁳엳핖쁢펢힎삖펂읊졶킻삖삲 careers.shakr.com