GCP+GKE Deep Dive Part 2: Advanced Cluster Management

PART 2 14:00~ Advanced Cluster Management

킪핟믾헒 • Kubernetes펞샎믾쫆헏핆힎킫핂핖몮   팮읺핂켦픒짾쫆몋픒헒헪옪삖삲 • Google Kubernetes Engine (GKE)
믾훎픊옪  컲졓삖삲

CronJob

쭒칾킪큲펞컪팖헣헏픊옪폖퍋핟펓픒쿦쿦핖픒밚

apiVersion: batch/v1beta1 kind: CronJob metadata: name: recurring-job spec: schedule: "*/1
* * * *" jobTemplate: spec: template: spec: containers: - name: recurringwork image: recurringwork:latest args: - ./do-recurring.sh restartPolicy: OnFailure cronjob.yml

apiVersion: batch/v1beta1 kind: CronJob metadata: name: recurring-job spec: schedule: "*/1
* * * *" concurrencyPolicy: Replace jobTemplate: spec: template: spec: containers: - name: recurringwork image: recurringwork:latest args: - ./do-recurring.sh restartPolicy: OnFailure cronjob.yml 핟펓킪핟픒킪맒핂쇦펖쁢섾핂헒핟펓핂퐒헒븫빦힎팘팦삲졂 Allow 솧킪킲픒푷 Forbid 솧킪킲픒믖힎 Replace 믾홂핟펓픒홓욚몮킲

$ kubectl apply -f cronjob.yml kubectl옪CronJob캫컿

CronJob • Cron syntax읊믆샎옪칺푷펺얺큲픦쭒칾핟펓픒묺솧 쿦핖삲 • ⚠ 콚쩖킲(At Least Once)픒쫂핳믾쌚줆펞펺
빦펺얺쩖킲쇮캏펞샎샎찒많푢  (de-duplication옪힏슿)

Afﬁnity

"핊헣혾멂픦NodePool펞잚  Pod핂큲흂쇦솒옫쿦펔픒밚?"

apiVersion: apps/v1 kind: Deployment metadata: name: gitlab labels: app: gitlab
spec: replicas: 1 selector: matchLabels: app: gitlab template: metadata: labels: apps: gitlab deployment.yml spec: nodeSelector: cloud.google.com/gke-preemptible: "true" containers: - name: gitlab image: gitlab/gitlab-ce:latest resources: requests: cpu: "0.5" memory: 1Gi env: - name: GITLAB_OMNIBUS_CONFIG value: ...

"핊헣혾멂픦NodePool펞  Pod핂큲흂쇦쁢멆컮힎잚 쭖많몋푾펞쁢  삲읆NodePool펞큲흂쇦솒옫쿦펔픒밚?"

Afﬁnity • 쫂삲퓮펾몮삲퍟혾멂슲옪Pod핂펂쎉멚큲흂쇮힎  헣픦쿦핖픚 • 펺얺찒묞펾칾핞칺푷많쁳  In, NotIn, Exists, DoesNotExist,
Gt(>), Lt(<) • Node믾훎픎Pod믾훎픊옪힎헣많쁳 BETA

Examples

"빦쁢GPU많핖쁢Node펞줂혾멂큲흂쇦펂퍊"

spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-accelerator
operator: In values: - nvidia-tesla-p100 - nvidia-tesla-v100 pod-gpuonly.yml

"빦쁢GPU많핖쁢Node펞큲흂쇦쁢멆컮힎잚  GPU많핖쁢Node많펔펂솒펂싢펢많쁢큲흂쇦펂퍊"

spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-accelerator
operator: In values: - nvidia-tesla-p100 - nvidia-tesla-v100 pod-gpu-preferred.yml

"빦쁢GPU많핖쁢Node펞줂혾멂큲흂쇦펂퍊몮   pool-a NodePool펞큲흂쇦졂홙멮힎잚  팖쇦펂솒멚캏뫎펔펂"

spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-accelerator
operator: In values: - nvidia-tesla-p100 - nvidia-tesla-v100 preferredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-nodepool operator: In values: - pool-a pod-gpu-and-nodepool.yml

"빦쁢app=gitlab핆섾   쇦솒옫빦퐎맧픎app펞콚콛쇪Pod뫊쁢삲읆  Availability Zone펞큲흂쇦몮탄펂

spec: affinity: podAffinity: preferredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app
operator: In values: - gitlab topologyKey: failure-domain.beta.kubernetes.io/zone different-zone-preferred.yml

Afﬁnity • 삶핊얺큲옪삲퍟풚옪슪읊힎풞쌚솒풎핂잜핂쇦쁢믾쁳 • ⚠ requiredDuringSchedulingIgnoredDuringExecution픒 잜핂칺푷컪큲흂쇮쿦펔쁢Pod핂  잜팒힎쁢멑픒핦뫎읺퍊 BETA

Pod Disruption Budgets

node-1 web api job node-2 web api job node-3 web
api job

web api job node-2 web api job node-3 web api
job

web api job node-2 web api job node-3 web api
job #

"짆켦읺팮읺핂켦픦훟삶펔핂  Node읊펔팮몮탄픎섾펂쎉멚힎

Pod Disruption Budget • 혾멂펞재쁢Pod핂캏헣힒맽쿦옪퓮힎쇮멑픒  맣헪쿦핖쁢믾쁳 • minAvailable짝maxUnavailable 퐃켦픊옪컲헣 •
⚠ 핞짪헏핆disruption픦몋푾펞잚PDB 훎쿦많쇦즎옪   폖믾팘픎/PEF픦핳팮짪캫킪펞쁢PDB많힎힎힎팘픒쿦핖픚 BETA

apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: gitlab-pdb spec: minAvailable: 2
selector: matchLabels: app: gitlab pod-disruption-budget.yml

Node Draining $ kubectl cordon NODE $ kubectl drain NODE
훊펂힒Node읊큲흂핂쇦힎팘솒옫힎헣삲 훊펂힒Node읊큲흂핂쇦힎팘솒옫힎헣몮   맏Pod읊칻헪삲

$ kubectl cordon gke-my-cluster-my-pool-592cda94-2w25 node "gke-my-cluster-my-pool-592cda94-2w25" cordoned $ kubectl describe
gke-my-cluster-my-pool-592cda94-2w25 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal NodeNotSchedulable 45s kubelet, ... Node status is now: NodeNotSchedulable kubectl cordon

Google Kubernetes Engine픎Node쪎몋킪drain픒팚팒컪읺삲

핞솧Node펓믆엖핂슪빦Autoscale 칺푷킪Pod Disruption Budget픎쿦

Service Catalog and Broker

Pod Secret serviceaccount.json: ... Cloud SQL Instance Kubernetes(GKE) 짤펞컪 쿦솧픊옪캫컿퍊

$ gcloud iam service-accounts create gitlab \ --display-name="GitLab Service Account"
Service Account캫컿

$ gcloud projects add-iam-policy-binding $PROJECT \ --member="serviceAccount:$EMAIL"\ --role="roles/cloudsql.client" 뭚쭎펺

$ gcloud iam service-accounts keys create \ ./artifacts/serviceaccount.json \ --iam-account
$EMAIL 슫

$ kubectl create secret generic gitlab-config \ ... --from-file=./artifacts/serviceaccount.json kubectl픒핂푷Secret캫컿
$

"Kubernetes뺂펞컪  푆쭎컪찒큲펞샎컲헣픒쿦쁢펔픒밚?"

Service CatalogBETA

Pod Secret serviceaccount.json: ... Cloud SQL Instance Kubernetes(GKE) 짤펞컪 쿦솧픊옪캫컿퍊

Cloud SQL Instance Service  Account Service Instance Service Instance

핺힎풞쇦쁢Service Instance홓윦 Service Account Cloud Spanner Cloud Pub/Sub Cloud SQL
(MySQL) BigQuery Cloud BigTable Cloud Storage

$ kubectl create clusterrolebinding cluster-admin-binding \ --clusterrole=cluster-admin \ --user=$(gcloud config
get-value account) clusterrolebinding "cluster-admin-binding" created Service Catalog컲 https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog

$ sc install account: [email protected] project: shakr-openinfra-demo zone: generated service
catalog deployment config in dir: /tmp/service- catalog544428136 Service Catalog installed successfully. Service Catalog컲 https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog

$ sc add-gcp-broker using project: shakr-openinfra-demo enabling a GCP API:
servicebroker.googleapis.com enabling a GCP API: bigtableadmin.googleapis.com enabling a GCP API: ml.googleapis.com ... The Service Broker has been added successfully. Service Catalog컲 https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog

$ gcloud projects add-iam-policy-binding $PROJECT_ID \ --member serviceAccount:$EMAIL \ --role=roles/owner
Service Catalog컲 https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog

$ kubectl -o "custom- columns=NAME:.spec.externalName,DESCRIPTION:.spec.description" \ get clusterserviceclasses NAME DESCRIPTION
cloud-spanner The first horizontally scalable... cloud-iam-service-account Specialized service which provisions... cloud-pubsub Ingest event streams from anywhere... cloud-sql-mysql A fully-managed MySQL database service bigquery A fast, highly scalable, cost-effective cloud-bigtable A high performance NoSQL database Service Catalog핆

apiVersion: servicecatalog.k8s.io/v1beta1 kind: ServiceInstance metadata: name: test-storage namespace: default spec:
clusterServiceClassExternalName: cloud-storage clusterServicePlanExternalName: beta parameters: bucketId: shakr-openinfra-demo-test-storage location: US storageClass: STANDARD serviceinstance.yml

$ kubectl apply -f serviceinstance.yml kubectl옪Service Instance캫컿

apiVersion: servicecatalog.k8s.io/v1beta1 kind: ServiceInstance metadata: name: test-storage namespace: default spec:
clusterServiceClassExternalName: cloud-storage clusterServicePlanExternalName: beta parameters: bucketId: shakr-openinfra-demo-test-storage location: US storageClass: STANDARD serviceinstance.yml

https://twitter.com/tenderlove/status/988887936128040960

콢픒칺푷펺Service Instance캫컿

$ svcat provision test-storage \ --class cloud-storage \ --plan beta
\ --namespace default \ --param bucketId=shakr-openinfra-demo-test-storage \ --param location=US \ --param storageClass=STANDARD svcat픊옪Service Instance캫컿

Service Binding

Cloud Storage Instance Service Instance Service Instance Service Binding Service
Binding Secret privateKeyData: ... Service Account

Cloud Storage Instance Service Instance Service Binding Secret privateKeyData: ...
createServiceAccount: true

$ svcat bind test-storage \ --name test-storage-binding \ --params-json \
'{ "serviceAccount": "test-storage-serviceaccount", "createServiceAccount": true, "roles": [ "roles/storage.objectCreator", "roles/storage.objectViewer" ] }' Service Binding캫컿

$ kubectl get secrets test-storage-binding NAME TYPE DATA AGE test-storage-binding
Opaque 2 5m Service Account Secret핆

spec: volumes: - name: test-storage-binding secret: secretName: test-storage-binding containers: -
name: my-app image: shakr/my-app:latest volumeMounts: - name: binding mountPath: /mnt/binding env: - name: GOOGLE_APPLICATION_CREDENTIALS value: /mnt/binding/privateKeyData - name: STORAGE_PROJECT valueFrom: secretKeyRef: name: user-storage-binding key: projectId - name: STORAGE_BUCKET valueFrom: secretKeyRef: name: user-storage-binding key: bucketId deployment.yml (pod spec)

Service Catalog TL;DR • GKE(Kubernetes) 얺큲펞컪 GCP픦컪찒큲읊Service Instance옪 캫컿펺짢옪칺푷쿦핖삲 •
Service Account JSON Key픦뽆픒  먿헣힎팘팒솒쇪삲 • ⚠ Service Instance 칻헪킪킲헪읺콚큲 GCS쩒 SQL핆큲큲 솒  칻헪쇦삖훊픦

https://svc-cat.io

쭒칾킪큲펞컪팖헣헏픊옪폖퍋핟펓픒쿦쿦핖픒밚

CronJob • Cron syntax읊믆샎옪칺푷펺얺큲픦쭒칾핟펓픒묺솧쿦핖삲 • ⚠ 콚쩖킲(At Least Once)픒쫂핳믾쌚줆펞펺빦펺얺쩖킲 쇮캏펞샎샎찒많푢(de-duplication옪힏슿)

"빦쁢GPU많핖쁢Node펞줂혾멂큲흂쇦펂퍊몮   pool-a NodePool펞큲흂쇦졂홙멮힎잚  팖쇦펂솒멚캏뫎펔펂"

Afﬁnity • 쫂삲퓮펾몮삲퍟혾멂슲옪Pod핂펂쎉멚큲흂쇮힎헣픦쿦핖픚 • 펺얺찒묞펾칾핞칺푷많쁳  In, NotIn, Exists, DoesNotExist, Gt(>),
Lt(<) • Node믾훎픎Pod믾훎픊옪힎헣많쁳 BETA

"짆켦읺팮읺핂켦픦훟삶펔핂  Node읊펔팮몮탄픎섾펂쎉멚힎

"Kubernetes뺂펞컪  푆쭎컪찒큲펞샎컲헣픒쿦쁢펔픒밚?"

Service Catalog • GKE(Kubernetes) 얺큲펞컪 GCP픦컪찒큲읊Service Instance옪 캫컿펺짢옪칺푷쿦핖삲 • Service
Account JSON Key픦뽆픒  먿헣힎팘팒솒쇪삲 • ⚠ Service Instance 칻헪킪킲헪읺콚큲 GCS쩒 SQL핆큲큲 솒  칻헪쇦삖훊픦

CronJob Afﬁnity Pod Disruption Budget Service Catalog

https://twitter.com/_tr/status/1007619178222665730

맞칺삖삲 '

GCP+GKE Deep Dive Minku Lee CTO Shakr Shakr펞컪쁳엳핖쁢펢힎삖펂읊졶킻삖삲 careers.shakr.com

GCP+GKE Deep Dive Part 2: Advanced Cluster Management

GCP+GKE Deep Dive Part 2: Advanced Cluster Management

More Decks by Minku Lee

Other Decks in Programming

Featured

Transcript