GCP+GKE Deep Dive Part 2: Advanced Cluster Management

Slide 1

Slide 1 text

PART 2 14:00~ Advanced Cluster Management

Slide 2

Slide 2 text

킪핟믾헒 • Kubernetes펞샎믾쫆헏핆힎킫핂핖몮   팮읺핂켦픒짾쫆몋픒헒헪옪삖삲 • Google Kubernetes Engine (GKE) 믾훎픊옪  컲졓삖삲

Slide 3

Slide 3 text

CronJob

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

쭒칾킪큲펞컪팖헣헏픊옪폖퍋핟펓픒쿦쿦핖픒밚

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

apiVersion: batch/v1beta1 kind: CronJob metadata: name: recurring-job spec: schedule: "*/1 * * * *" jobTemplate: spec: template: spec: containers: - name: recurringwork image: recurringwork:latest args: - ./do-recurring.sh restartPolicy: OnFailure cronjob.yml

Slide 8

Slide 8 text

apiVersion: batch/v1beta1 kind: CronJob metadata: name: recurring-job spec: schedule: "*/1 * * * *" concurrencyPolicy: Replace jobTemplate: spec: template: spec: containers: - name: recurringwork image: recurringwork:latest args: - ./do-recurring.sh restartPolicy: OnFailure cronjob.yml 핟펓킪핟픒킪맒핂쇦펖쁢섾핂헒핟펓핂퐒헒븫빦힎팘팦삲졂 Allow 솧킪킲픒푷 Forbid 솧킪킲픒믖힎 Replace 믾홂핟펓픒홓욚몮킲

Slide 9

Slide 9 text

$ kubectl apply -f cronjob.yml kubectl옪CronJob캫컿

Slide 10

Slide 10 text

CronJob • Cron syntax읊믆샎옪칺푷펺얺큲픦쭒칾핟펓픒묺솧 쿦핖삲 • ⚠ 콚쩖킲(At Least Once)픒쫂핳믾쌚줆펞펺 빦펺얺쩖킲쇮캏펞샎샎찒많푢  (de-duplication옪힏슿)

Slide 11

Slide 11 text

Afﬁnity

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

"핊헣혾멂픦NodePool펞잚  Pod핂큲흂쇦솒옫쿦펔픒밚?"

Slide 16

Slide 16 text

apiVersion: apps/v1 kind: Deployment metadata: name: gitlab labels: app: gitlab spec: replicas: 1 selector: matchLabels: app: gitlab template: metadata: labels: apps: gitlab deployment.yml spec: nodeSelector: cloud.google.com/gke-preemptible: "true" containers: - name: gitlab image: gitlab/gitlab-ce:latest resources: requests: cpu: "0.5" memory: 1Gi env: - name: GITLAB_OMNIBUS_CONFIG value: ...

Slide 17

Slide 17 text

"핊헣혾멂픦NodePool펞  Pod핂큲흂쇦쁢멆컮힎잚 쭖많몋푾펞쁢  삲읆NodePool펞큲흂쇦솒옫쿦펔픒밚?"

Slide 18

Slide 18 text

Afﬁnity • 쫂삲퓮펾몮삲퍟혾멂슲옪Pod핂펂쎉멚큲흂쇮힎  헣픦쿦핖픚 • 펺얺찒묞펾칾핞칺푷많쁳  In, NotIn, Exists, DoesNotExist, Gt(>), Lt(<) • Node믾훎픎Pod믾훎픊옪힎헣많쁳 BETA

Slide 19

Slide 19 text

Examples

Slide 20

Slide 20 text

"빦쁢GPU많핖쁢Node펞줂혾멂큲흂쇦펂퍊"

Slide 21

Slide 21 text

spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-accelerator operator: In values: - nvidia-tesla-p100 - nvidia-tesla-v100 pod-gpuonly.yml

Slide 22

Slide 22 text

"빦쁢GPU많핖쁢Node펞큲흂쇦쁢멆컮힎잚  GPU많핖쁢Node많펔펂솒펂싢펢많쁢큲흂쇦펂퍊"

Slide 23

Slide 23 text

Slide 24

Slide 24 text

"빦쁢GPU많핖쁢Node펞줂혾멂큲흂쇦펂퍊몮   pool-a NodePool펞큲흂쇦졂홙멮힎잚  팖쇦펂솒멚캏뫎펔펂"

Slide 25

Slide 25 text

spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-accelerator operator: In values: - nvidia-tesla-p100 - nvidia-tesla-v100 preferredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-nodepool operator: In values: - pool-a pod-gpu-and-nodepool.yml

Slide 26

Slide 26 text

"빦쁢app=gitlab핆섾   쇦솒옫빦퐎맧픎app펞콚콛쇪Pod뫊쁢삲읆  Availability Zone펞큲흂쇦몮탄펂

Slide 27

Slide 27 text

spec: affinity: podAffinity: preferredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - gitlab topologyKey: failure-domain.beta.kubernetes.io/zone different-zone-preferred.yml

Slide 28

Slide 28 text

Afﬁnity • 삶핊얺큲옪삲퍟풚옪슪읊힎풞쌚솒풎핂잜핂쇦쁢믾쁳 • ⚠ requiredDuringSchedulingIgnoredDuringExecution픒 잜핂칺푷컪큲흂쇮쿦펔쁢Pod핂  잜팒힎쁢멑픒핦뫎읺퍊 BETA

Slide 29

Slide 29 text

Pod Disruption Budgets

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

node-1 web api job node-2 web api job node-3 web api job

Slide 32

Slide 32 text

web api job node-2 web api job node-3 web api job

Slide 33

Slide 33 text

web api job node-2 web api job node-3 web api job #

Slide 34

Slide 34 text

"짆켦읺팮읺핂켦픦훟삶펔핂  Node읊펔팮몮탄픎섾펂쎉멚힎

Slide 35

Slide 35 text

Pod Disruption Budget • 혾멂펞재쁢Pod핂캏헣힒맽쿦옪퓮힎쇮멑픒  맣헪쿦핖쁢믾쁳 • minAvailable짝maxUnavailable 퐃켦픊옪컲헣 • ⚠ 핞짪헏핆disruption픦몋푾펞잚PDB 훎쿦많쇦즎옪   폖믾팘픎/PEF픦핳팮짪캫킪펞쁢PDB많힎힎힎팘픒쿦핖픚 BETA

Slide 36

Slide 36 text

apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: gitlab-pdb spec: minAvailable: 2 selector: matchLabels: app: gitlab pod-disruption-budget.yml

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Node Draining $ kubectl cordon NODE $ kubectl drain NODE 훊펂힒Node읊큲흂핂쇦힎팘솒옫힎헣삲 훊펂힒Node읊큲흂핂쇦힎팘솒옫힎헣몮   맏Pod읊칻헪삲

Slide 39

Slide 39 text

$ kubectl cordon gke-my-cluster-my-pool-592cda94-2w25 node "gke-my-cluster-my-pool-592cda94-2w25" cordoned $ kubectl describe gke-my-cluster-my-pool-592cda94-2w25 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal NodeNotSchedulable 45s kubelet, ... Node status is now: NodeNotSchedulable kubectl cordon

Slide 40

Slide 40 text

Google Kubernetes Engine픎Node쪎몋킪drain픒팚팒컪읺삲

Slide 41

Slide 41 text

핞솧Node펓믆엖핂슪빦Autoscale 칺푷킪Pod Disruption Budget픎쿦

Slide 42

Slide 42 text

Service Catalog and Broker

Slide 43

Slide 43 text

Pod Secret serviceaccount.json: ... Cloud SQL Instance Kubernetes(GKE) 짤펞컪 쿦솧픊옪캫컿퍊

Slide 44

Slide 44 text

$ gcloud iam service-accounts create gitlab \ --display-name="GitLab Service Account" Service Account캫컿

Slide 45

Slide 45 text

$ gcloud projects add-iam-policy-binding $PROJECT \ --member="serviceAccount:$EMAIL"\ --role="roles/cloudsql.client" 뭚쭎펺

Slide 46

Slide 46 text

$ gcloud iam service-accounts keys create \ ./artifacts/serviceaccount.json \ --iam-account $EMAIL 슫

Slide 47

Slide 47 text

$ kubectl create secret generic gitlab-config \ ... --from-file=./artifacts/serviceaccount.json kubectl픒핂푷Secret캫컿 $

Slide 48

Slide 48 text

"Kubernetes뺂펞컪  푆쭎컪찒큲펞샎컲헣픒쿦쁢펔픒밚?"

Slide 49

Slide 49 text

Service CatalogBETA

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

Slide 52

Slide 52 text

Pod Secret serviceaccount.json: ... Cloud SQL Instance Kubernetes(GKE) 짤펞컪 쿦솧픊옪캫컿퍊

Slide 53

Slide 53 text

Cloud SQL Instance Service  Account Service Instance Service Instance

Slide 54

Slide 54 text

핺힎풞쇦쁢Service Instance홓윦 Service Account Cloud Spanner Cloud Pub/Sub Cloud SQL (MySQL) BigQuery Cloud BigTable Cloud Storage

Slide 55

Slide 55 text

$ kubectl create clusterrolebinding cluster-admin-binding \ --clusterrole=cluster-admin \ --user=$(gcloud config get-value account) clusterrolebinding "cluster-admin-binding" created Service Catalog컲 https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

$ sc install account: [email protected] project: shakr-openinfra-demo zone: generated service catalog deployment config in dir: /tmp/service- catalog544428136 Service Catalog installed successfully. Service Catalog컲 https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog

Slide 58

Slide 58 text

$ sc add-gcp-broker using project: shakr-openinfra-demo enabling a GCP API: servicebroker.googleapis.com enabling a GCP API: bigtableadmin.googleapis.com enabling a GCP API: ml.googleapis.com ... The Service Broker has been added successfully. Service Catalog컲 https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog

Slide 59

Slide 59 text

$ gcloud projects add-iam-policy-binding $PROJECT_ID \ --member serviceAccount:$EMAIL \ --role=roles/owner Service Catalog컲 https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog

Slide 60

Slide 60 text

$ kubectl -o "custom- columns=NAME:.spec.externalName,DESCRIPTION:.spec.description" \ get clusterserviceclasses NAME DESCRIPTION cloud-spanner The first horizontally scalable... cloud-iam-service-account Specialized service which provisions... cloud-pubsub Ingest event streams from anywhere... cloud-sql-mysql A fully-managed MySQL database service bigquery A fast, highly scalable, cost-effective cloud-bigtable A high performance NoSQL database Service Catalog핆

Slide 61

Slide 61 text

apiVersion: servicecatalog.k8s.io/v1beta1 kind: ServiceInstance metadata: name: test-storage namespace: default spec: clusterServiceClassExternalName: cloud-storage clusterServicePlanExternalName: beta parameters: bucketId: shakr-openinfra-demo-test-storage location: US storageClass: STANDARD serviceinstance.yml

Slide 62

Slide 62 text

$ kubectl apply -f serviceinstance.yml kubectl옪Service Instance캫컿

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

&&&&

Slide 65

Slide 65 text

Slide 66

Slide 66 text

https://twitter.com/tenderlove/status/988887936128040960

Slide 67

Slide 67 text

콢픒칺푷펺Service Instance캫컿

Slide 68

Slide 68 text

콢픒칺푷펺Service Instance캫컿

Slide 69

Slide 69 text

$ svcat provision test-storage \ --class cloud-storage \ --plan beta \ --namespace default \ --param bucketId=shakr-openinfra-demo-test-storage \ --param location=US \ --param storageClass=STANDARD svcat픊옪Service Instance캫컿

Slide 70

Slide 70 text

Service Binding

Slide 71

Slide 71 text

Cloud Storage Instance Service Instance Service Instance Service Binding Service Binding Secret privateKeyData: ... Service Account

Slide 72

Slide 72 text

Cloud Storage Instance Service Instance Service Binding Secret privateKeyData: ... createServiceAccount: true

Slide 73

Slide 73 text

$ svcat bind test-storage \ --name test-storage-binding \ --params-json \ '{ "serviceAccount": "test-storage-serviceaccount", "createServiceAccount": true, "roles": [ "roles/storage.objectCreator", "roles/storage.objectViewer" ] }' Service Binding캫컿

Slide 74

Slide 74 text

$ kubectl get secrets test-storage-binding NAME TYPE DATA AGE test-storage-binding Opaque 2 5m Service Account Secret핆

Slide 75

Slide 75 text

spec: volumes: - name: test-storage-binding secret: secretName: test-storage-binding containers: - name: my-app image: shakr/my-app:latest volumeMounts: - name: binding mountPath: /mnt/binding env: - name: GOOGLE_APPLICATION_CREDENTIALS value: /mnt/binding/privateKeyData - name: STORAGE_PROJECT valueFrom: secretKeyRef: name: user-storage-binding key: projectId - name: STORAGE_BUCKET valueFrom: secretKeyRef: name: user-storage-binding key: bucketId deployment.yml (pod spec)

Slide 76

Slide 76 text

Service Catalog TL;DR • GKE(Kubernetes) 얺큲펞컪 GCP픦컪찒큲읊Service Instance옪 캫컿펺짢옪칺푷쿦핖삲 • Service Account JSON Key픦뽆픒  먿헣힎팘팒솒쇪삲 • ⚠ Service Instance 칻헪킪킲헪읺콚큲 GCS쩒 SQL핆큲큲 솒  칻헪쇦삖훊픦

Slide 77

Slide 77 text

https://svc-cat.io

Slide 78

Slide 78 text

Recap

Slide 79

Slide 79 text

쭒칾킪큲펞컪팖헣헏픊옪폖퍋핟펓픒쿦쿦핖픒밚

Slide 80

Slide 80 text

CronJob • Cron syntax읊믆샎옪칺푷펺얺큲픦쭒칾핟펓픒묺솧쿦핖삲 • ⚠ 콚쩖킲(At Least Once)픒쫂핳믾쌚줆펞펺빦펺얺쩖킲 쇮캏펞샎샎찒많푢(de-duplication옪힏슿)

Slide 81

Slide 81 text

"빦쁢GPU많핖쁢Node펞줂혾멂큲흂쇦펂퍊몮   pool-a NodePool펞큲흂쇦졂홙멮힎잚  팖쇦펂솒멚캏뫎펔펂"

Slide 82

Slide 82 text

Afﬁnity • 쫂삲퓮펾몮삲퍟혾멂슲옪Pod핂펂쎉멚큲흂쇮힎헣픦쿦핖픚 • 펺얺찒묞펾칾핞칺푷많쁳  In, NotIn, Exists, DoesNotExist, Gt(>), Lt(<) • Node믾훎픎Pod믾훎픊옪힎헣많쁳 BETA

Slide 83

Slide 83 text

"짆켦읺팮읺핂켦픦훟삶펔핂  Node읊펔팮몮탄픎섾펂쎉멚힎

Slide 84

Slide 84 text

Slide 85

Slide 85 text

"Kubernetes뺂펞컪  푆쭎컪찒큲펞샎컲헣픒쿦쁢펔픒밚?"

Slide 86

Slide 86 text

Service Catalog • GKE(Kubernetes) 얺큲펞컪 GCP픦컪찒큲읊Service Instance옪 캫컿펺짢옪칺푷쿦핖삲 • Service Account JSON Key픦뽆픒  먿헣힎팘팒솒쇪삲 • ⚠ Service Instance 칻헪킪킲헪읺콚큲 GCS쩒 SQL핆큲큲 솒  칻헪쇦삖훊픦

Slide 87

Slide 87 text

CronJob Afﬁnity Pod Disruption Budget Service Catalog

Slide 88

Slide 88 text

https://twitter.com/_tr/status/1007619178222665730

Slide 89

Slide 89 text

맞칺삖삲 '

Slide 90

Slide 90 text

GCP+GKE Deep Dive Minku Lee CTO Shakr Shakr펞컪쁳엳핖쁢펢힎삖펂읊졶킻삖삲 careers.shakr.com