GCP+GKE Deep Dive Part 2: Advanced Cluster Management

Part 1 바로가기: https://speakerdeck.com/premist/gcp-plus-gke-deep-dive-part-1-initial-app-development/

처음 GKE를 사용하기 시작하면 난해할 수 있는 서비스 배포, GKE 클러스터를 생성하고 여러 GCP의 서비스를 이용하여 첫 애플리케이션을 배포하는 과정까지 자세하게 살펴봅니다. 또한 애플리케이션을 배포한 이후 안정적인 서비스 운영을 위해 활용할 수 있는 클러스터 관리 테크닉을 소개합니다.

Part 2: Advanced Cluster Management
Kubernetes에는 분산 시스템을 구축하고 관리하는 것을 도와주는 다양한 기능이 있지만, 워낙 많은 사용 사례에 대비하다보니 어떤 기능이 있는지를 쉽게 간과하고 넘어가는 경우가 많습니다.

두 번째 파트에서는 클러스터와 애플리케이션을 보다 효율적으로 관리할 수 있도록 도와주는 여러 가지 리소스와 기능을 소개합니다. Google Cloud 서비스와 연동되어 Service Account 프로비저닝을 GKE 내에서 관리할 수 있게 도와주는 Service Broker, 한 번 실행되거나 시간대에 맞춰 자동으로 실행되는 CronJob, 애플리케이션의 SLO(Service Level Objective)를 유지하기 위한 Affinity 및 Pod Disruption Budget에 대해 알아봅니다.

필요 이해도: Kubernetes에 애플리케이션을 배포해 보고 관리해 보신 경험이 있는 분에게 적합합니다. Part 1을 듣고 이어서 들으셔도 무방합니다.

Transcript

1. PART 2 14:00~
   Advanced Cluster Management
   

   View Slide

2. 킪핟믾헒
   • Kubernetes펞샎믾쫆헏핆힎킫핂핖몮 

   팮읺핂켦픒짾쫆몋픒헒헪옪삖삲
   • Google Kubernetes Engine (GKE) 믾훎픊옪

   컲졓삖삲
   

   View Slide

3. CronJob
   

   View Slide

4. View Slide

5. 쭒칾킪큲펞컪팖헣헏픊옪폖퍋핟펓픒쿦쿦핖픒밚
   

   View Slide

6. View Slide

7. apiVersion: batch/v1beta1
   kind: CronJob
   metadata:
   name: recurring-job
   spec:
   schedule: "*/1 * * * *"
   jobTemplate:
   spec:
   template:
   spec:
   containers:
   - name: recurringwork
   image: recurringwork:latest
   args:
   - ./do-recurring.sh
   restartPolicy: OnFailure
   cronjob.yml
   

   View Slide

8. apiVersion: batch/v1beta1
   kind: CronJob
   metadata:
   name: recurring-job
   spec:
   schedule: "*/1 * * * *"
   concurrencyPolicy: Replace
   jobTemplate:
   spec:
   template:
   spec:
   containers:
   - name: recurringwork
   image: recurringwork:latest
   args:
   - ./do-recurring.sh
   restartPolicy: OnFailure
   cronjob.yml
   핟펓킪핟픒킪맒핂쇦펖쁢섾핂헒핟펓핂퐒헒븫빦힎팘팦삲졂
   Allow 솧킪킲픒푷
   Forbid 솧킪킲픒믖힎
   Replace 믾홂핟펓픒홓욚몮킲
   

   View Slide

9. $ kubectl apply -f cronjob.yml
   kubectl옪CronJob캫컿
   

   View Slide

10. CronJob
    • Cron syntax읊믆샎옪칺푷펺얺큲픦쭒칾핟펓픒묺솧
    쿦핖삲
    • ⚠ 콚쩖킲(At Least Once)픒쫂핳믾쌚줆펞펺
    빦펺얺쩖킲쇮캏펞샎샎찒많푢

    (de-duplication옪힏슿)
    

    View Slide

11. Affinity
    

    View Slide

12. View Slide

13. View Slide

14. View Slide

15. "핊헣혾멂픦NodePool펞잚

    Pod핂큲흂쇦솒옫쿦펔픒밚?"
    

    View Slide

16. apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: gitlab
    labels:
    app: gitlab
    spec:
    replicas: 1
    selector:
    matchLabels:
    app: gitlab
    template:
    metadata:
    labels:
    apps: gitlab
    deployment.yml
    spec:
    nodeSelector:
    cloud.google.com/gke-preemptible: "true"
    containers:
    - name: gitlab
    image: gitlab/gitlab-ce:latest
    resources:
    requests:
    cpu: "0.5"
    memory: 1Gi
    env:
    - name: GITLAB_OMNIBUS_CONFIG
    value: ...
    

    View Slide

17. "핊헣혾멂픦NodePool펞

    Pod핂큲흂쇦쁢멆컮힎잚
    쭖많몋푾펞쁢

    삲읆NodePool펞큲흂쇦솒옫쿦펔픒밚?"
    

    View Slide

18. Affinity
    • 쫂삲퓮펾몮삲퍟혾멂슲옪Pod핂펂쎉멚큲흂쇮힎

    헣픦쿦핖픚
    • 펺얺찒묞펾칾핞칺푷많쁳

    In, NotIn, Exists, DoesNotExist, Gt(>), Lt(<)
    • Node믾훎픎Pod믾훎픊옪힎헣많쁳
    BETA
    

    View Slide

19. Examples
    

    View Slide

20. "빦쁢GPU많핖쁢Node펞줂혾멂큲흂쇦펂퍊"
    

    View Slide

21. spec:
    affinity:
    nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
    - key: cloud.google.com/gke-accelerator
    operator: In
    values:
    - nvidia-tesla-p100
    - nvidia-tesla-v100
    pod-gpuonly.yml
    

    View Slide

22. "빦쁢GPU많핖쁢Node펞큲흂쇦쁢멆컮힎잚

    GPU많핖쁢Node많펔펂솒펂싢펢많쁢큲흂쇦펂퍊"
    

    View Slide

23. spec:
    affinity:
    nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
    - key: cloud.google.com/gke-accelerator
    operator: In
    values:
    - nvidia-tesla-p100
    - nvidia-tesla-v100
    pod-gpu-preferred.yml
    

    View Slide

24. "빦쁢GPU많핖쁢Node펞줂혾멂큲흂쇦펂퍊몮 

    pool-a NodePool펞큲흂쇦졂홙멮힎잚

    팖쇦펂솒멚캏뫎펔펂"
    

    View Slide

25. spec:
    affinity:
    nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
    - key: cloud.google.com/gke-accelerator
    operator: In
    values:
    - nvidia-tesla-p100
    - nvidia-tesla-v100
    preferredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
    - key: cloud.google.com/gke-nodepool
    operator: In
    values:
    - pool-a
    pod-gpu-and-nodepool.yml
    

    View Slide

26. "빦쁢app=gitlab핆섾 

    쇦솒옫빦퐎맧픎app펞콚콛쇪Pod뫊쁢삲읆

    Availability Zone펞큲흂쇦몮탄펂
    

    View Slide

27. spec:
    affinity:
    podAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
    matchExpressions:
    - key: app
    operator: In
    values:
    - gitlab
    topologyKey: failure-domain.beta.kubernetes.io/zone
    different-zone-preferred.yml
    

    View Slide

28. Affinity
    • 삶핊얺큲옪삲퍟풚옪슪읊힎풞쌚솒풎핂잜핂쇦쁢믾쁳
    • ⚠ requiredDuringSchedulingIgnoredDuringExecution픒
    잜핂칺푷컪큲흂쇮쿦펔쁢Pod핂

    잜팒힎쁢멑픒핦뫎읺퍊
    BETA
    

    View Slide

29. Pod Disruption Budgets
    

    View Slide

30. View Slide

31. node-1
    web api
    job
    node-2
    web api
    job
    node-3
    web api
    job
    

    View Slide

32. web api
    job
    node-2
    web api
    job
    node-3
    web api
    job
    

    View Slide

33. web api
    job
    node-2
    web api
    job
    node-3
    web api
    job
    #
    

    View Slide

34. "짆켦읺팮읺핂켦픦훟삶펔핂

    Node읊펔팮몮탄픎섾펂쎉멚힎
    

    View Slide

35. Pod Disruption Budget
    • 혾멂펞재쁢Pod핂캏헣힒맽쿦옪퓮힎쇮멑픒

    맣헪쿦핖쁢믾쁳
    • minAvailable짝maxUnavailable 퐃켦픊옪컲헣
    • ⚠ 핞짪헏핆disruption픦몋푾펞잚PDB 훎쿦많쇦즎옪 

    폖믾팘픎/PEF픦핳팮짪캫킪펞쁢PDB많힎힎힎팘픒쿦핖픚
    BETA
    

    View Slide

36. apiVersion: policy/v1beta1
    kind: PodDisruptionBudget
    metadata:
    name: gitlab-pdb
    spec:
    minAvailable: 2
    selector:
    matchLabels:
    app: gitlab
    pod-disruption-budget.yml
    

    View Slide

37. Pod Disruption Budget
    • 혾멂펞재쁢Pod핂캏헣힒맽쿦옪퓮힎쇮멑픒

    맣헪쿦핖쁢믾쁳
    • minAvailable짝maxUnavailable 퐃켦픊옪컲헣
    • ⚠ 핞짪헏핆disruption픦몋푾펞잚PDB 훎쿦많쇦즎옪 

    폖믾팘픎/PEF픦핳팮짪캫킪펞쁢PDB많힎힎힎팘픒쿦핖픚
    BETA
    

    View Slide

38. Node Draining
    $ kubectl cordon NODE
    $ kubectl drain NODE
    훊펂힒Node읊큲흂핂쇦힎팘솒옫힎헣삲
    훊펂힒Node읊큲흂핂쇦힎팘솒옫힎헣몮 

    맏Pod읊칻헪삲
    

    View Slide

39. $ kubectl cordon gke-my-cluster-my-pool-592cda94-2w25
    node "gke-my-cluster-my-pool-592cda94-2w25" cordoned
    $ kubectl describe gke-my-cluster-my-pool-592cda94-2w25
    Events:
    Type Reason Age From Message
    ---- ------ ---- ---- -------
    Normal NodeNotSchedulable 45s kubelet, ... Node status is now: NodeNotSchedulable
    kubectl cordon
    

    View Slide

40. Google Kubernetes Engine픎Node쪎몋킪drain픒팚팒컪읺삲
    

    View Slide

41. 핞솧Node펓믆엖핂슪빦Autoscale 칺푷킪Pod Disruption Budget픎쿦
    

    View Slide

42. Service Catalog and Broker
    

    View Slide

43. Pod
    Secret
    serviceaccount.json: ...
    Cloud SQL
    Instance
    Kubernetes(GKE) 짤펞컪
    쿦솧픊옪캫컿퍊
    

    View Slide

44. $ gcloud iam service-accounts create gitlab \
    --display-name="GitLab Service Account"
    Service Account캫컿
    

    View Slide

45. $ gcloud projects add-iam-policy-binding $PROJECT \
    --member="serviceAccount:$EMAIL"\
    --role="roles/cloudsql.client"
    뭚쭎펺
    

    View Slide

46. $ gcloud iam service-accounts keys create \
    ./artifacts/serviceaccount.json \
    --iam-account $EMAIL
    슫
    

    View Slide

47. $ kubectl create secret generic gitlab-config \
    ...
    --from-file=./artifacts/serviceaccount.json
    kubectl픒핂푷Secret캫컿
    $
    

    View Slide

48. "Kubernetes뺂펞컪

    푆쭎컪찒큲펞샎컲헣픒쿦쁢펔픒밚?"
    

    View Slide

49. Service CatalogBETA
    

    View Slide

50. View Slide

51. %
    

    View Slide

52. Pod
    Secret
    serviceaccount.json: ...
    Cloud SQL
    Instance
    Kubernetes(GKE) 짤펞컪
    쿦솧픊옪캫컿퍊
    

    View Slide

53. Cloud SQL
    Instance
    Service

    Account
    Service Instance
    Service Instance
    

    View Slide

54. 핺힎풞쇦쁢Service Instance홓윦
    Service Account
    Cloud Spanner Cloud Pub/Sub
    Cloud SQL (MySQL)
    BigQuery Cloud BigTable Cloud Storage
    

    View Slide

55. $ kubectl create clusterrolebinding cluster-admin-binding \
    --clusterrole=cluster-admin \
    --user=$(gcloud config get-value account)
    clusterrolebinding "cluster-admin-binding" created
    Service Catalog컲
    https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog
    

    View Slide

56. View Slide

57. $ sc install
    account: account@example.com
    project: shakr-openinfra-demo
    zone:
    generated service catalog deployment config in dir: /tmp/service-
    catalog544428136
    Service Catalog installed successfully.
    Service Catalog컲
    https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog
    

    View Slide

58. $ sc add-gcp-broker
    using project: shakr-openinfra-demo
    enabling a GCP API: servicebroker.googleapis.com
    enabling a GCP API: bigtableadmin.googleapis.com
    enabling a GCP API: ml.googleapis.com
    ...
    The Service Broker has been added successfully.
    Service Catalog컲
    https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog
    

    View Slide

59. $ gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member serviceAccount:$EMAIL \
    --role=roles/owner
    Service Catalog컲
    https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog
    

    View Slide

60. $ kubectl -o "custom-
    columns=NAME:.spec.externalName,DESCRIPTION:.spec.description" \
    get clusterserviceclasses
    NAME DESCRIPTION
    cloud-spanner The first horizontally scalable...
    cloud-iam-service-account Specialized service which provisions...
    cloud-pubsub Ingest event streams from anywhere...
    cloud-sql-mysql A fully-managed MySQL database service
    bigquery A fast, highly scalable, cost-effective
    cloud-bigtable A high performance NoSQL database
    Service Catalog핆
    

    View Slide

61. apiVersion: servicecatalog.k8s.io/v1beta1
    kind: ServiceInstance
    metadata:
    name: test-storage
    namespace: default
    spec:
    clusterServiceClassExternalName: cloud-storage
    clusterServicePlanExternalName: beta
    parameters:
    bucketId: shakr-openinfra-demo-test-storage
    location: US
    storageClass: STANDARD
    serviceinstance.yml
    

    View Slide

62. $ kubectl apply -f serviceinstance.yml
    kubectl옪Service Instance캫컿
    

    View Slide

63. View Slide

64. &&&&
    

    View Slide

65. apiVersion: servicecatalog.k8s.io/v1beta1
    kind: ServiceInstance
    metadata:
    name: test-storage
    namespace: default
    spec:
    clusterServiceClassExternalName: cloud-storage
    clusterServicePlanExternalName: beta
    parameters:
    bucketId: shakr-openinfra-demo-test-storage
    location: US
    storageClass: STANDARD
    serviceinstance.yml
    

    View Slide

66. https://twitter.com/tenderlove/status/988887936128040960
    

    View Slide

67. 콢픒칺푷펺Service Instance캫컿
    

    View Slide

68. 콢픒칺푷펺Service Instance캫컿
    

    View Slide

69. $ svcat provision test-storage \
    --class cloud-storage \
    --plan beta \
    --namespace default \
    --param bucketId=shakr-openinfra-demo-test-storage \
    --param location=US \
    --param storageClass=STANDARD
    svcat픊옪Service Instance캫컿
    

    View Slide

70. Service Binding
    

    View Slide

71. Cloud Storage
    Instance
    Service Instance
    Service Instance
    Service Binding
    Service Binding
    Secret
    privateKeyData: ...
    Service
    Account
    

    View Slide

72. Cloud Storage
    Instance
    Service Instance Service Binding
    Secret
    privateKeyData: ...
    createServiceAccount: true
    

    View Slide

73. $ svcat bind test-storage \
    --name test-storage-binding \
    --params-json \
    '{
    "serviceAccount": "test-storage-serviceaccount",
    "createServiceAccount": true,
    "roles": [
    "roles/storage.objectCreator",
    "roles/storage.objectViewer"
    ]
    }'
    Service Binding캫컿
    

    View Slide

74. $ kubectl get secrets test-storage-binding
    NAME TYPE DATA AGE
    test-storage-binding Opaque 2 5m
    Service Account Secret핆
    

    View Slide

75. spec:
    volumes:
    - name: test-storage-binding
    secret:
    secretName: test-storage-binding
    containers:
    - name: my-app
    image: shakr/my-app:latest
    volumeMounts:
    - name: binding
    mountPath: /mnt/binding
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
    value: /mnt/binding/privateKeyData
    - name: STORAGE_PROJECT
    valueFrom:
    secretKeyRef:
    name: user-storage-binding
    key: projectId
    - name: STORAGE_BUCKET
    valueFrom:
    secretKeyRef:
    name: user-storage-binding
    key: bucketId
    deployment.yml (pod spec)
    

    View Slide

76. Service Catalog TL;DR
    • GKE(Kubernetes) 얺큲펞컪 GCP픦컪찒큲읊Service Instance옪
    캫컿펺짢옪칺푷쿦핖삲
    • Service Account JSON Key픦뽆픒

    먿헣힎팘팒솒쇪삲
    • ⚠ Service Instance 칻헪킪킲헪읺콚큲 GCS쩒 SQL핆큲큲
    솒

    칻헪쇦삖훊픦
    

    View Slide

77. https://svc-cat.io
    

    View Slide

78. Recap
    

    View Slide

79. 쭒칾킪큲펞컪팖헣헏픊옪폖퍋핟펓픒쿦쿦핖픒밚
    

    View Slide

80. CronJob
    • Cron syntax읊믆샎옪칺푷펺얺큲픦쭒칾핟펓픒묺솧쿦핖삲
    • ⚠ 콚쩖킲(At Least Once)픒쫂핳믾쌚줆펞펺빦펺얺쩖킲
    쇮캏펞샎샎찒많푢(de-duplication옪힏슿)
    

    View Slide

81. "빦쁢GPU많핖쁢Node펞줂혾멂큲흂쇦펂퍊몮 

    pool-a NodePool펞큲흂쇦졂홙멮힎잚

    팖쇦펂솒멚캏뫎펔펂"
    

    View Slide

82. Affinity
    • 쫂삲퓮펾몮삲퍟혾멂슲옪Pod핂펂쎉멚큲흂쇮힎헣픦쿦핖픚
    • 펺얺찒묞펾칾핞칺푷많쁳

    In, NotIn, Exists, DoesNotExist, Gt(>), Lt(<)
    • Node믾훎픎Pod믾훎픊옪힎헣많쁳
    BETA
    

    View Slide

83. "짆켦읺팮읺핂켦픦훟삶펔핂

    Node읊펔팮몮탄픎섾펂쎉멚힎
    

    View Slide

84. Pod Disruption Budget
    • 혾멂펞재쁢Pod핂캏헣힒맽쿦옪퓮힎쇮멑픒

    맣헪쿦핖쁢믾쁳
    • minAvailable짝maxUnavailable 퐃켦픊옪컲헣
    • ⚠ 핞짪헏핆disruption픦몋푾펞잚PDB 훎쿦많쇦즎옪 

    폖믾팘픎/PEF픦핳팮짪캫킪펞쁢PDB많힎힎힎팘픒쿦핖픚
    BETA
    

    View Slide

85. "Kubernetes뺂펞컪

    푆쭎컪찒큲펞샎컲헣픒쿦쁢펔픒밚?"
    

    View Slide

86. Service Catalog
    • GKE(Kubernetes) 얺큲펞컪 GCP픦컪찒큲읊Service Instance옪
    캫컿펺짢옪칺푷쿦핖삲
    • Service Account JSON Key픦뽆픒

    먿헣힎팘팒솒쇪삲
    • ⚠ Service Instance 칻헪킪킲헪읺콚큲 GCS쩒 SQL핆큲큲
    솒

    칻헪쇦삖훊픦
    

    View Slide

87. CronJob
    Affinity
    Pod Disruption Budget
    Service Catalog
    

    View Slide

88. https://twitter.com/_tr/status/1007619178222665730
    

    View Slide

89. 맞칺삖삲 '
    

    View Slide

90. GCP+GKE Deep Dive
    Minku Lee
    CTO Shakr
    Shakr펞컪쁳엳핖쁢펢힎삖펂읊졶킻삖삲
    careers.shakr.com
    

    View Slide