$30 off During Our Annual Pro Sale. View Details »

GCP+GKE Deep Dive Part 2: Advanced Cluster Management

GCP+GKE Deep Dive Part 2: Advanced Cluster Management

Part 1 바로가기: https://speakerdeck.com/premist/gcp-plus-gke-deep-dive-part-1-initial-app-development/

처음 GKE를 사용하기 시작하면 난해할 수 있는 서비스 배포, GKE 클러스터를 생성하고 여러 GCP의 서비스를 이용하여 첫 애플리케이션을 배포하는 과정까지 자세하게 살펴봅니다. 또한 애플리케이션을 배포한 이후 안정적인 서비스 운영을 위해 활용할 수 있는 클러스터 관리 테크닉을 소개합니다.

Part 2: Advanced Cluster Management
Kubernetes에는 분산 시스템을 구축하고 관리하는 것을 도와주는 다양한 기능이 있지만, 워낙 많은 사용 사례에 대비하다보니 어떤 기능이 있는지를 쉽게 간과하고 넘어가는 경우가 많습니다.

두 번째 파트에서는 클러스터와 애플리케이션을 보다 효율적으로 관리할 수 있도록 도와주는 여러 가지 리소스와 기능을 소개합니다. Google Cloud 서비스와 연동되어 Service Account 프로비저닝을 GKE 내에서 관리할 수 있게 도와주는 Service Broker, 한 번 실행되거나 시간대에 맞춰 자동으로 실행되는 CronJob, 애플리케이션의 SLO(Service Level Objective)를 유지하기 위한 Affinity 및 Pod Disruption Budget에 대해 알아봅니다.

필요 이해도: Kubernetes에 애플리케이션을 배포해 보고 관리해 보신 경험이 있는 분에게 적합합니다. Part 1을 듣고 이어서 들으셔도 무방합니다.

Minku Lee

June 29, 2018
Tweet

More Decks by Minku Lee

Other Decks in Programming

Transcript

  1. PART 2 14:00~
    Advanced Cluster Management

    View Slide

  2. 킪핟믾헒
    • Kubernetes펞샎믾쫆헏핆힎킫핂핖몮 

    팮읺핂켦픒짾쫆몋픒헒헪옪삖삲
    • Google Kubernetes Engine (GKE) 믾훎픊옪

    컲졓삖삲

    View Slide

  3. CronJob

    View Slide

  4. View Slide

  5. 쭒칾킪큲펞컪팖헣헏픊옪폖퍋핟펓픒쿦쿦핖픒밚

    View Slide

  6. View Slide

  7. apiVersion: batch/v1beta1
    kind: CronJob
    metadata:
    name: recurring-job
    spec:
    schedule: "*/1 * * * *"
    jobTemplate:
    spec:
    template:
    spec:
    containers:
    - name: recurringwork
    image: recurringwork:latest
    args:
    - ./do-recurring.sh
    restartPolicy: OnFailure
    cronjob.yml

    View Slide

  8. apiVersion: batch/v1beta1
    kind: CronJob
    metadata:
    name: recurring-job
    spec:
    schedule: "*/1 * * * *"
    concurrencyPolicy: Replace
    jobTemplate:
    spec:
    template:
    spec:
    containers:
    - name: recurringwork
    image: recurringwork:latest
    args:
    - ./do-recurring.sh
    restartPolicy: OnFailure
    cronjob.yml
    핟펓킪핟픒킪맒핂쇦펖쁢섾핂헒핟펓핂퐒헒븫빦힎팘팦삲졂
    Allow 솧킪킲픒푷
    Forbid 솧킪킲픒믖힎
    Replace 믾홂핟펓픒홓욚몮킲

    View Slide

  9. $ kubectl apply -f cronjob.yml
    kubectl옪CronJob캫컿

    View Slide

  10. CronJob
    • Cron syntax읊믆샎옪칺푷펺얺큲픦쭒칾핟펓픒묺솧
    쿦핖삲
    • ⚠ 콚쩖킲(At Least Once)픒쫂핳믾쌚줆펞펺
    빦펺얺쩖킲쇮캏펞샎샎찒많푢

    (de-duplication옪힏슿)

    View Slide

  11. Affinity

    View Slide

  12. View Slide

  13. View Slide

  14. View Slide

  15. "핊헣혾멂픦NodePool펞잚

    Pod핂큲흂쇦솒옫쿦펔픒밚?"

    View Slide

  16. apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: gitlab
    labels:
    app: gitlab
    spec:
    replicas: 1
    selector:
    matchLabels:
    app: gitlab
    template:
    metadata:
    labels:
    apps: gitlab
    deployment.yml
    spec:
    nodeSelector:
    cloud.google.com/gke-preemptible: "true"
    containers:
    - name: gitlab
    image: gitlab/gitlab-ce:latest
    resources:
    requests:
    cpu: "0.5"
    memory: 1Gi
    env:
    - name: GITLAB_OMNIBUS_CONFIG
    value: ...

    View Slide

  17. "핊헣혾멂픦NodePool펞

    Pod핂큲흂쇦쁢멆컮힎잚
    쭖많몋푾펞쁢

    삲읆NodePool펞큲흂쇦솒옫쿦펔픒밚?"

    View Slide

  18. Affinity
    • 쫂삲퓮펾몮삲퍟혾멂슲옪Pod핂펂쎉멚큲흂쇮힎

    헣픦쿦핖픚
    • 펺얺찒묞펾칾핞칺푷많쁳

    In, NotIn, Exists, DoesNotExist, Gt(>), Lt(<)
    • Node믾훎픎Pod믾훎픊옪힎헣많쁳
    BETA

    View Slide

  19. Examples

    View Slide

  20. "빦쁢GPU많핖쁢Node펞줂혾멂큲흂쇦펂퍊"

    View Slide

  21. spec:
    affinity:
    nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
    - key: cloud.google.com/gke-accelerator
    operator: In
    values:
    - nvidia-tesla-p100
    - nvidia-tesla-v100
    pod-gpuonly.yml

    View Slide

  22. "빦쁢GPU많핖쁢Node펞큲흂쇦쁢멆컮힎잚

    GPU많핖쁢Node많펔펂솒펂싢펢많쁢큲흂쇦펂퍊"

    View Slide

  23. spec:
    affinity:
    nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
    - key: cloud.google.com/gke-accelerator
    operator: In
    values:
    - nvidia-tesla-p100
    - nvidia-tesla-v100
    pod-gpu-preferred.yml

    View Slide

  24. "빦쁢GPU많핖쁢Node펞줂혾멂큲흂쇦펂퍊몮 

    pool-a NodePool펞큲흂쇦졂홙멮힎잚

    팖쇦펂솒멚캏뫎펔펂"

    View Slide

  25. spec:
    affinity:
    nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
    - key: cloud.google.com/gke-accelerator
    operator: In
    values:
    - nvidia-tesla-p100
    - nvidia-tesla-v100
    preferredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
    - key: cloud.google.com/gke-nodepool
    operator: In
    values:
    - pool-a
    pod-gpu-and-nodepool.yml

    View Slide

  26. "빦쁢app=gitlab핆섾 

    쇦솒옫빦퐎맧픎app펞콚콛쇪Pod뫊쁢삲읆

    Availability Zone펞큲흂쇦몮탄펂

    View Slide

  27. spec:
    affinity:
    podAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
    matchExpressions:
    - key: app
    operator: In
    values:
    - gitlab
    topologyKey: failure-domain.beta.kubernetes.io/zone
    different-zone-preferred.yml

    View Slide

  28. Affinity
    • 삶핊얺큲옪삲퍟풚옪슪읊힎풞쌚솒풎핂잜핂쇦쁢믾쁳
    • ⚠ requiredDuringSchedulingIgnoredDuringExecution픒
    잜핂칺푷컪큲흂쇮쿦펔쁢Pod핂

    잜팒힎쁢멑픒핦뫎읺퍊
    BETA

    View Slide

  29. Pod Disruption Budgets

    View Slide

  30. View Slide

  31. node-1
    web api
    job
    node-2
    web api
    job
    node-3
    web api
    job

    View Slide

  32. web api
    job
    node-2
    web api
    job
    node-3
    web api
    job

    View Slide

  33. web api
    job
    node-2
    web api
    job
    node-3
    web api
    job
    #

    View Slide

  34. "짆켦읺팮읺핂켦픦훟삶펔핂

    Node읊펔팮몮탄픎섾펂쎉멚힎

    View Slide

  35. Pod Disruption Budget
    • 혾멂펞재쁢Pod핂캏헣힒맽쿦옪퓮힎쇮멑픒

    맣헪쿦핖쁢믾쁳
    • minAvailable짝maxUnavailable 퐃켦픊옪컲헣
    • ⚠ 핞짪헏핆disruption픦몋푾펞잚PDB 훎쿦많쇦즎옪 

    폖믾팘픎/PEF픦핳팮짪캫킪펞쁢PDB많힎힎힎팘픒쿦핖픚
    BETA

    View Slide

  36. apiVersion: policy/v1beta1
    kind: PodDisruptionBudget
    metadata:
    name: gitlab-pdb
    spec:
    minAvailable: 2
    selector:
    matchLabels:
    app: gitlab
    pod-disruption-budget.yml

    View Slide

  37. Pod Disruption Budget
    • 혾멂펞재쁢Pod핂캏헣힒맽쿦옪퓮힎쇮멑픒

    맣헪쿦핖쁢믾쁳
    • minAvailable짝maxUnavailable 퐃켦픊옪컲헣
    • ⚠ 핞짪헏핆disruption픦몋푾펞잚PDB 훎쿦많쇦즎옪 

    폖믾팘픎/PEF픦핳팮짪캫킪펞쁢PDB많힎힎힎팘픒쿦핖픚
    BETA

    View Slide

  38. Node Draining
    $ kubectl cordon NODE
    $ kubectl drain NODE
    훊펂힒Node읊큲흂핂쇦힎팘솒옫힎헣삲
    훊펂힒Node읊큲흂핂쇦힎팘솒옫힎헣몮 

    맏Pod읊칻헪삲

    View Slide

  39. $ kubectl cordon gke-my-cluster-my-pool-592cda94-2w25
    node "gke-my-cluster-my-pool-592cda94-2w25" cordoned
    $ kubectl describe gke-my-cluster-my-pool-592cda94-2w25
    Events:
    Type Reason Age From Message
    ---- ------ ---- ---- -------
    Normal NodeNotSchedulable 45s kubelet, ... Node status is now: NodeNotSchedulable
    kubectl cordon

    View Slide

  40. Google Kubernetes Engine픎Node쪎몋킪drain픒팚팒컪읺삲

    View Slide

  41. 핞솧Node펓믆엖핂슪빦Autoscale 칺푷킪Pod Disruption Budget픎쿦

    View Slide

  42. Service Catalog and Broker

    View Slide

  43. Pod
    Secret
    serviceaccount.json: ...
    Cloud SQL
    Instance
    Kubernetes(GKE) 짤펞컪
    쿦솧픊옪캫컿퍊

    View Slide

  44. $ gcloud iam service-accounts create gitlab \
    --display-name="GitLab Service Account"
    Service Account캫컿

    View Slide

  45. $ gcloud projects add-iam-policy-binding $PROJECT \
    --member="serviceAccount:$EMAIL"\
    --role="roles/cloudsql.client"
    뭚쭎펺

    View Slide

  46. $ gcloud iam service-accounts keys create \
    ./artifacts/serviceaccount.json \
    --iam-account $EMAIL

    View Slide

  47. $ kubectl create secret generic gitlab-config \
    ...
    --from-file=./artifacts/serviceaccount.json
    kubectl픒핂푷Secret캫컿
    $

    View Slide

  48. "Kubernetes뺂펞컪

    푆쭎컪찒큲펞샎컲헣픒쿦쁢펔픒밚?"

    View Slide

  49. Service CatalogBETA

    View Slide

  50. View Slide

  51. %

    View Slide

  52. Pod
    Secret
    serviceaccount.json: ...
    Cloud SQL
    Instance
    Kubernetes(GKE) 짤펞컪
    쿦솧픊옪캫컿퍊

    View Slide

  53. Cloud SQL
    Instance
    Service

    Account
    Service Instance
    Service Instance

    View Slide

  54. 핺힎풞쇦쁢Service Instance홓윦
    Service Account
    Cloud Spanner Cloud Pub/Sub
    Cloud SQL (MySQL)
    BigQuery Cloud BigTable Cloud Storage

    View Slide

  55. $ kubectl create clusterrolebinding cluster-admin-binding \
    --clusterrole=cluster-admin \
    --user=$(gcloud config get-value account)
    clusterrolebinding "cluster-admin-binding" created
    Service Catalog컲
    https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog

    View Slide

  56. View Slide

  57. $ sc install
    account: [email protected]
    project: shakr-openinfra-demo
    zone:
    generated service catalog deployment config in dir: /tmp/service-
    catalog544428136
    Service Catalog installed successfully.
    Service Catalog컲
    https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog

    View Slide

  58. $ sc add-gcp-broker
    using project: shakr-openinfra-demo
    enabling a GCP API: servicebroker.googleapis.com
    enabling a GCP API: bigtableadmin.googleapis.com
    enabling a GCP API: ml.googleapis.com
    ...
    The Service Broker has been added successfully.
    Service Catalog컲
    https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog

    View Slide

  59. $ gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member serviceAccount:$EMAIL \
    --role=roles/owner
    Service Catalog컲
    https://cloud.google.com/kubernetes-engine/docs/how-to/add-on/service-catalog/install-service-catalog

    View Slide

  60. $ kubectl -o "custom-
    columns=NAME:.spec.externalName,DESCRIPTION:.spec.description" \
    get clusterserviceclasses
    NAME DESCRIPTION
    cloud-spanner The first horizontally scalable...
    cloud-iam-service-account Specialized service which provisions...
    cloud-pubsub Ingest event streams from anywhere...
    cloud-sql-mysql A fully-managed MySQL database service
    bigquery A fast, highly scalable, cost-effective
    cloud-bigtable A high performance NoSQL database
    Service Catalog핆

    View Slide

  61. apiVersion: servicecatalog.k8s.io/v1beta1
    kind: ServiceInstance
    metadata:
    name: test-storage
    namespace: default
    spec:
    clusterServiceClassExternalName: cloud-storage
    clusterServicePlanExternalName: beta
    parameters:
    bucketId: shakr-openinfra-demo-test-storage
    location: US
    storageClass: STANDARD
    serviceinstance.yml

    View Slide

  62. $ kubectl apply -f serviceinstance.yml
    kubectl옪Service Instance캫컿

    View Slide

  63. View Slide

  64. &&&&

    View Slide

  65. apiVersion: servicecatalog.k8s.io/v1beta1
    kind: ServiceInstance
    metadata:
    name: test-storage
    namespace: default
    spec:
    clusterServiceClassExternalName: cloud-storage
    clusterServicePlanExternalName: beta
    parameters:
    bucketId: shakr-openinfra-demo-test-storage
    location: US
    storageClass: STANDARD
    serviceinstance.yml

    View Slide

  66. https://twitter.com/tenderlove/status/988887936128040960

    View Slide

  67. 콢픒칺푷펺Service Instance캫컿

    View Slide

  68. 콢픒칺푷펺Service Instance캫컿

    View Slide

  69. $ svcat provision test-storage \
    --class cloud-storage \
    --plan beta \
    --namespace default \
    --param bucketId=shakr-openinfra-demo-test-storage \
    --param location=US \
    --param storageClass=STANDARD
    svcat픊옪Service Instance캫컿

    View Slide

  70. Service Binding

    View Slide

  71. Cloud Storage
    Instance
    Service Instance
    Service Instance
    Service Binding
    Service Binding
    Secret
    privateKeyData: ...
    Service
    Account

    View Slide

  72. Cloud Storage
    Instance
    Service Instance Service Binding
    Secret
    privateKeyData: ...
    createServiceAccount: true

    View Slide

  73. $ svcat bind test-storage \
    --name test-storage-binding \
    --params-json \
    '{
    "serviceAccount": "test-storage-serviceaccount",
    "createServiceAccount": true,
    "roles": [
    "roles/storage.objectCreator",
    "roles/storage.objectViewer"
    ]
    }'
    Service Binding캫컿

    View Slide

  74. $ kubectl get secrets test-storage-binding
    NAME TYPE DATA AGE
    test-storage-binding Opaque 2 5m
    Service Account Secret핆

    View Slide

  75. spec:
    volumes:
    - name: test-storage-binding
    secret:
    secretName: test-storage-binding
    containers:
    - name: my-app
    image: shakr/my-app:latest
    volumeMounts:
    - name: binding
    mountPath: /mnt/binding
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
    value: /mnt/binding/privateKeyData
    - name: STORAGE_PROJECT
    valueFrom:
    secretKeyRef:
    name: user-storage-binding
    key: projectId
    - name: STORAGE_BUCKET
    valueFrom:
    secretKeyRef:
    name: user-storage-binding
    key: bucketId
    deployment.yml (pod spec)

    View Slide

  76. Service Catalog TL;DR
    • GKE(Kubernetes) 얺큲펞컪 GCP픦컪찒큲읊Service Instance옪
    캫컿펺짢옪칺푷쿦핖삲
    • Service Account JSON Key픦뽆픒

    먿헣힎팘팒솒쇪삲
    • ⚠ Service Instance 칻헪킪킲헪읺콚큲 GCS쩒 SQL핆큲큲
    솒

    칻헪쇦삖훊픦

    View Slide

  77. https://svc-cat.io

    View Slide

  78. Recap

    View Slide

  79. 쭒칾킪큲펞컪팖헣헏픊옪폖퍋핟펓픒쿦쿦핖픒밚

    View Slide

  80. CronJob
    • Cron syntax읊믆샎옪칺푷펺얺큲픦쭒칾핟펓픒묺솧쿦핖삲
    • ⚠ 콚쩖킲(At Least Once)픒쫂핳믾쌚줆펞펺빦펺얺쩖킲
    쇮캏펞샎샎찒많푢(de-duplication옪힏슿)

    View Slide

  81. "빦쁢GPU많핖쁢Node펞줂혾멂큲흂쇦펂퍊몮 

    pool-a NodePool펞큲흂쇦졂홙멮힎잚

    팖쇦펂솒멚캏뫎펔펂"

    View Slide

  82. Affinity
    • 쫂삲퓮펾몮삲퍟혾멂슲옪Pod핂펂쎉멚큲흂쇮힎헣픦쿦핖픚
    • 펺얺찒묞펾칾핞칺푷많쁳

    In, NotIn, Exists, DoesNotExist, Gt(>), Lt(<)
    • Node믾훎픎Pod믾훎픊옪힎헣많쁳
    BETA

    View Slide

  83. "짆켦읺팮읺핂켦픦훟삶펔핂

    Node읊펔팮몮탄픎섾펂쎉멚힎

    View Slide

  84. Pod Disruption Budget
    • 혾멂펞재쁢Pod핂캏헣힒맽쿦옪퓮힎쇮멑픒

    맣헪쿦핖쁢믾쁳
    • minAvailable짝maxUnavailable 퐃켦픊옪컲헣
    • ⚠ 핞짪헏핆disruption픦몋푾펞잚PDB 훎쿦많쇦즎옪 

    폖믾팘픎/PEF픦핳팮짪캫킪펞쁢PDB많힎힎힎팘픒쿦핖픚
    BETA

    View Slide

  85. "Kubernetes뺂펞컪

    푆쭎컪찒큲펞샎컲헣픒쿦쁢펔픒밚?"

    View Slide

  86. Service Catalog
    • GKE(Kubernetes) 얺큲펞컪 GCP픦컪찒큲읊Service Instance옪
    캫컿펺짢옪칺푷쿦핖삲
    • Service Account JSON Key픦뽆픒

    먿헣힎팘팒솒쇪삲
    • ⚠ Service Instance 칻헪킪킲헪읺콚큲 GCS쩒 SQL핆큲큲
    솒

    칻헪쇦삖훊픦

    View Slide

  87. CronJob
    Affinity
    Pod Disruption Budget
    Service Catalog

    View Slide

  88. https://twitter.com/_tr/status/1007619178222665730

    View Slide

  89. 맞칺삖삲 '

    View Slide

  90. GCP+GKE Deep Dive
    Minku Lee
    CTO Shakr
    Shakr펞컪쁳엳핖쁢펢힎삖펂읊졶킻삖삲
    careers.shakr.com

    View Slide