Upgrade to Pro — share decks privately, control downloads, hide ads and more …

1人運用を支えるAmazon EKSノウハウ / Amazon EKS Know-How

1人運用を支えるAmazon EKSノウハウ / Amazon EKS Know-How

Hiroshi Oda

July 28, 2020
Tweet

More Decks by Hiroshi Oda

Other Decks in Technology

Transcript

  1. … spec: containers: - name: myapp image: hi1280/myapp:0.0.1 ports: -

    containerPort: 3000 livenessProbe: httpGet: path: /healthz port: 3000 readinessProbe: httpGet: path: /healthz port: 3000 1PEͷηϧϑώʔϦϯάΛ༗ޮʹ͢Δ 1SPCFΛઃఆ͢Δ LVCFMFUʹΑΔ؂ࢹػೳͰίϯςφ͕ਖ਼ৗʹಈ͍͍ͯΔ͔֬ೝ͢Δػೳ 20 1PE $POUBJOFS LVCFMFU MJWFOFTT1SPCF SFBEJOFTT1SPCF MJWFOFTT1SPCFɿίϯςφ͕ੜ͖͍ͯΔ͔ΛνΣοΫ͢Δ SFBEJOFTT1SPCFɿίϯςφ͕Ԡ౴Մೳ͔Ͳ͏͔ΛνΣοΫ͢Δ ,VCFSOFUFTϚχϑΣετͷDPOUBJOFS෦෼ൈਮ ࢀߟ https://kubernetes.io/ja/docs/concepts/workloads/pods/pod-lifecycle/
  2. ෛՙʹԠͯࣗ͡ಈతʹ1PE਺Λௐ੔͢ΔͨΊʹɺ)PSJ[POUBM1PE"VUPTDBMFS )1" Λར༻͢Δ w )1"Λ࢖͏͜ͱͰɺ$16ɺϝϞϦɺ$VTUPN.FUSJDT ಠࣗͷNFUSJDT஋ ͱ͍ͬͨ஋ʹج͍ͮͯɺ1PEΛΦʔτε έʔϧ͢Δ͜ͱ͕ՄೳʹͳΔ w "84Ͱ͸ɺ$MPVE8BUDI.FUSJDTΛ)1"ͷNFUSJDTͱͯ͠࢖͏͜ͱ͕Ͱ͖Δɻྫ͑͹ɺ424

    2VFVF ͷϝοηʔ δ਺͕࢖͑ΔͷͰɺϝοηʔδ਺ͷ૿ՃΛ֬ೝͯ͠8PSLFSΛεέʔϧͯ͠ෛՙʹରͯ͠ͷࣄલ४උ͕Ͱ͖Δ 1PEΛΦʔτεέʔϧ͢Δ 21 ࢀߟ https://aws.amazon.com/jp/blogs/compute/scaling-kubernetes-deployments-with-amazon-cloudwatch-metrics/
  3. 1PE಺ͷ$POUBJOFSʹϦιʔε $16 ͷཁٻ஋Λઃఆ͢Δ )1"ͷར༻ํ๏ 22 %FQMPZNFOU 3FQMJDB4FU 1PE 1PE 1PE

    … spec: containers: - name: myapp image: hi1280/myapp:0.0.1 ports: - containerPort: 3000 resources: requests: cpu: 200m $16N $16N ,VCFSOFUFTϚχϑΣετͷDPOUBJOFS෦෼ൈਮ ϛϦ$16Λཁٻ͢Δ $16N
  4. )1"ͷར༻ํ๏ 23 apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: myapp spec:

    scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: myapp minReplicas: 1 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 50 )1"ͷZBNM ໨ඪ஋ͱͳΔ$16࢖༻཰Λࢦఆ͢Δ $16ͷར༻Λௐ੔͢ΔͨΊͷ໨ඪ஋Λઃఆ͢Δ %FQMPZNFOU 3FQMJDB4FU 1PE 1PE 1PE )1" .FUSJDT4FSWFS $PMMFDUNFUSJDT 1PE $16࢖༻཰ʹԠͯ͡1PE਺͕૿ݮ $PMMFDUNFUSJDT ࣮ࡍͷ$16࢖༻཰ΛܭଌՄೳʹ͢Δ ͨΊʹσϓϩΠ͢Δ
  5. 26 … nodeGroups: - name: ng1-public instanceType: m5.xlarge desiredCapacity: 10

    targetGroupARNs: - arn:aws:elasticloadbalancing:eu- west-2:01234567890:targetgroup/target- group-1/abcdef0123456789 iam: withAddonPolicies: autoScaler: true FLTDUMͷઃఆϑΝΠϧൈਮ $MVTUFS"VUPTDBMFS͕"VUP4DBMJOH(SPVQΛ࢖͑ ΔΑ͏ʹ*".ϙϦγʔΛઃఆ͢Δ <&,4>$MVTUFS"VUPTDBMFSͷར༻ํ๏
  6. 27 FLTDUMͷઃఆϑΝΠϧൈਮ $MVTUFS"VUPTDBMFS͕"VUP4DBMJOH(SPVQΛࣗಈ ݕग़Ͱ͖ΔΑ͏ʹ"VUP4DBMJOH(SPVQʹλά෇͚ ͢Δ w FLTDUMͷ৔߹ɺ"VUP4DBMJOH(SPVQ࡞੒࣌ʹࣗ ಈతʹλά෇͚͞ΕΔ <&,4>$MVTUFS"VUPTDBMFSͷར༻ํ๏ …

    nodeGroups: - name: ng1-public instanceType: m5.xlarge desiredCapacity: 10 targetGroupARNs: - arn:aws:elasticloadbalancing:eu- west-2:01234567890:targetgroup/target- group-1/abcdef0123456789 iam: withAddonPolicies: autoScaler: true
  7. … nodeGroups: - name: ng1-public instanceType: m5.xlarge desiredCapacity: 10 targetGroupARNs:

    - arn:aws:elasticloadbalancing:eu- west-2:01234567890:targetgroup/target- group-1/abcdef0123456789 iam: withAddonPolicies: autoScaler: true 28 FLTDUMͷઃఆϑΝΠϧൈਮ "VUP4DBMJOH(SPVQͱ-PBE#BMBODFSΛؔ࿈෇͚Δ w -PBE#BMBODFSͷUBSHFUHSPVQΛࢦఆ͢Δ ˞"QQMJDBUJPO-PBE#BMBODFSͷ৔߹ <&,4>$MVTUFS"VUPTDBMFSͷར༻ํ๏
  8. 1SPNFUIFVTͷઃఆྫ 31 groups: - name: container rules: - alert: ContainerMemoryAlert

    expr: (sum(container_memory_working_set_bytes) by (container) / sum(kube_pod_container_resource_limits_memory_bytes) by (container)) * 100 > 90 for: 5m "MFSU3VMFઃఆྫͷൈਮ global: slack_api_url: '<slack_webhook_url>' route: receiver: 'slack-notifications' receivers: - name: 'slack-notifications' slack_configs: {} "MFSUNBOBHFSઃఆྫͷൈਮ
  9. ,VCFSOFUFTΛ؆୯ʹར༻͢Δ࢓૊Έͮ͘Γ 36 ؆୯ʹ,VCFSOFUFTΛར༻Ͱ͖Δ࢓૊ΈΛ࡞Δ͜ͱͰɺ,VCFSOFUFT͕෼͔Βͳͯ͘΋ΞϓϦ։ൃऀʹ,VCFSOFUFTͷ ӡ༻ʹڠྗͯ͠΋Β͏ ,VCFSOFUFT ӡ༻ऀ ,VCFSOFUFT ར༻ऀ ΞϓϦ։ൃऀ ,VCFSOFUFTΫϥελ

    LVCFDUM LVCFDUM࣮ߦ؀ڥ ΞϓϦ։ൃऀ͕සൟʹߦ͏ૢ࡞ΛίϚϯυͱ͠ ͯఏڙ͢Δ w ίϯςφͷ؀ڥม਺Λมߋ͢Δ w ίϯςφͷΠϝʔδΛมߋ͢Δ গͳ͍ख਺Ͱ,VCFSOFUFTΛૢ࡞Ͱ͖ΔΑ͏ʹ ͢Δ LVCFDUM ίϚϯυ
  10. ։ൃऀ޲͚ͷ؀ڥͮ͘Γ 37 LVCFDUM࣮ߦ؀ڥ͸Πϛϡʔλϒϧͳ؀ڥΛ༻ҙ͢Δ w $PEF#VJMEΛ࢖༻֤ͯ͠ίϚϯυʹରԠͨ͠εΫϦϓτΛ࣮ߦ͢Δ εΫϦϓτ಺Ͱ͸LVTUPNJ[FΛར༻͢Δ w ։ൃ؀ڥ΍ຊ൪؀ڥຖͷࠩ෼ͷLVCFSOFUFTϚχϑΣετΛ༻ҙͯ͠ɺࠩ ෼͚͕ͩมߋ͞ΕͨϚχϑΣετΛ࡞Δ͜ͱ͕Ͱ͖Δ w

    ίϯςφͷ؀ڥม਺ͷมߋ΍ΠϝʔδมߋʹରԠͨ͠ػೳ͕͋Δʢޙड़ʣ w LVCFDUMʹಉࠝ͞Ε͍ͯΔLVTUPNJ[F͸ݹ͍ͷͰ࠷৽൛Λར༻͢Δ ,VCFSOFUFT ར༻ऀ ΞϓϦ։ൃऀ ,VCFSOFUFT Ϋϥελ LVCFDUM࣮ߦ؀ڥ LVCFDUM ίϚϯυ ࢀߟ https://github.com/kubernetes-sigs/kustomize
  11. ίϯςφͷ؀ڥม਺Λมߋ͢Δ 38 LVTUPNJ[FͷDPOpH.BQ(FOFSBUPS΍TFDSFU(FOFSBUPSΛར༻͢Δͱɺ؀ڥ ม਺ͷมߋΛͨ͠৔߹ʹEFQMPZNFOUʹมߋ͕൓ө͞ΕΔ ී௨ʹDPOpH.BQ΍TFDSFUΛ࡞Δͱ؀ڥม਺ͷมߋ͸൓ө͞Εͳ͍ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization commonLabels:

    app: nginx resources: - deployment.yaml configMapGenerator: - envs: - config.env name: config secretGenerator: - envs: - secret.env name: secret type: Opaque LVTUPNJ[BUJPOZBNM EFQMPZNFOUZBNM apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: replicas: 4 template: spec: containers: - name: nginx envFrom: - configMapRef: name: config - secretRef: name: secret ,VCFSOFUFT ར༻ऀ ΞϓϦ։ൃऀ ,VCFSOFUFT Ϋϥελ LVCFDUM࣮ߦ؀ڥ LVCFDUM ίϚϯυ
  12. ίϯςφͷΠϝʔδΛมߋ͢Δ 39 $ IMAGE=nginx $ VERSION=1.12 $ kustomize edit set

    image $IMAGE:$VERSION $ kustomize build . | kubectl apply -f - ˞࣮ࡍ͸ύϥϝʔλͰ*."(&΍7&34*0/͕౉͞ΕΔ૝ఆ LVTUPNJ[FFEJUίϚϯυͰ1PEͷJNBHFUBH෦෼Λมߋ͢Δ LVTUPNJ[BUJPOZBNMʹJNBHFUBH෦෼ͷࠩ෼৘ใ͕௥ه͞Εͯɺ LVTUPNJ[FCVJMEͰࠩ෼ΛؚΜͩશମͷϚχϑΣετ͕࡞ΒΕΔ apiVersion: kustomize.config.k8s.io/ v1beta1 kind: Kustomization … images: - name: nginx newTag: "1.12" Πϝʔδͷόʔδϣϯ ͕มߋ͞ΕΔ LVTUPNJ[BUJPOZBNMൈਮ 4IFMM ,VCFSOFUFT ར༻ऀ ΞϓϦ։ൃऀ ,VCFSOFUFT Ϋϥελ LVCFDUM࣮ߦ؀ڥ LVCFDUM ίϚϯυ
  13. ؀ڥຖʹΫϥελΛ༻ҙ͢Δ ,VCFSOFUFTΛ4*.1-&ʹͰ͖Δ w Ϋϥελ಺ͰͲͷΑ͏ʹ؀ڥΛ෼཭͢Δ͔ͱ͍ ͏ߟྀΛ͢Δඞཁ͕ͳ͘ͳΔ ΫϥελΛ࡞Δͱ͖ͷϙΠϯτ w αϒωοτͷαΠζ͸े෼ʹߟྀ͢Δɻ*1Ξυ Ϩε਺ͷ੍ݶʹΑͬͯɺ1PE͕ىಈͰ͖ͳ͍͜ ͱ͕͋Δ

    w ؀ڥຖʹαϒωοτΛ෼͚Δ 42 "84 1SPEVDUJPO Ϋϥελ 71$ 4UBHJOH Ϋϥελ ࢀߟ https://docs.aws.amazon.com/ja_jp/eks/latest/userguide/pod-networking.html 4UBHJOH༻ αϒωοτ 1SPEVDUJPO༻ αϒωοτ
  14. <&,4>ΫϥελͷΞοϓσʔτํ๏ 45 w ΫϥελͷΞοϓσʔτΛςετ͢Δ༻ʹΫϥελΛผͰ༻ҙ͢Δ w 7FMFSPΛ࢖ͬͯ4UBHJOH؀ڥ͔ΒϦετΞ͢ΔͱָʹΫϥελΛ࡞੒Ͱ͖Δ w &,4ͷެࣜυΩϡϝϯτʹ฿͏ w LVCFQSPYZͳͲͷΞυΦϯͷΞοϓσʔτखॱ͕ৄࡉʹهࡌ͞Ε͍ͯΔ

    w ,VCFSOFUFTʹΞοϓάϨʔυʹ͢ΔͨΊͷલఏ৚݅ͱ͍߲ͬͨ໨ͳͲɺΞοϓσʔτʹରͯ͠ͷαϙʔ τ͕ͳ͞Ε͍ͯΔ ࢀߟ https://velero.io/docs/main/ https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html
  15. <&,4>/PEFͷΞοϓσʔτํ๏ 46 ˞4FMGNBOBHFEXPSLFSOPEFFLTDUMͷ৔߹ ࣄલ४උ w 1PE%JTSVQUJPO#VEHFUΛద༻͢Δ w ϦΫΤετड෇Մೳͳ1PE͕ͳ͍ͱ͍͏ঢ়ଶ ʹͳΒͳ͍Α͏ʹ1PE਺Λอূͯ͠/PEFΛఀ ࢭ͢Δ

    apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: nginx-pdb spec: minAvailable: 50% selector: matchLabels: app: nginx 1PE%JTSVQUJPO#VEHFUͷZBNM ˢͷ1PE਺Λอূ͢Δ
  16. <&,4>/PEFͷΞοϓσʔτํ๏ खॱ w $MVTUFS"VUPTDBMFSΛఀࢭ͢Δ w ৽όʔδϣϯͷ/PEF(SPVQΛ࡞੒͢Δ w چόʔδϣϯͷ/PEF(SPVQΛ࡟আ͢Δ w $MVTUFS"VUPTDBMFSΛ։࢝͢Δ

    47 $ kubectl scale deployments/cluster-autoscaler --replicas=0 -n kube-system $ kubectl scale deployments/cluster-autoscaler --replicas=1 -n kube-system $ eksctl create nodegroup --config-file=<path> $ eksctl delete nodegroup --cluster <cluster> --name <ng-name>
  17. 1SPNFUIFVT (SBGBOBʹΑΔ؂ࢹӡ༻ 53 1SPNFUIFVT (SBGBOBͷࣗલӡ༻͕πϥ͘ͳ͖ͬͯͨ w .FUSJDTΛ௕ظؒอ࣋࢝͠Ίͨͱ͜Ζɺ(SBGBOBΛݟΔͨͼʹ1SPNFUIFVTͷϝϞϦ࢖༻཰͕ٸʹ૿Ճ͢Δ w 1SPNFUIFVTʹ͸1FSTJTUFOU7PMVNF &#4

    Λ࢖༻͍ͯ͠ΔͨΊɺ";ݻఆʹͳͬͯ͠·͍ɺ";Λ௒͑ͨ৑௕Խ͕Ͱ ͖ͳ͍ 1SPNFUIFVT (SBGBOBͰ&"4:ʹ؂ࢹ͢Δͱ͜Ζ͔Βن໛֦େʹ൐ͬͯࠜຊతͳݟ௚͕͠ඞཁʹͳ͖ͬͯͨɻ࣮֬ʹ ,VCFSOFUFT؀ڥͷϝτϦΫεΛऩू͍ͨ͠ͱ͍͏ͷ͕ཁٻͰ͋Δ 1SPNFUIVFT (SBGBOBͰ͸-POHUFSNTUPSBHFΛར༻͢Δͱ͍ͬͨํ๏͕͋Δ͕ɺࣗલӡ༻͸΍͸Γݫ͍͠ %BUBEPH΍$MPVE8BUDI$POUBJOFS*OTJHIUTͱ͍ͬͨϚωʔδυͳ؂ࢹαʔϏε΁ͷҠߦ͕ྑ͍͔΋͠Εͳ͍