Working with Amazon EKS - Tips for Effective Workload -

2019/3/6開催のAmazon EKS 東京リージョンローンチ記念 AWS Container Meetupの資料です

Transcript

1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Fumihide Nario, Solution Architect Amazon Web Services Japan K.K. 2019
   3 6 Working with Amazon EKS - Tips for Effective Workload -
2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ! Amazon EKS  
      !  Tips 
3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    • Cluster Upgrade • Cost Optimization (Spot Instance) • CNI
4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster Upgrade
5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster Upgrade &,!/0( •  
    • )! • $ -.'*&% → "0+#  
6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster Upgrade - Version Kubernetes Version • 1.11 Kubernetes Patch Version / EKS Platform Version • 1.11.5 / eks.1 (EKS
   1.11) • 1.10 Kubernetes Patch Version / EKS Platform Version • 1.10.11 / eks.3 (CVE-2018-1002105) • 1.10.3 / eks.2 ( Admission Controllers  HPA) • 1.10.3 / eks.1 (EKS
   1.10)
7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster Upgrade – Control plane "# !#"$Master Nodes% • ""#  • Cluster  
   • AWS CLI   • aws eks update-cluster-version --name Your-EKS-Cluster --kubernetes-version 1.11 •    aws eks list-updates --name Your-EKS-Cluster •   aws eks describe-update --name Your-EKS-Cluster --update-id UUID
8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster Upgrade – Control plane +:1;859;:<Master Nodes=   • kube-proxy daemonset 3.-% kubectl patch daemonset kube-proxy -n kube-system \ -p '{"spec": {"template": {"spec": {"containers": [{"image": "602401143452.dkr.ecr.us-west- 2.amazonaws.com/eks/kube-proxy:v1.11.5","name":"kube-proxy"}]}}}}’ • CoreDNS ':,1;8 Kubernetes 1.11 
   $ EKS Cluster !04(81 DNS ! CoreDNS 1.11 "&.5*9;2  kube-dns ! CoreDNS %#! 2)76:1#  →   /,1
   /,1 https://docs.aws.amazon.com/ja_jp/eks/latest/userguide/update-cluster.html https://docs.aws.amazon.com/cli/latest/reference/eks/update-cluster-version.html
9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster Upgrade – Data plane )/(+-/.0Worker Nodes1" • 
   3" " #& 1.  Worker Node Group ! 2. " Worker Node Group & → */',."Node &$%2" &
10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster Upgrade – Data plane FOCGMONPWorker NodesQ2 # • ,% Worker Node Group 1H<?MO@JN 1.  Version 2 Nodes : 5 Worker Node Group : 2.  Worker Node Group (/)86&B=IKE;?LOG: 3. aws-auth configmap :", EKS Cluster 2RBAC: ,  Worker Node Group : 4. Cluster Autoscaler :,.%8 13 replicas=0 1 5.  Worker Node Group 1 pod ( +90%6&Pkubectl taint nodesQ 6. kube-dns Por CoreDNSQ ( 2 pods 
    %.%8'$ %.0*94 replicas=2 1 7.  Worker Node Group 2 Nodes 1,. drain 2 8. 2. 2!$ CloudFormation / Worker Node Group 2ACD>2! 9. aws-auth configmap :", EKS Cluster '7 Worker Node Group :! 10. Cluster Autoscaler :,.%8 13 replicas=1 1 11. kube-dns :,.%-7 replicas=1 1 https://docs.aws.amazon.com/eks/latest/userguide/migrate-stack.html
11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster Upgrade – Data plane 38156879Worker Nodes:'  •  ' Worker Node Group 0 1. kube-dns 9or CoreDNS:  2 pods 
    #.#% /) replicas=2 & 2. Cluster Autoscaler 0!#.&( replicas=0 & 3. CloudFormation ' Update Stack +-'275684$ AMI 0!#
    NodeAutoScalingGroupMaxSize (' +- %$*; % 0 4. Cluster Autoscaler 0!#.&( replicas=1 & 5. kube-dns 0!#", replicas=1 & https://docs.aws.amazon.com/eks/latest/userguide/update-stack.html
12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance)
13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance) '/),#2'(2' • Amazon EC2%1.&*"! $2+02-  90% • '/),#2'(2' 
    
14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance) DJFH?MDEMD48 • DJFH?MDEMD8 AWS 6912 -<;,4)':$/78 2 6 • Amazon EC27*AKICG>) 3*5+5104* • ( )LB@DH.0 (%
     &)=DJFH?MDEMD )
    104*  "!"   #          
15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance) #*' 1.   $!*' %$!*') • Cluster Autoscaler  ( Daemonset ) https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler 2. Data Plane (Worker Nodes)  Spot Instance  "$%  • Auto Scaling Group ( AWS CloudFormation ) 3. Spot Instance 
      Node &() • kube-spot-termination-notice-handler  ( Daemonset – Spot Instance  ) https://github.com/kube-aws/kube-spot-termination-notice-handler
16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance) https://aws.amazon.com/jp/blogs/compute/run-your-kubernetes-workloads-on-amazon-ec2-spot-instances-with-amazon-eks/
17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance)   awslab  ec2-spot-labs   1. Provision the worker nodes • CloudFormation Template   https://github.com/awslabs/ec2-spot-labs/blob/master/ec2-spot-eks- solution/provision-worker-nodes/amazon-eks-nodegroup-with-spot.yaml • Template ! " UserData  /etc/eks/bootstrap.sh ${ClusterName} ${SpotBootstrapArguments} ${SpotBootstrapArguments}  
      --kubelet-extra-args --node-labels=lifecycle=Ec2Spot --kubelet-extra-args --node-labels=lifecycle=OnDemand
18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance) 2. Deploying Cluster Autoscaler • manifest
     Node  Auto Scaling Group    kubectl create -f cluster-autoscaler-ds.yaml https://github.com/awslabs/ec2-spot-labs/blob/master/ec2-spot-eks-solution/cluster- autoscaler/cluster-autoscaler-ds.yaml
19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance) 3. Deploying PDB • PDB ( Pod Disruption Budget )  drain 
     Running pod  kubectl create -f cluster-autoscaler-pdb.yaml https://github.com/awslabs/ec2-spot-labs/blob/master/ec2-spot-eks- solution/cluster-autoscaler/cluster-autoscaler-pdb.yaml
20. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance) 4. Deploy the Spot Instance interrupt handler • kube-spot-termination-notice-handler  nodeSelector lifecycle: Ec2Spot  Node  
     kubectl apply -f spot-interrupt-handler.yaml https://github.com/awslabs/ec2-spot-labs/blob/master/ec2-spot-eks-solution/spot- termination-handler/deploy-k8-pod/spot-interrupt-handler.yaml
21. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance) • spot-termination-handler entrypoint.sh  < = POLL_INTERVAL=${POLL_INTERVAL:-5} NOTICE_URL=${NOTICE_URL:-http://169.254.169.254/latest/meta-data/spot/termination-time} while http_status=$(curl -o /dev/null -w '%{http_code}' -sL ${NOTICE_URL}); [ ${http_status} -ne 200 ]; do verbose && echo $(date): ${http_status} sleep ${POLL_INTERVAL} done kubectl drain ${NODE_NAME} --force --ignore-daemonsets
     2  ':0;58 5 1(2) # drain &%"$ → *:34'7;,+'. 69/- !
22. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CNI
23. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CNI • Container Network Interface •    
     • Plugin   • Amazon VPC CNI plugin for Kubernetes •   kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2 •    kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni- k8s/master/config/v1.3/aws-k8s-cni.yaml
24. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon VPC CNI plugin for Kubernetes •  • !&1H8;'NAT$"&1H8;# • !&=I:'NAT$"!&1H8;# • 1H8;!)IP,:E5' (*!)&# IP,:E5 → Kubernetes Pod ' VPC <79GI/
    # IP ,:E5+ → A<I4:2I>5#&3IBE5$  → VPC?FIF0%!IP9D?-7/% ) +.C@6C
25. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon VPC CNI plugin for Kubernetes
26. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon VPC CNI plugin for Kubernetes • L-IPAM (Local IP Address Manager) • #   (%IP!')$")&
     Pod   
27. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon VPC CNI plugin for Kubernetes • Pod +  *IP5=A9,+ • CPU Memory ,@:C9+#!&'. Pod # ($*" (Pending) • kubectl get events 42) failed to assign an IP address to container #2 → 6B9;B9;6>+ 1'032IP,#!2 F  DENI * ENI,IPE - 1D?9<E https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI • 7C@B8+ • Pod  IP5=A9-
     Pod + %3*" • Github +!2 data_store.go +/2) -,#%3'"2 addressCoolingPeriod = 30 * time.Second
28. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon VPC CNI plugin for Kubernetes • CNI Metrics Helper • CNI 
    #.+0&( CloudWatch  • &/()'-2+! ENI • -*, "ENI • -*, "!IP$,1( • IP$,1(  • ipamD%/2 https://aws.amazon.com/jp/blogs/opensource/cni-metrics-helper/
29. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CNI Custom Networking •
    VPC*+2>7KMB59NRH0;GEAC3%' :R= •  %) 2;GEAC**"2IP4DO=-,%)%/(' • >7KMB5-!1+2;GEAC0>7KMB59NRH3%' → CNI6=@IEACPR8 •  • CNI Plugin Version 1.2.1  • AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG 3  &2 • ENIConfig 6=@IM?R=3&2 • 6=@IEACPR83%)FRD,4FBR<LQ3$2 %#.D7KJQC3  https://docs.aws.amazon.com/ja_jp/eks/latest/userguide/cni-custom-network.html
30. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ! • Roadmap EKS$ Container Service " Roadmap!# github  https://github.com/aws/containers-roadmap • Amazon EKS Workshop X-Ray '-13+2)Elasticsearch5Fluentd5Kibana' LoggingCI/CD %Monitoring  
    ! & %! https://eksworkshop.com/ *2,.0(/'4