© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Fumihide Nario, Solution Architect Amazon Web Services Japan K.K. 2019 3 6 Working with Amazon EKS - Tips for Effective Workload -

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ! Amazon EKS ! Tips

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • Cluster Upgrade • Cost Optimization (Spot Instance) • CNI

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster Upgrade

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster Upgrade &,!/0( • • )! • $ -.'*&% → "0+#

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster Upgrade - Version Kubernetes Version • 1.11 Kubernetes Patch Version / EKS Platform Version • 1.11.5 / eks.1 (EKS 1.11) • 1.10 Kubernetes Patch Version / EKS Platform Version • 1.10.11 / eks.3 (CVE-2018-1002105) • 1.10.3 / eks.2 ( Admission Controllers HPA) • 1.10.3 / eks.1 (EKS 1.10)

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster Upgrade – Control plane "# !#"$Master Nodes% • ""# • Cluster • AWS CLI • aws eks update-cluster-version --name Your-EKS-Cluster --kubernetes-version 1.11 • aws eks list-updates --name Your-EKS-Cluster • aws eks describe-update --name Your-EKS-Cluster --update-id UUID

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster Upgrade – Control plane +:1;859;:

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster Upgrade – Data plane )/(+-/.0Worker Nodes1" • 3" " #& 1. Worker Node Group ! 2. " Worker Node Group & → */',."Node &$%2" &

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster Upgrade – Data plane FOCGMONPWorker NodesQ2 # • ,% Worker Node Group 1H2! 9. aws-auth configmap :", EKS Cluster '7 Worker Node Group :! 10. Cluster Autoscaler :,.%8 13 replicas=1 1 11. kube-dns :,.%-7 replicas=1 1 https://docs.aws.amazon.com/eks/latest/userguide/migrate-stack.html

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster Upgrade – Data plane 38156879Worker Nodes:' • ' Worker Node Group 0 1. kube-dns 9or CoreDNS: 2 pods #.#% /) replicas=2 & 2. Cluster Autoscaler 0!#.&( replicas=0 & 3. CloudFormation ' Update Stack +-'275684$ AMI 0!# NodeAutoScalingGroupMaxSize (' +-%$*; % 0 4. Cluster Autoscaler 0!#.&( replicas=1 & 5. kube-dns 0!#", replicas=1 & https://docs.aws.amazon.com/eks/latest/userguide/update-stack.html

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance)

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance) '/),#2'(2' • Amazon EC2%1.&*"! $2+02- 90% • '/),#2'(2'

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance) DJFH?MDEMD48 • DJFH?MDEMD8 AWS 6912 -<;,4)':$/78 2 6 • Amazon EC27*AKICG>)3*5+5104* • ()[email protected] (% &)=DJFH?MDEMD )104* "!" #

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance) #*' 1. $!*' %$!*') • Cluster Autoscaler ( Daemonset ) https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler 2. Data Plane (Worker Nodes) Spot Instance "$% • Auto Scaling Group ( AWS CloudFormation ) 3. Spot Instance Node &() • kube-spot-termination-notice-handler ( Daemonset – Spot Instance ) https://github.com/kube-aws/kube-spot-termination-notice-handler

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance) https://aws.amazon.com/jp/blogs/compute/run-your-kubernetes-workloads-on-amazon-ec2-spot-instances-with-amazon-eks/

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance) awslab ec2-spot-labs 1. Provision the worker nodes • CloudFormation Template https://github.com/awslabs/ec2-spot-labs/blob/master/ec2-spot-eks- solution/provision-worker-nodes/amazon-eks-nodegroup-with-spot.yaml • Template ! " UserData /etc/eks/bootstrap.sh ${ClusterName} ${SpotBootstrapArguments} ${SpotBootstrapArguments} --kubelet-extra-args --node-labels=lifecycle=Ec2Spot --kubelet-extra-args --node-labels=lifecycle=OnDemand

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance) 2. Deploying Cluster Autoscaler • manifest Node Auto Scaling Group kubectl create -f cluster-autoscaler-ds.yaml https://github.com/awslabs/ec2-spot-labs/blob/master/ec2-spot-eks-solution/cluster- autoscaler/cluster-autoscaler-ds.yaml

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance) 3. Deploying PDB • PDB ( Pod Disruption Budget ) drain Running pod kubectl create -f cluster-autoscaler-pdb.yaml https://github.com/awslabs/ec2-spot-labs/blob/master/ec2-spot-eks- solution/cluster-autoscaler/cluster-autoscaler-pdb.yaml

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance) 4. Deploy the Spot Instance interrupt handler • kube-spot-termination-notice-handler nodeSelector lifecycle: Ec2Spot Node kubectl apply -f spot-interrupt-handler.yaml https://github.com/awslabs/ec2-spot-labs/blob/master/ec2-spot-eks-solution/spot- termination-handler/deploy-k8-pod/spot-interrupt-handler.yaml

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimization (Spot Instance) • spot-termination-handler entrypoint.sh <= POLL_INTERVAL=${POLL_INTERVAL:-5} NOTICE_URL=${NOTICE_URL:-http://169.254.169.254/latest/meta-data/spot/termination-time} while http_status=$(curl -o /dev/null -w '%{http_code}' -sL ${NOTICE_URL}); [ ${http_status} -ne 200 ]; do verbose && echo $(date): ${http_status} sleep ${POLL_INTERVAL} done kubectl drain ${NODE_NAME} --force --ignore-daemonsets 2 ':0;58 5 1(2) # drain &%"$ → *:34'7;,+'.69/- !

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CNI

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CNI • Container Network Interface • • Plugin • Amazon VPC CNI plugin for Kubernetes • kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2 • kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni- k8s/master/config/v1.3/aws-k8s-cni.yaml

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon VPC CNI plugin for Kubernetes • • !&1H8;'NAT$"&1H8;# • !&=I:'NAT$"!&1H8;# • 1H8;!)IP,:E5'(*!) IP,:E5 → Kubernetes Pod ' VPC <79GI/# IP ,:E5+ → A5#&3IBE5$ → VPC?FIF0%!IP9D?-7/% ) +.C@6C

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon VPC CNI plugin for Kubernetes

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon VPC CNI plugin for Kubernetes • L-IPAM (Local IP Address Manager) • # (%IP!')$")& Pod

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon VPC CNI plugin for Kubernetes • Pod + *IP5=A9,+ • CPU Memory ,@:C9+#!&'. Pod # ($*" (Pending) • kubectl get events 42) failed to assign an IP address to container #2 → 6B9;B9;6>+ 1'032IP,#!2 F DENI * ENI,IPE - 1D?9

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon VPC CNI plugin for Kubernetes • CNI Metrics Helper • CNI #.+0&( CloudWatch • &/()'-2+! ENI • -*, "ENI • -*, "!IP$,1( • IP$,1( • ipamD%/2 https://aws.amazon.com/jp/blogs/opensource/cni-metrics-helper/

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CNI Custom Networking • VPC*+2>7KMB59NRH0;GEAC3%' :R= • %) 2;GEAC**"2IP4DO=-,%)%/(' • >7KMB5-!1+2;GEAC0>7KMB59NRH3%' → CNI6=@IEACPR8 • • CNI Plugin Version 1.2.1 • AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG 3 &2 • ENIConfig 6=@IM?R=3&2 • 6=@IEACPR83%)FRD,4FBR

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ! • Roadmap EKS$ Container Service " Roadmap!# github https://github.com/aws/containers-roadmap • Amazon EKS Workshop X-Ray '-13+2)Elasticsearch5Fluentd5Kibana' LoggingCI/CD %Monitoring ! & %! https://eksworkshop.com/ *2,.0(/'4