© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Christoph Kassen Solutions Architect - AWS @christoph_k Kubernetes Patterns on AWS

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Customer use cases Microservices PaaS Platform-as-a-Service Enterprise App Migration Machine Learning

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Amazon EKS

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Amazon EKS Architecture https://00000000000000000000000000000000.xyz.eu-central-1.eks.amazonaws.com EKS workers kubectl AZ 1 AZ 2 AZ 3 Your AWS account VPC Amazon EKS

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Multiple Availability Zones VPC Amazon EKS Architecture Highly available and single tenant infrastructure Multiple Availability Zones ASG with fixed size Fronted by a LoadBalancer NLB Etcd API Servers

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Open source and EKS AWS VPC CNI Plugin AWS IAM Authenticator Amazon EKS AMI Key components of EKS are open source EKS runs 100% upstream Kubernetes /kubernetes /kubernetes/autoscaler /aws-labs/aws-service-operator /weaveworks/eksctl EBS, EFS, FSX CSI drivers kops Team contributes to or manages 20+ OSS projects

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Transparent roadmap https://github.com/aws/containers-roadmap/

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k CLI to provision EKS clusters - Manages worker node groups - Configures k8s CLI https://eksctl.io – Official open-source CLI tool for EKS eksctl create cluster --name=eks-patterns --nodes=4 --region=eu-central-1

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k eksctl customizations kubeletExtraConfig: kubeReserved: cpu: "300m" memory: "300Mi" ephemeral-storage: "1Gi" kubeReservedCgroup: "/kube- reserved" systemReserved: cpu: "300m" memory: "300Mi" ephemeral-storage: "1Gi" evictionHard: memory.available: "200Mi" nodefs.available: "10%" YAML based configuration kubelet parameters Resource reservations Labels & Tains Custom VPC setups Autoscaling Multiple Node Groups Spot & GPU node support Windows worker IAM Policies gitops

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Major version upgrades Components AWS EKS CNI coreDNS kube-proxy EKS Control Plane Worker Nodes

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Worker Node Management

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k • Amazon $ aws ssm get-parameter \ --name /aws/service/eks/optimized-ami/1.14/amazon-linux-2/recommended/image_id \ --region eu-central-1 --query Parameter.Value --output text Amazon https://github.com/awslabs/amazon-eks-ami • https://cloud-images.ubuntu.com/aws-eks/ Amazon EKS worker nodes

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Set resource requirements for each Pod

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Worker Node operations • Auto-Scaling • k8s Cluster Autoscaler • AWS Auto Scaling with custom Cloudwatch Metrics • Node Draining • Drain node before termination • 2-minute warning • Handlers • Listen for CloudWatch Events - https://github.com/aws-samples/amazon-k8s-node-drainer • Run handler on clusters - https://github.com/kube-aws/kube-spot-termination-notice-handler

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Autoscaling operations • Use Auto Scaling groups per Availability Zone • EBS Volumes are zonal, prevent scheduling issues • AZ Rebalancing • Balance nodes equally across zones • Automatic Node Draining via Lifecycle-Hook • Disable AZRebalance • Scaling with Resource Reservations • Enable faster or predictive scaling • Anomaly detection

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Export resource reservations

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Running EKS with Spot Instances • Utilize Spot capacity to reduce instance spend • Dev/Test clusters • Ephemeral clusters • Auto Scaling Groups • Combine Spot & On-Demand • Mix instance types with care • Switch to spot via • LaunchTemplate • eksctl

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k eksctl – Spot configuration apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: eks-patterns-demo region: eu-central-1 nodeGroups: - name: ng-1 minSize: 4 maxSize: 4 instancesDistribution: maxPrice: 0.5 instanceTypes: [“m5d.large", “m5d.xlarge"] onDemandBaseCapacity: 0 onDemandPercentageAboveBaseCapacity: 0 spotInstancePools: 2

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Security

Shared responsibility for Amazon EKS CUSTOMER IAM APPLICATION POD AWS ENDPOINTS AWS IAM FOUNDATION SERVICES COMPUTE STORAGE DATABASES NETWORKING AWS GLOBAL INFRASTRACTURE REGIONS AVAILABILITY ZONES EDGE LOCATIONS MANAGED BY AWS CUSTOMERS MANAGED BY AWS KUBELET RBAC APP HOST AWS IAM CONTAINER PATCHING HARDENING MONITORING WORKER NODE CONFIGURATION PATCHING HARDENING MONITORING NETWORK CONFIGURATION VPC NETWORK POLICIES ROUTE TABLES NACLs DATA NETWORK TRAFFIC PROTECTION CLIENT-SIDE ENCRYPTION SERVER-SIDE ENCRYPTION EKS CONTROL PLANE CONTROL PLANE CONFIGURATION PRIVATE CONTROL RBAC POLICIES

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Shared Responsibility applied to Amazon EKS Network Security IAM & RBAC Pod Security

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k IAM Roles for ServiceAccounts ... not for Pods

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Step 1: Create an IAM policy { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "sqs:GetQueueUrl", "sqs:ListDeadLetterSourceQueues", "sqs:ReceiveMessage", "sqs:GetQueueAttributes", "sqs:ListQueueTags”, "sqs:ListQueues" ], "Resource": ”*" } ] } IAM Roles for ServiceAccount

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Step 2: Create ServiceAccount, IAM Role and correct association eksctl create iamserviceaccount \ --name sqs-orderservice \ --namespace default \ --cluster eks-patterns-demo \ --attach-policy-arn IAM_policy_ARN \ --approve \ --override-existing-serviceaccounts Note: Cluster name is required but namespace and service account are optional IAM Roles for ServiceAccounts

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Step 3: Use ServiceAccount in Pod definition apiVersion: apps/v1 kind: Deployment metadata: name: orderservice spec: replicas: 1 selector: matchLabels: app: orderservice template: metadata: labels: app: orderservice spec: serviceAccountName: sqs-orderservice containers: - name: orderservice image: .../eks-patterns-demo:latest IAM Roles for ServiceAccount https://github.com/aws/amazon-eks-pod-identity-webhook

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k IAM Roles for ServiceAccount • SDK support required • Does not block access to Amazon EC2 metadata by default • Cross-Account IAM • Chained AssumeRole • OIDC provider from a different account

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Shared Responsibility applied to Amazon EKS Network Security IAM & RBAC Pod Security

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k PodSecurityPolicy • Released with k8s 1.13 (beta) • EKS default policy for compatibility • Define PSPs scoped for your environment • Less restrictive in dev • Different per project/team • Educate developers about security contexts • Test PSPs • Permissions defined via RBAC roles and bindings. • Examples at: https://github.com/therandomsecurityguy/kubernetes- security/tree/master/PodSecurityPolicies apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: eks.restrictive spec: hostNetwork: false seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny runAsUser: rule: RunAsAny fsGroup: rule: RunAsAny volumes: - '*'

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Secrets management Self-managed AWS Secrets Manager + ExternalSecrets https://github.com/godaddy/kubernetes-external-secrets Vault https://www.vaultproject.io/ SealedSecrets https://github.com/bitnami-labs/sealed-secrets Kubernetes managed (v1.12+ beta) KMS Provider https://kubernetes.io/docs/tasks/administer- cluster/kms-provider/ AWS Implementation https://github.com/kubernetes-sigs/aws-encryption- provider • Secrets might be stored in a different system • Strong encryption required for secrets • Customer Key to seal secrets

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Shared Responsibility applied to Amazon EKS Network Security IAM & RBAC Pod Security

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Networking

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Amazon EKS network architecture EKS VPC Customer VPC Worker Nodes EKS-owned elastic network interface Kubernetes API calls Exec, Logs, Proxy Public / Private Endpoint TLS TLS

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k EKS networking model • Kubernetes networking • Flat network, no NAT • No overhead • Native VPC IP per Pod • Multiple ENI per EC2 • Custom Network Configs • Additional CIDR ranges • SNAT / External SNAT • Private worker nodes • Configurable warm pool EC2

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k VPC CNI Pod networking Private IPs: 192.168.26.51, 192.168.17.109 Secondary IPs: 192.168.28.12, 192.168.7.109, … etcd API Server Scheduler …. …….. Worker Node 2 (EC2) DNS

NetworkPolicy with Calico Frontend Cart Recommender prod-namespace apiVersion: extensions/v1beta1 kind: NetworkPolicy metadata: name: frontend-to-cart spec: podSelector: matchLabels: role: cart ingress: - from: - podSelector: matchLabels: role: frontend ports: - protocol: TCP port: 80

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Shared Responsibility applied to Kubernetes Network Security Policies applied IAM & RBAC enabled Pod Security defined

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k ALB Ingress Controller AWS Resources Kubernetes Cluster Node Node Kubernetes API Server ALB Ingress Controller Node HTTPS Listener HTTP Listener Rule: /recommend Rule: /cart TargetGroup: Blue (Instance Mode) NodePort NodePort

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Ingress in Kubernetes kind: Ingress metadata: name: recommend namespace: microservices annotations: kubernetes.io/ingress.class: alb alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:… spec: rules: - http: paths: - path: /recommend backend: serviceName: recommend servicePort: 8080

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Observability

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k CloudWatch Container Insights

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k EKS Worker Pod fluentd daemonset Logging with CloudWatch Container Insights CWAgent daemonset

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Auto Scaling with Custom Metrics apiVersion: metrics.aws/v1alpha1 kind: ExternalMetric metadata: name: orderservice-queue-length spec: name: orderservice-queue-length resource: resource: "deployment" queries: - id: sqs_orderservice metricStat: metric: namespace: "AWS/SQS" metricName: "ApproximateNumberOfMessagesVisible" dimensions: - name: QueueName value: "eks-patterns-demo-queue" period: 300 stat: Average unit: Count returnData: true metrics-server CloudWatch Adapter CloudWatch Metric https://github.com/awslabs/k8s-cloudwatch-adapter

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Prometheus & Grafana Well-known open source monitoring & dashboarding Commonly used by our customers Long Term Storage Options Cortex -> DynamoDB / S3 Thanos -> S3 Install adapter to expose metrics to metrics-server https://github.com/DirectXMan12/k8s-prometheus-adapter

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Monitoring Patterns CloudWatch • Fully managed solution • All AWS service metrics available • Custom metrics support • Container Insights • Source for AWS AutoScaling Prometheus • In-Cluster vs. External hosting • Auto-discovery • Flexible and adaptable • Visualizations & Dashboards • Very popular with customers • Great integration with k8s Import / Export

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Application Architecture

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k StorageClass 1. Admin pre-provisions StorageClass based on workload needs 2. End user requests for specific volume types (For ex, encrypted io1 volume) 3. Control loop watches PVC request and allocates volume if PV exists MySQL Pods 4. End user creates stateful workload EBS (gp2) EBS (io1) EFS Local Volume Types workflow

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Storage architecture • Persistent Volume (PV) • Survives pod restarts • HostPath PV • Local storage PV • External storage systems • PVs are attached via PV claims • PV Claims (PVC) • Dynamic • Abstraction to underlying storage • ReadWriteOnce Pod Data Volume Pod (pvc) Persistent Vol Persistent Vol Pod (pvc) • Best resiliency • Low performance • No resiliency • Best performance • Medium resiliency • Best performance

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Container Storage Interface (CSI) for EFS and EBS EBS • Static Provisioning / Dynamic Provisioning • Mount Option • Block Volume • Volume Snapshot • NVMe Integrated k8s support CSI Driver EFS NFS shared filesystem Shared mount Single or Multiple Container Support only via CSI driver https://github.com/kubernetes-sigs/aws-ebs-csi-driver https://github.com/kubernetes-sigs/aws-efs-csi-driver

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Provisioning EFS apiVersion: v1 kind: PersistentVolume metadata: name: efs-pv spec: capacity: storage: 5Gi volumeMode: Filesystem accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Recycle storageClassName: efs-sc csi: driver: efs.csi.aws.com volumeHandle: [VolumeID] kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: efs-sc provisioner: efs.csi.aws.com apiVersion: v1 kind: PersistentVolumeClaim metadata: name: efs-claim spec: accessModes: - ReadWriteMany storageClassName: efs-sc resources: requests: storage: 5Gi

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k AWS Service Operator How do you provision AWS resources through k8s? AWS Service Operator • Define CloudFormation templates • Deploy to your k8s cluster • Operator takes care of creating the resources through CloudFormation Github: https://github.com/awslabs/aws-service-operator

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Operational Excellence

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Kubernetes performance envelope Number of Nodes Pod Churn Pod Density Networking Secrets Anti-affinity Active Namespaces

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Heavy monolithic pods in a very large cluster Number of Nodes Pod Churn Pod Density Networking Secrets Anti-affinity Active Namespaces

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Numerous densely bin packed microservice pods Number of Nodes Pod Churn Pod Density Networking Secrets Anti-affinity Active Namespaces

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Anti-affinity Anti-affinity constraints can keep heavy CPU using pods away from each other, on different hosts affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpessions: - key: app operator: In values: - web topologyKey: “kubernetes.io/hostname“ Helpful during node operations (upgrades, scaling) Tradeoff: heavier control plane scheduling burden, application pod performance bonus

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Preventing Pod disruptions Protect Pods against removal Helpful during node operations (draining, upgrades, scaling) Ensures safe auto-scaling actions Tradeoff: Cluster operations more complex apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: platform-pdb spec: minAvailable: 2 selector: matchLabels: app: web

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Optimizing your k8s operations • Taints & Tolerations • Optimize scheduling • Pod and Node level • Affinity & anti-affinity • Affinity pods scheduled on SSD/NVMe • Anti-Affinity: e.g masters aren’t scheduled with replicas • Configure PodDisruptionBudget • DaemonSet • e.g. metrics agent, LocalVolume provisioner

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k CoreDNS tuning • Scale CoreDNS deployment • Manual or via HPA • Caching • Modify cache TTL • autopath plugin • k8s NodeLocal DNS cache (1.15 – beta) • Instance local DNS cache $ kubectl describe configmap/coredns –n kube-system Corefile: ---- .:53 { errors health kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance }

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Best practices for EKS clusters • Small container images • Image pull policy • Storage • SSD/NVMe for performance • EBS for increased resiliency • Monitor key metrics • Watch overcommitted state • Pod churn • Cluster Autoscaler & HPA • Set resource limits • Multiple NodeGroups • Spot, GPU • Choose right size nodes • High Memory instances • CPU optimized • EKS CNI plugin • Use latest version • sysctl • conntrack

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Keep learning

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Keep learning AWS Containers Blog https://aws.amazon.com/blogs/containers/ Public Roadmap https://github.com/aws/containers- roadmap/projects/1?card_filter_query=label%3Aeks Spot – Definitive Guide https://itnext.io/the-definitive-guide-to-running-ec2-spot-instances-as- kubernetes-worker-nodes-68ef2095e767 Workshop https://eksworkshop.com

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Keep learning EKS Helm Charts https://github.com/aws/eks-charts https://github.com/aws-samples/amazon-k8s-node-drainer https://github.com/awslabs/amazon-eks-serverless-drainer Additional Resources https://github.com/ramitsurana/awesome-kubernetes https://k8spatterns.io/

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Thank You! Christoph Kassen Solutions Architect @christoph_k Feedback: http://bit.ly/36cv4E0