rights reserved. Amazon Confidential and Trademark – @christoph_k Amazon EKS Architecture https://00000000000000000000000000000000.xyz.eu-central-1.eks.amazonaws.com EKS workers kubectl AZ 1 AZ 2 AZ 3 Your AWS account VPC Amazon EKS
rights reserved. Amazon Confidential and Trademark – @christoph_k Multiple Availability Zones VPC Amazon EKS Architecture Highly available and single tenant infrastructure Multiple Availability Zones ASG with fixed size Fronted by a LoadBalancer NLB Etcd API Servers
rights reserved. Amazon Confidential and Trademark – @christoph_k Open source and EKS AWS VPC CNI Plugin AWS IAM Authenticator Amazon EKS AMI Key components of EKS are open source EKS runs 100% upstream Kubernetes /kubernetes /kubernetes/autoscaler /aws-labs/aws-service-operator /weaveworks/eksctl EBS, EFS, FSX CSI drivers kops Team contributes to or manages 20+ OSS projects
rights reserved. Amazon Confidential and Trademark – @christoph_k Major version upgrades Components AWS EKS CNI coreDNS kube-proxy EKS Control Plane Worker Nodes
rights reserved. Amazon Confidential and Trademark – @christoph_k Autoscaling operations • Use Auto Scaling groups per Availability Zone • EBS Volumes are zonal, prevent scheduling issues • AZ Rebalancing • Balance nodes equally across zones • Automatic Node Draining via Lifecycle-Hook • Disable AZRebalance • Scaling with Resource Reservations • Enable faster or predictive scaling • Anomaly detection
rights reserved. Amazon Confidential and Trademark – @christoph_k Step 2: Create ServiceAccount, IAM Role and correct association eksctl create iamserviceaccount \ --name sqs-orderservice \ --namespace default \ --cluster eks-patterns-demo \ --attach-policy-arn IAM_policy_ARN \ --approve \ --override-existing-serviceaccounts Note: Cluster name is required but namespace and service account are optional IAM Roles for ServiceAccounts
rights reserved. Amazon Confidential and Trademark – @christoph_k IAM Roles for ServiceAccount • SDK support required • Does not block access to Amazon EC2 metadata by default • Cross-Account IAM • Chained AssumeRole • OIDC provider from a different account
rights reserved. Amazon Confidential and Trademark – @christoph_k PodSecurityPolicy • Released with k8s 1.13 (beta) • EKS default policy for compatibility • Define PSPs scoped for your environment • Less restrictive in dev • Different per project/team • Educate developers about security contexts • Test PSPs • Permissions defined via RBAC roles and bindings. • Examples at: https://github.com/therandomsecurityguy/kubernetes- security/tree/master/PodSecurityPolicies apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: eks.restrictive spec: hostNetwork: false seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny runAsUser: rule: RunAsAny fsGroup: rule: RunAsAny volumes: - '*'
rights reserved. Amazon Confidential and Trademark – @christoph_k EKS networking model • Kubernetes networking • Flat network, no NAT • No overhead • Native VPC IP per Pod • Multiple ENI per EC2 • Custom Network Configs • Additional CIDR ranges • SNAT / External SNAT • Private worker nodes • Configurable warm pool EC2
rights reserved. Amazon Confidential and Trademark – @christoph_k Shared Responsibility applied to Kubernetes Network Security Policies applied IAM & RBAC enabled Pod Security defined
rights reserved. Amazon Confidential and Trademark – @christoph_k EKS Worker Pod fluentd daemonset Logging with CloudWatch Container Insights CWAgent daemonset
rights reserved. Amazon Confidential and Trademark – @christoph_k Prometheus & Grafana Well-known open source monitoring & dashboarding Commonly used by our customers Long Term Storage Options Cortex -> DynamoDB / S3 Thanos -> S3 Install adapter to expose metrics to metrics-server https://github.com/DirectXMan12/k8s-prometheus-adapter
rights reserved. Amazon Confidential and Trademark – @christoph_k Monitoring Patterns CloudWatch • Fully managed solution • All AWS service metrics available • Custom metrics support • Container Insights • Source for AWS AutoScaling Prometheus • In-Cluster vs. External hosting • Auto-discovery • Flexible and adaptable • Visualizations & Dashboards • Very popular with customers • Great integration with k8s Import / Export
rights reserved. Amazon Confidential and Trademark – @christoph_k StorageClass 1. Admin pre-provisions StorageClass based on workload needs 2. End user requests for specific volume types (For ex, encrypted io1 volume) 3. Control loop watches PVC request and allocates volume if PV exists MySQL Pods 4. End user creates stateful workload EBS (gp2) EBS (io1) EFS Local Volume Types workflow
rights reserved. Amazon Confidential and Trademark – @christoph_k Storage architecture • Persistent Volume (PV) • Survives pod restarts • HostPath PV • Local storage PV • External storage systems • PVs are attached via PV claims • PV Claims (PVC) • Dynamic • Abstraction to underlying storage • ReadWriteOnce Pod Data Volume Pod (pvc) Persistent Vol Persistent Vol Pod (pvc) • Best resiliency • Low performance • No resiliency • Best performance • Medium resiliency • Best performance
rights reserved. Amazon Confidential and Trademark – @christoph_k Container Storage Interface (CSI) for EFS and EBS EBS • Static Provisioning / Dynamic Provisioning • Mount Option • Block Volume • Volume Snapshot • NVMe Integrated k8s support CSI Driver EFS NFS shared filesystem Shared mount Single or Multiple Container Support only via CSI driver https://github.com/kubernetes-sigs/aws-ebs-csi-driver https://github.com/kubernetes-sigs/aws-efs-csi-driver
rights reserved. Amazon Confidential and Trademark – @christoph_k AWS Service Operator How do you provision AWS resources through k8s? AWS Service Operator • Define CloudFormation templates • Deploy to your k8s cluster • Operator takes care of creating the resources through CloudFormation Github: https://github.com/awslabs/aws-service-operator
rights reserved. Amazon Confidential and Trademark – @christoph_k Kubernetes performance envelope Number of Nodes Pod Churn Pod Density Networking Secrets Anti-affinity Active Namespaces
rights reserved. Amazon Confidential and Trademark – @christoph_k Heavy monolithic pods in a very large cluster Number of Nodes Pod Churn Pod Density Networking Secrets Anti-affinity Active Namespaces
rights reserved. Amazon Confidential and Trademark – @christoph_k Numerous densely bin packed microservice pods Number of Nodes Pod Churn Pod Density Networking Secrets Anti-affinity Active Namespaces
rights reserved. Amazon Confidential and Trademark – @christoph_k Anti-affinity Anti-affinity constraints can keep heavy CPU using pods away from each other, on different hosts affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpessions: - key: app operator: In values: - web topologyKey: “kubernetes.io/hostname“ Helpful during node operations (upgrades, scaling) Tradeoff: heavier control plane scheduling burden, application pod performance bonus