AWS Loft Munich 2019 - Kubernetes Patterns

AWS Loft Munich 2019 - Kubernetes Patterns

1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Christoph Kassen Solutions Architect - AWS @christoph_k Kubernetes Patterns on AWS
2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Customer use cases Microservices PaaS Platform-as-a-Service Enterprise App Migration Machine Learning
3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Amazon EKS
4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Amazon EKS Architecture https://00000000000000000000000000000000.xyz.eu-central-1.eks.amazonaws.com EKS workers kubectl AZ 1 AZ 2 AZ 3 Your AWS account VPC Amazon EKS
5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Multiple Availability Zones VPC Amazon EKS Architecture Highly available and single tenant infrastructure Multiple Availability Zones ASG with fixed size Fronted by a LoadBalancer NLB Etcd API Servers
6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Open source and EKS AWS VPC CNI Plugin AWS IAM Authenticator Amazon EKS AMI Key components of EKS are open source EKS runs 100% upstream Kubernetes /kubernetes /kubernetes/autoscaler /aws-labs/aws-service-operator /weaveworks/eksctl EBS, EFS, FSX CSI drivers kops Team contributes to or manages 20+ OSS projects
7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Transparent roadmap https://github.com/aws/containers-roadmap/
8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k CLI to provision EKS clusters - Manages worker node groups - Configures k8s CLI https://eksctl.io – Official open-source CLI tool for EKS eksctl create cluster --name=eks-patterns --nodes=4 --region=eu-central-1
9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k eksctl customizations kubeletExtraConfig: kubeReserved: cpu: "300m" memory: "300Mi" ephemeral-storage: "1Gi" kubeReservedCgroup: "/kube- reserved" systemReserved: cpu: "300m" memory: "300Mi" ephemeral-storage: "1Gi" evictionHard: memory.available: "200Mi" nodefs.available: "10%" YAML based configuration kubelet parameters Resource reservations Labels & Tains Custom VPC setups Autoscaling Multiple Node Groups Spot & GPU node support Windows worker IAM Policies gitops
10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Major version upgrades Components AWS EKS CNI coreDNS kube-proxy EKS Control Plane Worker Nodes
11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Worker Node Management
12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k • Amazon $ aws ssm get-parameter \ --name /aws/service/eks/optimized-ami/1.14/amazon-linux-2/recommended/image_id \ --region eu-central-1 --query Parameter.Value --output text Amazon https://github.com/awslabs/amazon-eks-ami • https://cloud-images.ubuntu.com/aws-eks/ Amazon EKS worker nodes
13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Set resource requirements for each Pod
14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Worker Node operations • Auto-Scaling • k8s Cluster Autoscaler • AWS Auto Scaling with custom Cloudwatch Metrics • Node Draining • Drain node before termination • 2-minute warning • Handlers • Listen for CloudWatch Events - https://github.com/aws-samples/amazon-k8s-node-drainer • Run handler on clusters - https://github.com/kube-aws/kube-spot-termination-notice-handler
15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Autoscaling operations • Use Auto Scaling groups per Availability Zone • EBS Volumes are zonal, prevent scheduling issues • AZ Rebalancing • Balance nodes equally across zones • Automatic Node Draining via Lifecycle-Hook • Disable AZRebalance • Scaling with Resource Reservations • Enable faster or predictive scaling • Anomaly detection
16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Export resource reservations
17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Running EKS with Spot Instances • Utilize Spot capacity to reduce instance spend • Dev/Test clusters • Ephemeral clusters • Auto Scaling Groups • Combine Spot & On-Demand • Mix instance types with care • Switch to spot via • LaunchTemplate • eksctl
18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k eksctl – Spot configuration apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: eks-patterns-demo region: eu-central-1 nodeGroups: - name: ng-1 minSize: 4 maxSize: 4 instancesDistribution: maxPrice: 0.5 instanceTypes: [“m5d.large", “m5d.xlarge"] onDemandBaseCapacity: 0 onDemandPercentageAboveBaseCapacity: 0 spotInstancePools: 2
19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Security
20. Shared responsibility for Amazon EKS CUSTOMER IAM APPLICATION POD AWS ENDPOINTS AWS IAM FOUNDATION SERVICES COMPUTE STORAGE DATABASES NETWORKING AWS GLOBAL INFRASTRACTURE REGIONS AVAILABILITY ZONES EDGE LOCATIONS MANAGED BY AWS CUSTOMERS MANAGED BY AWS KUBELET RBAC APP HOST AWS IAM CONTAINER PATCHING HARDENING MONITORING WORKER NODE CONFIGURATION PATCHING HARDENING MONITORING NETWORK CONFIGURATION VPC NETWORK POLICIES ROUTE TABLES NACLs DATA NETWORK TRAFFIC PROTECTION CLIENT-SIDE ENCRYPTION SERVER-SIDE ENCRYPTION EKS CONTROL PLANE CONTROL PLANE CONFIGURATION PRIVATE CONTROL RBAC POLICIES
21. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Shared Responsibility applied to Amazon EKS Network Security IAM & RBAC Pod Security
22. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k IAM Roles for ServiceAccounts ... not for Pods
23. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Step 1: Create an IAM policy { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "sqs:GetQueueUrl", "sqs:ListDeadLetterSourceQueues", "sqs:ReceiveMessage", "sqs:GetQueueAttributes", "sqs:ListQueueTags”, "sqs:ListQueues" ], "Resource": ”*" } ] } IAM Roles for ServiceAccount
24. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Step 2: Create ServiceAccount, IAM Role and correct association eksctl create iamserviceaccount \ --name sqs-orderservice \ --namespace default \ --cluster eks-patterns-demo \ --attach-policy-arn IAM_policy_ARN \ --approve \ --override-existing-serviceaccounts Note: Cluster name is required but namespace and service account are optional IAM Roles for ServiceAccounts
25. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Step 3: Use ServiceAccount in Pod definition apiVersion: apps/v1 kind: Deployment metadata: name: orderservice spec: replicas: 1 selector: matchLabels: app: orderservice template: metadata: labels: app: orderservice spec: serviceAccountName: sqs-orderservice containers: - name: orderservice image: .../eks-patterns-demo:latest IAM Roles for ServiceAccount https://github.com/aws/amazon-eks-pod-identity-webhook
26. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k IAM Roles for ServiceAccount • SDK support required • Does not block access to Amazon EC2 metadata by default • Cross-Account IAM • Chained AssumeRole • OIDC provider from a different account
27. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Shared Responsibility applied to Amazon EKS Network Security IAM & RBAC Pod Security
28. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k PodSecurityPolicy • Released with k8s 1.13 (beta) • EKS default policy for compatibility • Define PSPs scoped for your environment • Less restrictive in dev • Different per project/team • Educate developers about security contexts • Test PSPs • Permissions defined via RBAC roles and bindings. • Examples at: https://github.com/therandomsecurityguy/kubernetes- security/tree/master/PodSecurityPolicies apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: eks.restrictive spec: hostNetwork: false seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny runAsUser: rule: RunAsAny fsGroup: rule: RunAsAny volumes: - '*'
29. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Secrets management Self-managed AWS Secrets Manager + ExternalSecrets https://github.com/godaddy/kubernetes-external-secrets Vault https://www.vaultproject.io/ SealedSecrets https://github.com/bitnami-labs/sealed-secrets Kubernetes managed (v1.12+ beta) KMS Provider https://kubernetes.io/docs/tasks/administer- cluster/kms-provider/ AWS Implementation https://github.com/kubernetes-sigs/aws-encryption- provider • Secrets might be stored in a different system • Strong encryption required for secrets • Customer Key to seal secrets
30. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Shared Responsibility applied to Amazon EKS Network Security IAM & RBAC Pod Security
31. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Networking
32. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Amazon EKS network architecture EKS VPC Customer VPC Worker Nodes EKS-owned elastic network interface Kubernetes API calls Exec, Logs, Proxy Public / Private Endpoint TLS TLS
33. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k EKS networking model • Kubernetes networking • Flat network, no NAT • No overhead • Native VPC IP per Pod • Multiple ENI per EC2 • Custom Network Configs • Additional CIDR ranges • SNAT / External SNAT • Private worker nodes • Configurable warm pool EC2
34. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k VPC CNI Pod networking Private IPs: 192.168.26.51, 192.168.17.109 Secondary IPs: 192.168.28.12, 192.168.7.109, … etcd API Server Scheduler …. …….. Worker Node 2 (EC2) DNS
35. NetworkPolicy with Calico Frontend Cart Recommender prod-namespace apiVersion: extensions/v1beta1 kind: NetworkPolicy metadata: name: frontend-to-cart spec: podSelector: matchLabels: role: cart ingress: - from: - podSelector: matchLabels: role: frontend ports: - protocol: TCP port: 80
36. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Shared Responsibility applied to Kubernetes Network Security Policies applied IAM & RBAC enabled Pod Security defined
37. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k ALB Ingress Controller AWS Resources Kubernetes Cluster Node Node Kubernetes API Server ALB Ingress Controller Node HTTPS Listener HTTP Listener Rule: /recommend Rule: /cart TargetGroup: Blue (Instance Mode) NodePort NodePort
38. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Ingress in Kubernetes kind: Ingress metadata: name: recommend namespace: microservices annotations: kubernetes.io/ingress.class: alb alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:… spec: rules: - http: paths: - path: /recommend backend: serviceName: recommend servicePort: 8080
39. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Observability
40. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k CloudWatch Container Insights
41. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k EKS Worker Pod fluentd daemonset Logging with CloudWatch Container Insights CWAgent daemonset
42. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Auto Scaling with Custom Metrics apiVersion: metrics.aws/v1alpha1 kind: ExternalMetric metadata: name: orderservice-queue-length spec: name: orderservice-queue-length resource: resource: "deployment" queries: - id: sqs_orderservice metricStat: metric: namespace: "AWS/SQS" metricName: "ApproximateNumberOfMessagesVisible" dimensions: - name: QueueName value: "eks-patterns-demo-queue" period: 300 stat: Average unit: Count returnData: true metrics-server CloudWatch Adapter CloudWatch Metric https://github.com/awslabs/k8s-cloudwatch-adapter
43. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Prometheus & Grafana Well-known open source monitoring & dashboarding Commonly used by our customers Long Term Storage Options Cortex -> DynamoDB / S3 Thanos -> S3 Install adapter to expose metrics to metrics-server https://github.com/DirectXMan12/k8s-prometheus-adapter
44. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Monitoring Patterns CloudWatch • Fully managed solution • All AWS service metrics available • Custom metrics support • Container Insights • Source for AWS AutoScaling Prometheus • In-Cluster vs. External hosting • Auto-discovery • Flexible and adaptable • Visualizations & Dashboards • Very popular with customers • Great integration with k8s Import / Export
45. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Application Architecture
46. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k StorageClass 1. Admin pre-provisions StorageClass based on workload needs 2. End user requests for specific volume types (For ex, encrypted io1 volume) 3. Control loop watches PVC request and allocates volume if PV exists MySQL Pods 4. End user creates stateful workload EBS (gp2) EBS (io1) EFS Local Volume Types workflow
47. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Storage architecture • Persistent Volume (PV) • Survives pod restarts • HostPath PV • Local storage PV • External storage systems • PVs are attached via PV claims • PV Claims (PVC) • Dynamic • Abstraction to underlying storage • ReadWriteOnce Pod Data Volume Pod (pvc) Persistent Vol Persistent Vol Pod (pvc) • Best resiliency • Low performance • No resiliency • Best performance • Medium resiliency • Best performance
48. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Container Storage Interface (CSI) for EFS and EBS EBS • Static Provisioning / Dynamic Provisioning • Mount Option • Block Volume • Volume Snapshot • NVMe Integrated k8s support CSI Driver EFS NFS shared filesystem Shared mount Single or Multiple Container Support only via CSI driver https://github.com/kubernetes-sigs/aws-ebs-csi-driver https://github.com/kubernetes-sigs/aws-efs-csi-driver
49. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Provisioning EFS apiVersion: v1 kind: PersistentVolume metadata: name: efs-pv spec: capacity: storage: 5Gi volumeMode: Filesystem accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Recycle storageClassName: efs-sc csi: driver: efs.csi.aws.com volumeHandle: [VolumeID] kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: efs-sc provisioner: efs.csi.aws.com apiVersion: v1 kind: PersistentVolumeClaim metadata: name: efs-claim spec: accessModes: - ReadWriteMany storageClassName: efs-sc resources: requests: storage: 5Gi
50. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k AWS Service Operator How do you provision AWS resources through k8s? AWS Service Operator • Define CloudFormation templates • Deploy to your k8s cluster • Operator takes care of creating the resources through CloudFormation Github: https://github.com/awslabs/aws-service-operator
51. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Operational Excellence
52. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Kubernetes performance envelope Number of Nodes Pod Churn Pod Density Networking Secrets Anti-affinity Active Namespaces
53. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Heavy monolithic pods in a very large cluster Number of Nodes Pod Churn Pod Density Networking Secrets Anti-affinity Active Namespaces
54. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Numerous densely bin packed microservice pods Number of Nodes Pod Churn Pod Density Networking Secrets Anti-affinity Active Namespaces
55. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Anti-affinity Anti-affinity constraints can keep heavy CPU using pods away from each other, on different hosts affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpessions: - key: app operator: In values: - web topologyKey: “kubernetes.io/hostname“ Helpful during node operations (upgrades, scaling) Tradeoff: heavier control plane scheduling burden, application pod performance bonus
56. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Preventing Pod disruptions Protect Pods against removal Helpful during node operations (draining, upgrades, scaling) Ensures safe auto-scaling actions Tradeoff: Cluster operations more complex apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: platform-pdb spec: minAvailable: 2 selector: matchLabels: app: web
57. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Optimizing your k8s operations • Taints & Tolerations • Optimize scheduling • Pod and Node level • Affinity & anti-affinity • Affinity pods scheduled on SSD/NVMe • Anti-Affinity: e.g masters aren’t scheduled with replicas • Configure PodDisruptionBudget • DaemonSet • e.g. metrics agent, LocalVolume provisioner
58. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k CoreDNS tuning • Scale CoreDNS deployment • Manual or via HPA • Caching • Modify cache TTL • autopath plugin • k8s NodeLocal DNS cache (1.15 – beta) • Instance local DNS cache $ kubectl describe configmap/coredns –n kube-system Corefile: ---- .:53 { errors health kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance }
59. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Best practices for EKS clusters • Small container images • Image pull policy • Storage • SSD/NVMe for performance • EBS for increased resiliency • Monitor key metrics • Watch overcommitted state • Pod churn • Cluster Autoscaler & HPA • Set resource limits • Multiple NodeGroups • Spot, GPU • Choose right size nodes • High Memory instances • CPU optimized • EKS CNI plugin • Use latest version • sysctl • conntrack
60. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Keep learning
61. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Keep learning AWS Containers Blog https://aws.amazon.com/blogs/containers/ Public Roadmap https://github.com/aws/containers- roadmap/projects/1?card_filter_query=label%3Aeks Spot – Definitive Guide https://itnext.io/the-definitive-guide-to-running-ec2-spot-instances-as- kubernetes-worker-nodes-68ef2095e767 Workshop https://eksworkshop.com
62. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark – @christoph_k Keep learning EKS Helm Charts https://github.com/aws/eks-charts https://github.com/aws-samples/amazon-k8s-node-drainer https://github.com/awslabs/amazon-eks-serverless-drainer Additional Resources https://github.com/ramitsurana/awesome-kubernetes https://k8spatterns.io/
63. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Thank You! Christoph Kassen Solutions Architect @christoph_k Feedback: http://bit.ly/36cv4E0