Container Orchestration on Azure @ DevOn

Transcript

1. Container orchestration in Azure Azure Kubernetes Service & Azure Container Apps Pascal Naber
3. WHY I LIKE Containers
4. They are FAST
5. They are PORTABLE
6. They are ISOLATED
7. Moving to production Scaling, load balancing, fault tolerance
8. Scaling, Load balancing & Fault tolerance Server Frontend Backend
9. Scaling, Load balancing & Fault tolerance Server Server Server Container orchestration
10. Server Scaling, Load balancing & Fault tolerance Server Server Container orchestration
11. Container orchestrators Cluster High Availability Hyper-Scale Hybrid Operations High Density Rolling Upgrades Stateful services Low Latency Fast startup & shutdown Container Orchestration & lifecycle management Replication & Failover Load balancing Self-healing Automated Rollback Health Monitoring Placement Constraints Microservices Service Fabric
12. Orchestrators cluster Node (Worker) Node (Worker) Master
13. Kubernetes Kubernetes cluster Node (Minion) Node (Minion) Master
14. Master Node Kubernetes architecture Master API Server Scheduler Controller Manager Node etcd kubelet kube-proxy Pod Pod Pod Kubectl (CLI) Dashboard (UI) REST client (Code)
15. History of Kubernetes June 2014 First commit September 2014 July 2015 March 2018 Announced Kubernetes v1 CNCF 1st Graduate
16. How HOT is Kubernetes actually?
17. Most loved and wanted tools. https://survey.stackoverflow.co/2022
18. 59,164 responses % of developers who are not developing with but have expressed interest in developing with it https://survey.stackoverflow.co/2022
19. Fundamental tool for Professional Developers https://survey.stackoverflow.co/2022
20. https://www.gartner.com/en/newsroom/press-releases/2020- 06-25-gartner-forecasts-strong-revenue-growth-for-global-co On 25-6-2020: “by 2022, more than 75% of global organizations will be running containerized applications in production, up from less than 30% today”
21. PUBLISHED: FEBRUARY 10, 2022 https://www.cncf.io/reports/cncf-annual-survey-2021/
23. Kubernetes • k8s • OOS • Extensible • Large, rapidly growing ecosystem • Facilitates declarative configuration and automation Κυβερνήτης -- Greek: meaning helmsman or pilot 1 2 3 4 5 6 7 8
24. Hosting of Kubernetes • Local • Minikube • Docker for Windows • Docker for Mac • Raspberry Pi • Private datacenter Public Cloud GKE (26-8-2015) EKS (5-6-2018) ACS (19-4-2016) AKS (13-6-2018)
25. Partly managed Kubernetes Only pay for worker nodes Upgrades managed by Azure Workers are Managed IaaS Azure Kubernetes Service (AKS) Azure Kubernetes
26. Azure Kubernetes Service (AKS) master master master worker worker worker AKS 100% managed by Microsoft IaaS managed by Microsoft € 0 € … (VM pricing)
27. Uptime SLA Service-level objective 99.5% Since 15-5-2020: • SLA for uptime for API Server - 99.9% (without Availability zones) - 99,95% (with Availability zones) €0.099 per cluster per hour When the API is down, Applications are not impacted
28. Workernodes availability options Highest RTO and RPO Lowest RTO and RPO Highest Cost and Complexity Lowest Cost and Complexity RTO&RPO Cost & Complexity Back up and Restore Availability Zones Multiple Clusters In Region Multiple Clusters Across Regions 99,99% 99,95% > 99,99%
29. Availability zones Availability Set (99.5%) Availability Zones (99.99%) Availability Zone 1 Availability Zone 2 Availability Zone 3 Azure region €0.009 per GB inbound & outbound traffic
30. High availability ✓ Start: Everything as code & Backup strategy ✓ Backups shipped to different regions ✓ Cross-Availability-Zone traffic charges ✓ ACR in other regions also ✓ Challenge is always the state
31. NodePool: Windows | k8s 1.22.11 Standard_DS2_v3 NodePool: Linux | k8s 1.22.11 Standard_DS2_v2 NodePool: Linux | k8s 1.22.11 Standard_NC6 (GPU optimized) NodePool: Linux | k8s 1.19.11 Standard_DS2_v2 Cluster management: Nodepools
32. Cluster upgrades • Support for N-2 • Semantic versioning (Major, Minor, Patch) • Current version 1.22.x Supported versions: 1.22.x 1.21.x 1.20.x • Upgrades are sequential • Quota • Subnet address availability • Pod Disruption Budget (PDB)
33. 1. Provision new cluster next to existing cluster 2. Run your containers on the new cluster 3. Switch to new endpoint 4. Remove old cluster • Upgrade cluster Using new cluster using NodePools az aks upgrade --name $clustername --resource-group $rg --Kubernetes-version 1.22.11 Risk Upgrade AKS 1. Upgrade the Control Plane to latest version az aks upgrade -n $clustername -g $rg -k 1.22.11 --control-plane-only 2. Add a new NodePool with latest version 3. Taint the new NodePool 4. Test run your containers on the new NodePool 5. Deploy containers again 6. Remove old NodePool Cost & Complexity
34. Automatic reboots of worker nodes • Safe & automatic reboots of nodes with Kured https://github.com/weaveworks/kured helm upgrade kured --install stable/kured --namespace kured
35. Missing pieces? Deploy to Production
36. Install AKS ARM Ansible Portal
37. Install AKS Bicep Terraform
38. Install AKS using CLI ## Create resourcegroup az group create --name devon --location westeurope ## Create Kubernetes cluster az aks create -g aksdemo -n devon --enable-managed-identity ## Create Kubernetes cluster az aks create --resource-group devon --name aksdemo --enable-managed-identity --kubernetes-version 1.22.11 --node-vm-size Standard_DS2_v2 --node-count 3 --enable-addons monitoring --max-pods $AKS_MAX_PODS_PER_NODE --ssh-key-value "C:\repos\pascal\ssh\public.pub" --load-balancer-sku standard --network-plugin azure --vm-set-type VirtualMachineScaleSets --vnet-subnet-id $AKSSUBNETID --docker-bridge-address $AKS_NETWORKING_DOCKER_BRIDGE_ADDRESS --dns-service-ip $AKS_NETWORKING_DNS_SERVICE_IP --service-cidr $AKS_NETWORKING_SERVICE_CIDR
39. https://azure.github.io/AKS-Construction/
40. Azure resources for AKS devon Upgrade Scale
41. Azure resources for AKS MC_ devon_aksdemo_westeurope devon
42. ## download & install kubectl az aks install-cli ## download credentials az aks get-credentials --resource-group devon --name aksdemo Post Install steps
43. Deployment flow Create Docker Images Apply K8s deployments Push Docker Images to registry 1 2 3
44. Deployment Pipelines Github Actions Azure DevOps Pipelines
45. Kubernetes concepts
46. Pod • Group of 1 or more containers • Shared Storage • Shared Network • Same IP-address • Shared port-range Pod 10.0.0.1 Storage Pod 10.0.0.2 Storage Port 80 Port 8080 Port 80
47. leaderboard-api.yaml Pod Pod Pod Replica Set Deployment Label: backend Selector Label: backend Label: backend Label: backend Label: backend Selector Label: backend C:> kubectl apply –f leaderboard-api.yaml
48. C:> kubectl apply –f update.yaml Zero-downtime deployment Can be rolled back Rolling updates
49. Environment variables & Secrets
50. C:> kubectl apply –f secret.yaml ## Create Secret to access Azure Container Registry kubectl create secret docker-registry clouddemoimages --docker-server=https://clouddemo.azurecr.io --docker-username=clouddemo --docker-password=kD98ddl$= --docker-email=pnaber@xpirit.com Secrets: Base64 encoded values Environment variables & Secrets
51. Health checks livenessProbe Indicates whether the Container is running Restart in case of failure Container: https://www.app-metrics.io/ HealthChecks library https://github.com/dotnet-architecture/HealthChecks readinessProbe Indicates whether the Container is ready to service requests No traffic is routed to the Pod C:> kubectl apply –f leaderboard-api.yaml Microsoft.AspNetCore.Diagnostics.HealthChecks
52. Kubernetes – Self healing Every n seconds check Restarts container During rolling update deployment Stop deployment During container startup No traffic Default endpoint Health endpoint (/health) Health endpoints returns != 200?
53. Resource Management requests: Minimum required resources limits: Capped resource usage 100m = 0.1 cpu 500m = 0.5 cpu 64Mi = 64 MB memory 128Mi = 128 MB memory C:> kubectl apply –f leaderboard-api.yaml
54. Horizontal Pod Autoscaler Required: Resource requests & limit Heapster Scaling based on 1 or multiple items CPU Memory Custom Metrics C:> kubectl apply –f leaderboard-api.yaml
55. Horizontal Pod Autoscaler Required: Resource requests & limit Heapster C:\kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE autoscaler leaderboard-api 0% / 50% 1 10 1 9m C:\kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE autoscaler leaderboard-api 218% / 50% 1 10 1 10m C:\kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE autoscaler leaderboard-api 218% / 50% 1 10 4 10m C:\kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE autoscaler leaderboard-api 44% / 50% 1 10 7 13m C:\kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE autoscaler leaderboard-api 51% / 50% 1 10 6 21m
56. Azure Kubernetes Service (AKS) - Scaling worker worker worker worker replicas replicas 5 4 Pod Autoscaler 4-20 > 60% CPU Cluster Autoscaler
57. How to access the Pods? From inside and outside the cluster
58. Node 1 Pod Label: backend 10.0.0.2 Pod Label: backend 10.0.0.3 Node 0 Pod Label: backend 10.0.0.1 Node 2 Pod Label: backend 10.0.0.4 Pod Label: backend 10.0.0.5 Pods are mortal
59. Service Service Type: ClusterIP Selector Label: backend Node 1 Pod Label: backend 10.0.0.2 Node 0 Pod Label: backend 10.0.0.1 Node 2 Pod Label: backend 10.0.0.4 Pod Label: backend 10.0.0.5 192.168.0.1
60. Service Service Type: LoadBalancer Selector Label: backend Node 1 Pod Label: backend 10.0.0.2 Node 0 Pod Label: backend 10.0.0.1 Node 2 Pod Label: backend 10.0.0.4 Pod Label: backend 10.0.0.5 Public ip address: 37.17.208.21 192.168.0.2
61. Frontend Backend Pod Pod Pod Pod Pod Pod Pod Pod Service Type: ClusterIP Service Type: LoadBalancer Public ip address: 37.17.208.21 C:> kubectl apply –f leaderboard-service.yaml C:> kubectl apply –f gamingwebapp.yaml 192.168.0.2 192.168.0.1
62. Too many ip-addresses. Now what? I want to access all my services through the same ip-address
63. Ingress Pod 10.0.0.1 Pod 10.0.0.2 Pod 10.0.0.3 Service Type: ClusterIP Pod 10.0.0.4 Pod 10.0.0.5 Pod 10.0.0.6 Service Type: ClusterIP Pod 10.0.0.7 Pod 10.0.0.8 Pod 10.0.0.9 Service Type: ClusterIP Service Type: LoadBalancer Public ip address: 37.17.208.21 Pod quay.io/kubernetes-ingress-controller/ nginx-ingress-controller:0.15.0 Selector Label: ingresscontroller Label: ingresscontroller Ingress host: gaming. community.cf serviceName: gamingwebapp Ingress host: erp. community.cf serviceName: erpsvc Ingress host: www. community.cf/crm serviceName: crmsvc nginx.conf gaming. matrixproject.cf service: gamingwebapp erp. matrixproject.cf service: erpsvc matrixproject.cf/crm Service: crmsvc Automatically generated Label: erp Label: frontend Label: crm
64. • Deployment • Replica set • Pod • Label • Rolling update • Health check • Environment variables • Secret • Resource management • Horizontal Pod Autoscaler Namespace Service Ingress Annotation Persistent Volume Cron Job Deamon Set Job Stateful Set Config Map
65. AKS Networking Kubenet Basic networking (default) Azure CNI Advanced networking Slightly faster Windows Nodes support Node 1 Node 2 IP3 IP4 IP1 IP2 Node 1 Node 2 IP1 IP2 Overlay Network IP1 IP2
66. Application Gateway Ingress Controller (AGIC) https://azure.github.io/application-gateway-kubernetes-ingress/
67. Security • Cluster & Node security • Secure access to API (Bastion) • Regular Cluster upgrades • Authenticate using AAD • Apply security updates (Kured) • Containers • Scan for vulnerabilities (Twistlock, Aqua) • Regularly update the base image • No root access (privileged access) • Network security • WAF ingress • Firewall for egress • Network policies
68. Kubernetes Ecosystem
69. = + Kubernetes Ecosystem +
70. Helm Helm is a tool for managing packages of pre-configured Kubernetes resources https://github.com/kubernetes/helm Linkerd Logstash Magento Mariadb Mongodb Mssql-linux Mysql Neo4j Newrelic Openvpn Postgresql Presto Prometheus Rabbitmq Redis Selenium Sonarqube Spinnaker Sysdig Tensorflow Traefik Wordpress Artifactory Cert-manager Consul Cassandra Couchdb Datadog Docker-registry Drupal Elasticsearch Ethereum fluentd Gitlab Grafana Hadoop Heapster Jenkins Joomla Kafka Kibana Kong Kubeless Lamp
71. Azure Kubernetes Service (AKS) master master master worker worker worker AKS 100% managed by Microsoft IaaS managed by Microsoft
72. Starts in seconds No VM Management Billed per second Linux and Windows containers Azure Container Instances (ACI) Azure Azure Container Instances (ACI) Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod
73. AKS + ACI AKS 100% managed by Microsoft 100% managed by Microsoft worker master master master
74. P4l Naber Coding Azure Architect Freelancer @ Tech Driven @pascalnaber http://pascalnaber.wordpress.com https://github.com/pascalnaber/devon