Ignite the Tour Paris: 8 lessons learned running K8S with Azure Kubernetes Service

Transcript

1. Think ahead. Act now.
2. Think ahead. Act now. Pascal Naber @pascalnaber 8 lessons learned using k8s with Azure Kubernetes Service
4. Think ahead. Act now.
5. https://aka.ms/LearnKubernetes
6. Provisioning Infrastructure as Code Networking Maintenance Monitoring Updates CI/CD Patch management Security Configuration Running apps Key Vault Storage
7. Think ahead. Act now.
8. Install AKS using CLI ## Create resourcegroup az group create --name ignitetour --location eastus ## Create Kubernetes cluster az aks create -g aksdemo -n ignitetour --generate-ssh-keys
10. Provisioning We need an AKS cluster that: • supports multiple teams & environments • is only accessible through firewall • is highly available
11. Multiple teams and environments One cluster per team or environment Physical Development Staging Production Team Morpheus Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Production Team Trinity Development, Staging Production Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Team Morpheus Team Trinity Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Staging Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Team Morpheus Team Trinity One cluster is reused over environments and teams Logical
12. One dev cluster per team Development Team Morpheus Pod Pod Pod Pod Pod Pod Master branch Feature X branch Pod Pod Pod Pod Pod Pod Feature Y branch Pod Pod Pod Pod Pod Pod Staging Production Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod One cluster is reused over environments and teams Multiple teams and environments Team Morpheus Team Trinity Team Morpheus Team Trinity
13. West Europe AKS cluster Application Gateway & Firewall
14. Azure Application Gateway & Firewall Web traffic load balancer URL based routing Endpoint in Azure or External • HTTP rate limiting • Scanner detection • Session fixation • LFI/RFI WAG v2 SSL termination or End-to-End SSL Automatic failover Web Application Firewall - Layer 7 OWASP rulesets • SQL Injection • Cross site scripting • Protocol violations • Generic attacks
15. West Europe AKS cluster Application Gateway & Firewall
16. West Europe VNET AKS cluster Application Gateway & Firewall Public IP-Address
17. Azure Application Gateway & Firewall Rule Listener Http Setting Backend Pool Frontend IP Config Frontend Port Health Probe Protocol Protocol Port Public/Private Application Gateway 1 1 2 1 2 1 3 SSL certificate pfx 1 Backend certificate cer 1 HTTP/HTTPS HTTP/HTTPS Public IP-Address Private IP-Address
18. West Europe VNET Traffic Manager AKS cluster Application Gateway & Firewall Public IP-Address North Europe VNET AKS cluster Application Gateway & Firewall Public IP-Address http://ignitetour.trafficmanager.net
19. Azure Traffic Manager Traffic Manager http://ignitetour.trafficmanager.net DNS based load balancer Distribute traffic across regions Endpoint in Azure or External Automatic failover Routing based on • Performance • Weighted • Priority • Geographical
20. Think ahead. Act now. Azure Front Door Global Load Balancer Traffic Manager + Application Gateway + CDN in one product Global Http Load Balancing URL redirection SSL termination WAF rules Static Content Cache
21. West Europe VNET Traffic Manager AKS cluster Application Gateway & Firewall Public IP-Address North Europe VNET AKS cluster Application Gateway & Firewall Public IP-Address http://ignitetour.trafficmanager.net
22. DNS Zone West Europe VNET Traffic Manager AKS cluster Application Gateway & Firewall Public IP-Address North Europe VNET AKS cluster Application Gateway & Firewall Public IP-Address http://www.ignitetour.cf http://ignitetour.trafficmanager.net
23. Traffic Manager VNET AKS cluster Application Gateway & Firewall DNS Zone Log Analytics VNET AKS cluster Application Gateway & Firewall Log Analytics West Europe Public IP-Address North Europe Public IP-Address
24. Traffic Manager VNET AKS cluster Application Gateway & Firewall Service Principal (AKS) DNS Zone Log Analytics VNET AKS cluster Application Gateway & Firewall Service Principal (AKS) Log Analytics Service Principal (Deploy) Key Vault Service Principal (Deploy) Key Vault West Europe Public IP-Address North Europe Public IP-Address
25. Traffic Manager West Europe VNET North Europe AKS cluster Application Gateway & Firewall Public IP-Address Service Principal (AKS) DNS Zone Log Analytics VNET AKS cluster Application Gateway & Firewall Public IP-Address Service Principal (AKS) Log Analytics WORKSPACE_RESOURCEID=$(az resource show -g $RESOURCEGROUP_K8S -n $WORKSPACE_NAME --query 'id' -o tsv) AKSSUBNETID=$(az network vnet subnet show -g $RESOURCEGROUP_NETWORK -n aks-subnet --vnet-name $VNET_NAME --query id --output tsv) # Provision AKS az aks create -g $RESOURCEGROUP_K8S -n $AKS_NAME \ --kubernetes-version 1.11.5 \ --node-count $AKS_NODE_COUNT \ --node-vm-size $AKS_VM_SIZE \ --max-pods 110 \ --ssh-key-value "$(az keyvault secret show --name ssh-publickey --vault-name $KEYVAULT_ --service-principal $(az keyvault secret show --name spn-aks-id --vault-name $KEYVAULT_ --client-secret "$(az keyvault secret show --name spn-aks-password --vault-name $KEYVAU --network-plugin azure \ --vnet-subnet-id $AKSSUBNETID \ --docker-bridge-address 172.17.0.1/16 --dns-service-ip 10.2.0.10 --service-cidr 10.2.0.0/24 --enable-addons monitoring --workspace-resource-id $WORKSPACE_RESOURCEID \ --nodepool-name "linuxpool" \ --vm-set-type VirtualMachineScaleSets \ --node-zones 1 2 3 \ --load-balancer-sku standard + Network Contributor Service Principal (Deploy) Key Vault Service Principal (Deploy) Key Vault ## Create Kubernetes cluster az aks create -g aksdemo -n ignitetour --generate-ssh-keys
26. Think ahead. Act now. DEMO AKS Provisioning
27. Availability zones Availability Set (99.5%) Availability Zones (99.99%) Availability Zone 1 Availability Zone 2 Availability Zone 3 Azure region €0.009 per GB inbound & outbound traffic
28. Master nodes AKS offers 99.5% (SLO) Worker nodes AKS offers 99.95% (Azure VM SLA in VMSS) With Availability Zones AKS offers 99.99% for 2+ VMs AKS Availability SLA Downtime per year Downtime per month 99,999% 5m 15.6s 26.3s 99,99% 52m 35s 4m 23s 99,95% 4h 22m 21m 54s 99,9% 8h 45m 43m 49s 99,5% 1d 19h 49m 3h 39m
29. Master nodes AKS offers 99.5% (SLO) Worker nodes AKS offers 99.95% (Azure VM SLA in VMSS) With Availability Zones AKS offers 99.99% for 2+ VMs AKS Availability SLA Downtime per year Downtime per month 99,999% 5m 15.6s 26.3s 99,99% 52m 35s 4m 23s 99,95% 4h 22m 21m 54s 99,9% 8h 45m 43m 49s 99,5% 1d 19h 49m 3h 39m
30. Master nodes AKS offers 99.5% (SLO) Worker nodes AKS offers 99.95% (Azure VM SLA in VMSS) With Availability Zones AKS offers 99.99% for 2+ VMs AKS Availability SLA Downtime per year Downtime per month 99,999% 5m 15.6s 26.3s 99,99% 52m 35s 4m 23s 99,95% 4h 22m 21m 54s 99,9% 8h 45m 43m 49s 99,5% 1d 19h 49m 3h 39m
31. Master nodes AKS offers 99.5% (SLO) Worker nodes AKS offers 99.95% (Azure VM SLA in VMSS) With Availability Zones AKS offers 99.99% for 2+ VMs AKS Availability SLA Downtime per year Downtime per month 99,999% 5m 15.6s 26.3s 99,99% 52m 35s 4m 23s 99,95% 4h 22m 21m 54s 99,9% 8h 45m 43m 49s 99,5% 1d 19h 49m 3h 39m 47 minutes
32. Traffic Manager West Europe VNET North Europe AKS cluster Application Gateway & Firewall Public IP-Address Service Principal (AKS) Service Principal (Deploy) Key Vault DNS Zone Log Analytics VNET AKS cluster Application Gateway & Firewall Public IP-Address Service Principal (AKS) Service Principal (Deploy) Key Vault Log Analytics Service Principal (Registry) Container Registry Common AKS cluster Production Development
33. Traffic Manager West Europe North Europe AKS cluster Key Vault DNS Zone AKS cluster Key Vault Container Registry Common
34. Traffic Manager West Europe North Europe AKS cluster Key Vault DNS Zone AKS cluster Key Vault Container Registry Common SQL Azure MySQL Write Write SQL Azure MySQL Read Read
35. Traffic Manager West Europe North Europe AKS cluster Key Vault DNS Zone AKS cluster Key Vault Container Registry Common Cosmos DB SQL Azure MySQL Write Write Write Cosmos DB SQL Azure MySQL Read Read Write
36. Think ahead. Act now. Lessons learned regarding Provisioning Minimize time to market Availability and Firewalls requirements have complex consequences Use Key Vault for secrets Provisioning AKS (and everything related) for production is complex Automate everything!
37. Think ahead. Act now. Configuration We need an AKS cluster that: • shows a dashboard • has Ingress configured • minimize maintenance
38. kubectl proxy Dashboard kubectl create clusterrolebinding kubernetes-dashboard -n kube-system --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard kubectl create clusterrolebinding kubernetes-dashboard -n kube-system --clusterrole=view --serviceaccount=kube-system:kubernetes-dashboard
39. Ingress Pod 10.0.0.1 Pod 10.0.0.2 Pod 10.0.0.3 Service Type: ClusterIP Pod 10.0.0.4 Pod 10.0.0.5 Pod 10.0.0.6 Service Type: ClusterIP Pod 10.0.0.7 Pod 10.0.0.8 Pod 10.0.0.9 Service Type: ClusterIP Service Type: LoadBalancer Public or Private ip address: 37.17.208.21 Pod quay.io/kubernetes-ingress-controller/ nginx-ingress-controller:0.15.0 Ingress host: gaming. ignitetour.cf serviceName: gamingwebapp Ingress host: erp. ignitetour.cf serviceName: erpsvc Ingress host: www. ignitetour.cf/crm serviceName: crmsvc Label: erp Label: frontend Label: crm helm upgrade nginxingress --install stable/nginx-ingress --namespace kube-system -f internalingress.yaml Addon Http Application Routing
40. Helm Helm is a tool for managing packages of pre-configured Kubernetes resources Linkerd Logstash Magento Mariadb Mongodb Mssql-linux Mysql Neo4j Newrelic Openvpn Postgresql Presto Prometheus Rabbitmq Redis Selenium Sonarqube Spinnaker Sysdig Tensorflow Traefik Wordpress Artifactory Cert-manager Consul Cassandra Couchdb Datadog Docker-registry Drupal Elasticsearch Ethereum fluentd Gitlab Grafana Hadoop Heapster Jenkins Joomla Kafka Kibana Kong Kubeless Lamp # helm service account and other needed resources kubectl apply -f helm.yaml # install helm locally & on the k8s cluster (tiller) helm init --service-account tiller-serviceaccount --upgrade --force-upgrade # latest charts helm repo update https://github.com/kubernetes/helm
41. Automatic reboots of worker nodes Safe & automatic reboots of nodes with Kured https://github.com/weaveworks/kured helm upgrade kured --install stable/kured --namespace kured
42. NodePool: Windows | k8s 1.14.8 Standard_DS2_v3 NodePool: Linux | k8s 1.14.8 Standard_DS2_v2 NodePool: Linux | k8s 1.14.8 Standard_NC6 (GPU optimized) NodePool: Linux | k8s 1.14.7 Standard_DS2_v2 Nodepools
43. 1. Provision new cluster next to existing cluster 2. Run your containers on the new cluster 3. Switch to new endpoint 4. Remove old cluster Upgrade cluster Using new cluster using NodePools az aks upgrade --name $clustername --resource-group $rg --Kubernetes-version 1.14.8 Risk Upgrade AKS 1. Upgrade the Control Plane to latest version az aks upgrade -n $clustername -g $rg -k 1.14.8 --control-plane-only 2. Add a new NodePool with latest version 3. Run your containers on the new NodePool 4. Remove old NodePool Cost & Complexity
44. Think ahead. Act now. DEMO AKS Configuration
45. Kubernetes cluster Master Azure Container Instances (ACI) Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Partly Managed IaaS Cluster Managed Cluster Kubernetes cluster Worker Node Worker Node Master Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Kubectl get nodes NAME STATUS ROLES AGE VERSION aks-agentpool-17613766-0 Ready agent 4h v1.14.8 aks-agentpool-17613766-1 Ready agent 4h v1.14.8 aks-agentpool-17613766-2 Ready agent 4h v1.14.8 virtual-node-aci-linux Ready agent 4h v1.14.8
46. Think ahead. Act now. AKS Worker Nodes are (managed) IaaS Use Helm to install applications to K8S Lessons learned regarding Configuration Dashboard doesn’t work out of the box Kubernetes as PaaS possible with Virtual Kubelet Kubernetes ecosystem is amazing Use NodePools for upgrades and different workloads
47. Think ahead. Act now. Running Applications Applications within AKS cluster : • Automated deployments • Stateful applications • Monitoring • Keyvault
48. Think ahead. Act now. Create Helm chart ## Create helm chart helm create podinfo Alternatives: Draft, Dev Spaces ## Install or upgrade helm chart helm upgrade podinfo --install ./podinfo --set image.tag=213 --set ingress.host=podinfo-master.dev-ignitetour.cf
49. Pod Volumes Secret Storage Account Azure File Azure Disk
50. Pod Volumes Persistent Volumes Pod Persistent VolumeClaim StorageClass Persistent Volume Secret Pod Persistent VolumeClaim StorageClass Persistent Volume Secret Dynamic Static Secret Storage Account Azure File Azure Disk Storage Account Azure File Azure Disk Storage Account Azure File Azure Disk MC_...
51. AzureDisk - ReadWriteOnce only - 1 Pod only - Slow in provisioning - Limited nr of disks - No direct access to data + Premium (SSD) for production AzureFile + All AccessModes + Shared by Pods + Fast in provisioning + Unlimited volumes + Direct access to data - SMB = slow Pod Volumes Secret Storage Account Azure File Azure Disk
52. Secrets Goal: Read secrets from a Pod Use Key Vault Solution: Use Managed Identity (MI) to access the Key Vault Secrets in Kubernetes: base64 encoded
53. Think ahead. Act now. No integration between AKS & Key Vault (yet) AAD Pod Identity https://github.com/Azure/aad-pod-identity/ Key Vault Flex Volume: https://github.com/Azure/kubernetes-keyvault-flexvol Secrets
54. Think ahead. Act now. Lessons learned regarding Applications Use Helm to deploy your applications Use Key Vault for your application secrets Use Log Analytics + Container Solution for logging and monitoring Prevent stateful applications Automate everything: application deployment also!
55. Wrap up Running Docker containers is cloud independent with Kubernetes, what about the other services… Applying DevOps to an application landscape using AKS takes a lot of knowledge and time for the whole team Automate everything Once you have a complete DevOps way of working for your applications, your team is Faster, Cheaper and Better
56. Think ahead. Act now. https://pages.xpirit.com/magazine2018 https://github.com/pascalnaber/ignitetourparis2019 @pascalnaber Pascal Naber Coding Azure Architect Xpirit Netherlands