Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ignite the Tour Paris: 8 lessons learned running K8S with Azure Kubernetes Service

Pascal Naber
November 14, 2019
210

Ignite the Tour Paris: 8 lessons learned running K8S with Azure Kubernetes Service

Pascal Naber

November 14, 2019
Tweet

More Decks by Pascal Naber

Transcript

  1. Provisioning Infrastructure as Code Networking Maintenance Monitoring Updates CI/CD Patch

    management Security Configuration Running apps Key Vault Storage
  2. Install AKS using CLI ## Create resourcegroup az group create

    --name ignitetour --location eastus ## Create Kubernetes cluster az aks create -g aksdemo -n ignitetour --generate-ssh-keys
  3. Provisioning We need an AKS cluster that: • supports multiple

    teams & environments • is only accessible through firewall • is highly available
  4. Multiple teams and environments One cluster per team or environment

    Physical Development Staging Production Team Morpheus Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Production Team Trinity Development, Staging Production Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Team Morpheus Team Trinity Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Staging Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Team Morpheus Team Trinity One cluster is reused over environments and teams Logical
  5. One dev cluster per team Development Team Morpheus Pod Pod

    Pod Pod Pod Pod Master branch Feature X branch Pod Pod Pod Pod Pod Pod Feature Y branch Pod Pod Pod Pod Pod Pod Staging Production Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod One cluster is reused over environments and teams Multiple teams and environments Team Morpheus Team Trinity Team Morpheus Team Trinity
  6. Azure Application Gateway & Firewall Web traffic load balancer URL

    based routing Endpoint in Azure or External • HTTP rate limiting • Scanner detection • Session fixation • LFI/RFI WAG v2 SSL termination or End-to-End SSL Automatic failover Web Application Firewall - Layer 7 OWASP rulesets • SQL Injection • Cross site scripting • Protocol violations • Generic attacks
  7. Azure Application Gateway & Firewall Rule Listener Http Setting Backend

    Pool Frontend IP Config Frontend Port Health Probe Protocol Protocol Port Public/Private Application Gateway 1 1 2 1 2 1 3 SSL certificate pfx 1 Backend certificate cer 1 HTTP/HTTPS HTTP/HTTPS Public IP-Address Private IP-Address
  8. West Europe VNET Traffic Manager AKS cluster Application Gateway &

    Firewall Public IP-Address North Europe VNET AKS cluster Application Gateway & Firewall Public IP-Address http://ignitetour.trafficmanager.net
  9. Azure Traffic Manager Traffic Manager http://ignitetour.trafficmanager.net DNS based load balancer

    Distribute traffic across regions Endpoint in Azure or External Automatic failover Routing based on • Performance • Weighted • Priority • Geographical
  10. Think ahead. Act now. Azure Front Door Global Load Balancer

    Traffic Manager + Application Gateway + CDN in one product Global Http Load Balancing URL redirection SSL termination WAF rules Static Content Cache
  11. West Europe VNET Traffic Manager AKS cluster Application Gateway &

    Firewall Public IP-Address North Europe VNET AKS cluster Application Gateway & Firewall Public IP-Address http://ignitetour.trafficmanager.net
  12. DNS Zone West Europe VNET Traffic Manager AKS cluster Application

    Gateway & Firewall Public IP-Address North Europe VNET AKS cluster Application Gateway & Firewall Public IP-Address http://www.ignitetour.cf http://ignitetour.trafficmanager.net
  13. Traffic Manager VNET AKS cluster Application Gateway & Firewall DNS

    Zone Log Analytics VNET AKS cluster Application Gateway & Firewall Log Analytics West Europe Public IP-Address North Europe Public IP-Address
  14. Traffic Manager VNET AKS cluster Application Gateway & Firewall Service

    Principal (AKS) DNS Zone Log Analytics VNET AKS cluster Application Gateway & Firewall Service Principal (AKS) Log Analytics Service Principal (Deploy) Key Vault Service Principal (Deploy) Key Vault West Europe Public IP-Address North Europe Public IP-Address
  15. Traffic Manager West Europe VNET North Europe AKS cluster Application

    Gateway & Firewall Public IP-Address Service Principal (AKS) DNS Zone Log Analytics VNET AKS cluster Application Gateway & Firewall Public IP-Address Service Principal (AKS) Log Analytics WORKSPACE_RESOURCEID=$(az resource show -g $RESOURCEGROUP_K8S -n $WORKSPACE_NAME --query 'id' -o tsv) AKSSUBNETID=$(az network vnet subnet show -g $RESOURCEGROUP_NETWORK -n aks-subnet --vnet-name $VNET_NAME --query id --output tsv) # Provision AKS az aks create -g $RESOURCEGROUP_K8S -n $AKS_NAME \ --kubernetes-version 1.11.5 \ --node-count $AKS_NODE_COUNT \ --node-vm-size $AKS_VM_SIZE \ --max-pods 110 \ --ssh-key-value "$(az keyvault secret show --name ssh-publickey --vault-name $KEYVAULT_ --service-principal $(az keyvault secret show --name spn-aks-id --vault-name $KEYVAULT_ --client-secret "$(az keyvault secret show --name spn-aks-password --vault-name $KEYVAU --network-plugin azure \ --vnet-subnet-id $AKSSUBNETID \ --docker-bridge-address 172.17.0.1/16 --dns-service-ip 10.2.0.10 --service-cidr 10.2.0.0/24 --enable-addons monitoring --workspace-resource-id $WORKSPACE_RESOURCEID \ --nodepool-name "linuxpool" \ --vm-set-type VirtualMachineScaleSets \ --node-zones 1 2 3 \ --load-balancer-sku standard + Network Contributor Service Principal (Deploy) Key Vault Service Principal (Deploy) Key Vault ## Create Kubernetes cluster az aks create -g aksdemo -n ignitetour --generate-ssh-keys
  16. Availability zones Availability Set (99.5%) Availability Zones (99.99%) Availability Zone

    1 Availability Zone 2 Availability Zone 3 Azure region €0.009 per GB inbound & outbound traffic
  17. Master nodes AKS offers 99.5% (SLO) Worker nodes AKS offers

    99.95% (Azure VM SLA in VMSS) With Availability Zones AKS offers 99.99% for 2+ VMs AKS Availability SLA Downtime per year Downtime per month 99,999% 5m 15.6s 26.3s 99,99% 52m 35s 4m 23s 99,95% 4h 22m 21m 54s 99,9% 8h 45m 43m 49s 99,5% 1d 19h 49m 3h 39m
  18. Master nodes AKS offers 99.5% (SLO) Worker nodes AKS offers

    99.95% (Azure VM SLA in VMSS) With Availability Zones AKS offers 99.99% for 2+ VMs AKS Availability SLA Downtime per year Downtime per month 99,999% 5m 15.6s 26.3s 99,99% 52m 35s 4m 23s 99,95% 4h 22m 21m 54s 99,9% 8h 45m 43m 49s 99,5% 1d 19h 49m 3h 39m
  19. Master nodes AKS offers 99.5% (SLO) Worker nodes AKS offers

    99.95% (Azure VM SLA in VMSS) With Availability Zones AKS offers 99.99% for 2+ VMs AKS Availability SLA Downtime per year Downtime per month 99,999% 5m 15.6s 26.3s 99,99% 52m 35s 4m 23s 99,95% 4h 22m 21m 54s 99,9% 8h 45m 43m 49s 99,5% 1d 19h 49m 3h 39m
  20. Master nodes AKS offers 99.5% (SLO) Worker nodes AKS offers

    99.95% (Azure VM SLA in VMSS) With Availability Zones AKS offers 99.99% for 2+ VMs AKS Availability SLA Downtime per year Downtime per month 99,999% 5m 15.6s 26.3s 99,99% 52m 35s 4m 23s 99,95% 4h 22m 21m 54s 99,9% 8h 45m 43m 49s 99,5% 1d 19h 49m 3h 39m 47 minutes
  21. Traffic Manager West Europe VNET North Europe AKS cluster Application

    Gateway & Firewall Public IP-Address Service Principal (AKS) Service Principal (Deploy) Key Vault DNS Zone Log Analytics VNET AKS cluster Application Gateway & Firewall Public IP-Address Service Principal (AKS) Service Principal (Deploy) Key Vault Log Analytics Service Principal (Registry) Container Registry Common AKS cluster Production Development
  22. Traffic Manager West Europe North Europe AKS cluster Key Vault

    DNS Zone AKS cluster Key Vault Container Registry Common
  23. Traffic Manager West Europe North Europe AKS cluster Key Vault

    DNS Zone AKS cluster Key Vault Container Registry Common SQL Azure MySQL Write Write SQL Azure MySQL Read Read
  24. Traffic Manager West Europe North Europe AKS cluster Key Vault

    DNS Zone AKS cluster Key Vault Container Registry Common Cosmos DB SQL Azure MySQL Write Write Write Cosmos DB SQL Azure MySQL Read Read Write
  25. Think ahead. Act now. Lessons learned regarding Provisioning Minimize time

    to market Availability and Firewalls requirements have complex consequences Use Key Vault for secrets Provisioning AKS (and everything related) for production is complex Automate everything!
  26. Think ahead. Act now. Configuration We need an AKS cluster

    that: • shows a dashboard • has Ingress configured • minimize maintenance
  27. kubectl proxy Dashboard kubectl create clusterrolebinding kubernetes-dashboard -n kube-system --clusterrole=cluster-admin

    --serviceaccount=kube-system:kubernetes-dashboard kubectl create clusterrolebinding kubernetes-dashboard -n kube-system --clusterrole=view --serviceaccount=kube-system:kubernetes-dashboard
  28. Ingress Pod 10.0.0.1 Pod 10.0.0.2 Pod 10.0.0.3 Service Type: ClusterIP

    Pod 10.0.0.4 Pod 10.0.0.5 Pod 10.0.0.6 Service Type: ClusterIP Pod 10.0.0.7 Pod 10.0.0.8 Pod 10.0.0.9 Service Type: ClusterIP Service Type: LoadBalancer Public or Private ip address: 37.17.208.21 Pod quay.io/kubernetes-ingress-controller/ nginx-ingress-controller:0.15.0 Ingress host: gaming. ignitetour.cf serviceName: gamingwebapp Ingress host: erp. ignitetour.cf serviceName: erpsvc Ingress host: www. ignitetour.cf/crm serviceName: crmsvc Label: erp Label: frontend Label: crm helm upgrade nginxingress --install stable/nginx-ingress --namespace kube-system -f internalingress.yaml Addon Http Application Routing
  29. Helm Helm is a tool for managing packages of pre-configured

    Kubernetes resources Linkerd Logstash Magento Mariadb Mongodb Mssql-linux Mysql Neo4j Newrelic Openvpn Postgresql Presto Prometheus Rabbitmq Redis Selenium Sonarqube Spinnaker Sysdig Tensorflow Traefik Wordpress Artifactory Cert-manager Consul Cassandra Couchdb Datadog Docker-registry Drupal Elasticsearch Ethereum fluentd Gitlab Grafana Hadoop Heapster Jenkins Joomla Kafka Kibana Kong Kubeless Lamp # helm service account and other needed resources kubectl apply -f helm.yaml # install helm locally & on the k8s cluster (tiller) helm init --service-account tiller-serviceaccount --upgrade --force-upgrade # latest charts helm repo update https://github.com/kubernetes/helm
  30. Automatic reboots of worker nodes Safe & automatic reboots of

    nodes with Kured https://github.com/weaveworks/kured helm upgrade kured --install stable/kured --namespace kured
  31. NodePool: Windows | k8s 1.14.8 Standard_DS2_v3 NodePool: Linux | k8s

    1.14.8 Standard_DS2_v2 NodePool: Linux | k8s 1.14.8 Standard_NC6 (GPU optimized) NodePool: Linux | k8s 1.14.7 Standard_DS2_v2 Nodepools
  32. 1. Provision new cluster next to existing cluster 2. Run

    your containers on the new cluster 3. Switch to new endpoint 4. Remove old cluster Upgrade cluster Using new cluster using NodePools az aks upgrade --name $clustername --resource-group $rg --Kubernetes-version 1.14.8 Risk Upgrade AKS 1. Upgrade the Control Plane to latest version az aks upgrade -n $clustername -g $rg -k 1.14.8 --control-plane-only 2. Add a new NodePool with latest version 3. Run your containers on the new NodePool 4. Remove old NodePool Cost & Complexity
  33. Kubernetes cluster Master Azure Container Instances (ACI) Pod Pod Pod

    Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Partly Managed IaaS Cluster Managed Cluster Kubernetes cluster Worker Node Worker Node Master Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Pod Kubectl get nodes NAME STATUS ROLES AGE VERSION aks-agentpool-17613766-0 Ready agent 4h v1.14.8 aks-agentpool-17613766-1 Ready agent 4h v1.14.8 aks-agentpool-17613766-2 Ready agent 4h v1.14.8 virtual-node-aci-linux Ready agent 4h v1.14.8
  34. Think ahead. Act now. AKS Worker Nodes are (managed) IaaS

    Use Helm to install applications to K8S Lessons learned regarding Configuration Dashboard doesn’t work out of the box Kubernetes as PaaS possible with Virtual Kubelet Kubernetes ecosystem is amazing Use NodePools for upgrades and different workloads
  35. Think ahead. Act now. Running Applications Applications within AKS cluster

    : • Automated deployments • Stateful applications • Monitoring • Keyvault
  36. Think ahead. Act now. Create Helm chart ## Create helm

    chart helm create podinfo Alternatives: Draft, Dev Spaces ## Install or upgrade helm chart helm upgrade podinfo --install ./podinfo --set image.tag=213 --set ingress.host=podinfo-master.dev-ignitetour.cf
  37. Pod Volumes Persistent Volumes Pod Persistent VolumeClaim StorageClass Persistent Volume

    Secret Pod Persistent VolumeClaim StorageClass Persistent Volume Secret Dynamic Static Secret Storage Account Azure File Azure Disk Storage Account Azure File Azure Disk Storage Account Azure File Azure Disk MC_...
  38. AzureDisk - ReadWriteOnce only - 1 Pod only - Slow

    in provisioning - Limited nr of disks - No direct access to data + Premium (SSD) for production AzureFile + All AccessModes + Shared by Pods + Fast in provisioning + Unlimited volumes + Direct access to data - SMB = slow Pod Volumes Secret Storage Account Azure File Azure Disk
  39. Secrets Goal: Read secrets from a Pod Use Key Vault

    Solution: Use Managed Identity (MI) to access the Key Vault Secrets in Kubernetes: base64 encoded
  40. Think ahead. Act now. No integration between AKS & Key

    Vault (yet) AAD Pod Identity https://github.com/Azure/aad-pod-identity/ Key Vault Flex Volume: https://github.com/Azure/kubernetes-keyvault-flexvol Secrets
  41. Think ahead. Act now. Lessons learned regarding Applications Use Helm

    to deploy your applications Use Key Vault for your application secrets Use Log Analytics + Container Solution for logging and monitoring Prevent stateful applications Automate everything: application deployment also!
  42. Wrap up Running Docker containers is cloud independent with Kubernetes,

    what about the other services… Applying DevOps to an application landscape using AKS takes a lot of knowledge and time for the whole team Automate everything Once you have a complete DevOps way of working for your applications, your team is Faster, Cheaper and Better