AKS-Series 5 [Video]: Network & Security Best Practices in AKS - Mohammed Nofal

Stream Link: https://www.youtube.com/watch?v=r8DD8bnEpNM
Meetup Link: https://www.meetup.com/de-DE/microsoft-azure-zurich-user-group/events/285626132/

In this session we will delve into how you can build a secure and compliant clusters in Azure Kubernetes Service, during the session we will cover the below topics:

Image Build Security
Registry Security
Cluster and Nodes Security
Network Security
Identity and access management

Bio:
Mohammad works as Cloud Native Apps Architect at Microsoft and is part of the Azure Global Black Belt team. He is focused on everything with Containers and Kubernetes on Azure and the surrounding cloud native ecosystem. Previously he worked 4 years at Amazon Web Services in different capacities, and before that 10 years in the telecommunication industry building distributed systems.

Links:
Github: @mohmdnofal
Twitter: @mohmd_nofal
Linkedin: https://www.linkedin.com/in/mnofal/

Transcript

1. Network and security best practices Azure Kubernetes Service
2. Mohammad Nofal Cloud Native Apps Architect – Global Black Belt Team – EMEA @mohmd_nofal
3. Agenda • Network Plugins • Network Policies • Ingress and Egress traffic • API Server Visibility Models • Identity and access management • Security workloads
4. What Network Plugin Should I Use?
5. AKS Network Plugins • AKS supported network modes • Kubenet • Azure CNI v1 • Azure CNI v2 Dynamic Allocation of IPs • Bring your own CNI -> CNI = None
6. Kubenet • Implements a network bridge in each node (cbr0) • Pods are assigned a logical subnet • Pod to Pod cross node communication is handled automatically by Azure by implementing a route table, each node will add one route entry to the route table • Simple and stable • Pods and Nodes are on different subnets • Supports Calico Network Policy and Linux Nodes Only (No Windows Support) • VNET subnet sizing Number of IPs = No. of Nodes + Node Surge Count + Scale Out Nodes + No. of ILB Services • Example 10 node cluster, with surge size 2, scale out to 12, and 200 pods, 10 ILB SVCs 10 + 2 + 2 + 10= 24 IPs or /27 Subnet
7. Azure CNI V1 • Implementation of the up-stream Container Network Interface (CNI) specification • Native to Azure, pods are attached to the virtual network • Pods and Nodes are on the same subnet • Pods can assume VM NIC performance, no IP forwarding involved • Supports Calico and Azure CNI Network Policies • VNET subnet sizing Number of IPs = No. of Nodes + Node Surge Count + Scale Out Nodes + No. of ILB Services + (Max No. of Pods Per Node * No. of Nodes) • Example 10 node cluster, with surge size 2, scale out to 12, and 200 pods, 10 ILB SVCs 10 + 2 + 2 + 10 + (14 * 30) = 442 IPs or /23 Subnet
8. Azure CNI v2 – Dynamic Allocation of IPs (GA) • Designed to overcome the number of IPs required for CNI V1 • Dynamic IP allocation for pods, only allocates IPs when the pods are running • Nodes and Pods live in different subnets within the same VNET and can be scaled individually • Allows to have a different Network security policies for nodes and pods, i.e. Nodes are allowed to access update.ubuntu.com but pods can’t • Multiple subnets per cluster support • Multiple clusters can share the same subnet within a VNET • Only Linux clusters are supported for now!
9. Azure CNI vNext (Preview) VNET Forecasting • Number of IPs • Nodes Subnet: No. of Nodes + Node Surge Count + Scale Out Nodes • Pods Subnet: No. of Pods + ILB Services • Example 10 node cluster, with surge size 2, scale out to 12, and 200 pods, 10 ILB SVCs • Nodes Subnet: 10+2+2=14 or /27 Subnet (taking into account the reserved IPs) • Pods Subnet: 200+10=210 or /24 Subnet
10. Bring your own CNI [Preview] • A new mode to simplify bringing your own 3rd party CNI like Calico or Cilium • Nodes come up with no CNI configuration • No need to reboot pods after installing the CNI • East-West/Pods networking becomes your CNI provider responsibility • Support cases related to pod networking should be handled with the CNI provider az aks create \ -l <Region> \ -g <ResourceGroupName> \ -n <ClusterName> \ --network-plugin none
11. Network Plugins Comparison CNI v2 CNI v1 Kubenet BYO CNI When to use Always (unless constrained on IPs) Only use if there is a good reason not to use V2 Use when you are constrained on IPs, network team only provide small subnets Looking for specific features which aren’t provided by the managed plugins Performance VM Like VM Like Good Performance (IP Forwarding has an overhead) Good Performance NW Policy Support Calico and Azure Network Policy Calico and Azure Network Policy Calico Native Support Best Best Best CNI Provider Resbonsilibty Integrations with Azure Services Full Full Lags behind, Partially Full Test Max Cluster Size As big as the subnet As big as the subnet 400 Nodes (Route Table Limits) 200 in case of dual stack Depends
12. AKS Dual Stack Support - IPv6 • Dual Stack graduated to Stable in K8s 1.23 • Only Kubenet support for now • Pod IPs will be NATed to Node IPs based on their origin IPv4<->IPv4 | IPv6<->IPv6 • Dual stack clusters are limited to 200 nodes, (each node requires 2 route tables -> max route tables per vnet =400) • AKS support made it easy to deploy with the introduction of the “ip-families” flag az aks create \ -l <Region> \ -g <ResourceGroupName> \ -n <ClusterName> \ --ip-families ipv4,ipv6
13. Up and Coming - Network Modes • Migration path for CNI V1 to CNI V2 • CNI V2 for windows nodes
14. Network Policies
15. Network Policies in AKS kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: backend-policy namespace: development spec: podSelector: matchLabels: app: webapp role: backend ingress: - from: - namespaceSelector: matchLabels: purpose: development podSelector: matchLabels: app: webapp role: frontend
16. Calico Extra Features • Now supports windows nodes • Global Network Set: group network CIDRs under a set that can be referenced in a policy • Global Network Set: group network CIDRs under a set that can be referenced in a policy • Host Endpoint: apply policies to the host Network Interface • Ordering of rules
17. Calico Global Network Policy Example $ cat <<EOF | kubectl apply -f - apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: deny-circle-blue spec: selector: color == 'red' ingress: - action: Deny protocol: TCP source: selector: color == 'blue' namespaceSelector: shape == 'circle’ EOF
18. FQDN/L7 Based Policies with OSM • Calico and Azure Network Policy are layer4, if you need layer 7 network policies you can use Open Service Mesh Egress Policies kubectl apply -f - <<EOF kind: Egress apiVersion: policy.openservicemesh.io/v1alpha1 metadata: name: httpbin-80 namespace: curl spec: sources: - kind: ServiceAccount name: curl namespace: curl hosts: - httpbin.org ports: - number: 80 protocol: http EOF
19. 3rd Party Network Policies • Cilium, Tigera Enterprise/Calico, and Some Service Meshes allow you to create L7 network policies or FQDN based policies apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "fqdn" spec: endpointSelector: matchLabels: org: prod egress: - toFQDNs: - matchPattern: "*.ubuntu.com" - toEndpoints: - matchLabels: "k8s:io.kubernetes.pod.namespace": kube-system "k8s:k8s-app": kube-dns toPorts: - ports: - port: "53" protocol: ANY rules: dns: - matchPattern: "*"
20. Network Plugins Comparison Azure CNI Network Policy Calico 3rd Party What is it Developed by Azure Networking team Developed by Tigera team and its open source Any 3rd party i.e. Cilium Supported Network Policies Azure CNI Azure CNI, Kubenet, 3rd party distributions Any Support Supported by Azure NW Engineering Community support On your own or you buy it Advantages One stop shop for support +combine with OSM to get L7 support Most wildly used network policy with the largest community and ships with extra neat features +combine with OSM to get L7 support Varies i.e. L7 network policies
21. Controlling Egress and Ingress Traffic
22. Outbound-type=userDefinedRouting • AKS now allows to configure the outbound-type for the Cluster egress traffic • The parameter will allow you to choose between ‘Loadbalancer’ current default behavior, or • Outbound-type=userDefinedRouting , in this model the assumption is that you have a NVA/FW in place which you can route all the traffic to, and this NVA will handle the egress to the internet, or • Outbound-type=managedNATGateway
23. Egress to Azure Firewall OutboundType= userDefinedRouting Internet ACR Azure required services SLB AKS- HCP Azure Managed Account Kubectl CI /CD Toolchain Egress traffic ingress traffic Peering AKS Spoke VNET AKS1-VMSS1- NicX Worker-Subnet1 NSG :9000, :443 NSG HUB VNET Worker-Subnet1 UDR Azure Firewall Service Endpoint Traffic AZFW-PublicIP
24. Dedicated Ingress subnet • Dedicating a subnet for the ingress traffic provides clear seperation of concerns between ingress and egress traffic • You can add a network security group on the subnet to add more custom security rules for the ingress traffic • Ingress can be owned by features/service team and the WAF (web application firewall) can serve as the contract between the feature and security teams • Using a WAF instead of a Firewall provides better security for Web Applications and you gain the benefits of a Layer7 loadbalancer apiVersion: v1 kind: Service metadata: name: internal-app annotations: service.beta.kubernetes.io/azure-load- balancer-internal: "true" service.beta.kubernetes.io/azure-load- balancer-internal-subnet: "apps-subnet" spec: type: LoadBalancer ports: - port: 80 selector: app: internal-app
25. AKS Ingress Subnet + AZ Firewall Internet ACR Azure required services SLB Egress – SLB Public IP AKS- HCP Azure Managed Account Kubectl CI /CD Toolchain Peering On Prem Network Egress traffic ingress traffic Peering AKS Spoke VNET AKS1-VMSS1- NicX Worker-Subnet1 NSG :9000, :443 NSG HUB VNET Worker-Subnet1 AzFWInternalIP UDR Ingress-Subnet Ingress – SLB Private IP ILB Service Endpoint Azure MySQL NSG Service Endpoint Traffic Azure Firewall AZFW-PublicIP
26. AKS Ingress Subnet + WAF + AZ Firewall Internet ACR Azure required services SLB Egress – SLB Public IP AKS- HCP Azure Managed Account Kubectl CI /CD Toolchain Peering On Prem Network Egress traffic ingress traffic Peering AKS Spoke VNET AKS1-VMSS1- NicX Worker-Subnet1 NSG :9000, :443 NSG HUB VNET Worker-Subnet1 AzFWInternalIP UDR Ingress-Subnet Service Endpoint Azure MySQL NSG Service Endpoint Traffic Azure Firewall AZFW-PublicIP App GW Private IP App GW Public IP
27. Outbound-type =ManagedNATGateway [GA]
28. Subnet Per Node Pool [GA] az aks nodepool add \ --resource-group myResourceGroup \ --cluster-name myAKSCluster \ --name mynodepool \ --node-count 3 \ --vnet-subnet-id <YOUR_SUBNET_RESOURCE_ID>
29. Subnet Per Node Pools Internet ACR Azure required services AKS- HCP Azure Managed Account Peering On Prem Network Egress traffic ingress traffic Peering AKS SPOKE VNET AKSNP1-VMSS1- NicX NodePool1-Subnet1 NSG Ingress-Subnet NSG Kubectl CI /CD Toolchain HUB VNET Worker-Subnet1 UDR Private Linkt Azure MySQL Private Link Azure Firewall Ingress – SLB Private IP AKSNP2-VMSS2- NicX NSG NodePool2-Subnet2 UDR AZFW-PublicIP
30. AKS Private Clusters
31. AKS Private Cluster AKS PE AKS private Link Internet ACR Azure required services SLB APIServer Azure Managed Account jumphost CI /CD Toolchain Egress traffic ingress traffic Peering AKS Spoke VNET AKS1-VMSS1- NicX Worker-Subnet1 NSG HUB VNET Worker-Subnet1 UDR UDR Azure Firewall Azure MySQL Service Endpoint Traffic Private Link Traffic AZFW-PublicIP AzureMysql private Link
32. Private AKS DNS Zone Options • System==default • AKS creates a private DNS zone with an FQDN this will be used by the nodes • AKS creates a public FQDN with an A record that resolves to private IP, to simplify connecting from other connected networks [Can be disabled at provisioning or during run time] • Bring your own • Approved format: privatelink.<region>.azmk8s.io or <subzone>.privatelink.<region>.azmk8s.io • You pass the resource ID and the FQDN subdomain • AKS will update the entry (FQDN subdomain) in the private zone with the API server private IP • The zone can be pre-linked with the HUB in order to ensure smooth resolution • None • Defaults to Public DNS that points to a private IP • AKS won’t create a Private DNS zone No need to setup your own DNS infrastrucrure
33. AKS Command Invoke – Manage Private Clusters az aks command invoke \ --resource-group $RG \ --name $AKS_CLUSTER_NAME \ --command "kubectl get pods -n kube-system && kubectl get pods -n aks-command" command started at 2022-06-14 11:33:01+00:00, finished at 2022- 06-14 11:33:02+00:00 with exitcode=0
34. Closing thoughts on private clusters • Private clusters are great at hiding your API server from the public internet, however, introduces its own challenges with having to design all the surrounding services to be privately connected to your cluster • AKS Public API server IP can be secured by using Authorized IP Ranges to block all unwanted traffic, which is easier to manage • can be added at any stage of the cluster lifecycle az aks create \ --resource-group myResourceGroup \ --name myAKSCluster \ --node-count 1 \ --api-server-authorized-ip-ranges 73.140.245.0/24 Or az aks update \ --resource-group myResourceGroup \ --name myAKSCluster \ --api-server-authorized-ip-ranges 73.140.245.0/24
35. Identity and Access Management
36. az aks create \ --resource-group myResourceGroup \ --name myManagedCluster \ --enable-managed-identity \ --assign-identity <identity-resource-id> \ --assign-kubelet-identity <kubelet-identity- resource-id>
37. User Identities https://github.com/Azure/kubelogin
38. Azure RBAC for Kubernetes Authorization az aks create -g MyResourceGroup -n MyManagedCluster --enable-aad --enable-azure-rbac #Assign reader role on ”testnamespace” for a given user az role assignment create --role "Azure Kubernetes Service RBAC Reader" --assignee <AAD-ENTITY-ID> --scope $AKS_ID/namespaces/testnamespace
39. Disable local accounts [GA] az aks get-credentials -n $AKS_CLUSTER_NAME -g $RG –admin az create \ -g <resource-group>\ -n <cluster-name> \ --enable-aad \ --enable-azure-rbac \ --aad-admin-group-object-ids <aad- group-id> \ --disable-local
40. What's happening with Pod Identity v1?
41. Workload Identity • Removes the scale and performance issues that existed for identity assignment • Supports Kubernetes clusters hosted in any cloud or on-premises • Supports both Linux and Windows workloads • Removes the need for Custom Resource Definitions and pods that intercept traffic • Avoids the complication and error-prone installation steps such as cluster role assignment from the previous iteration. • Relies on Identity Federation in AAD • Only supported for Azure AD Applications, managed identity support is in private preview now
42. How it works
43. The path forward for Pod and Workload identity
44. Microservice Identities - mTLS
45. Open Service Mesh [GA] az create \ -n \ -g <myosmaksgroup>\ --enable-managed-identity -a
46. Securing Applications
47. Azure Defender for Container – Vulnerability assessment
48. Azure Defender for Container – Run Time Security
49. Azure Policy for Kubernetes
50. Secret Management
51. Secret store CSI Driver For Azure KeyVault
52. KMS ETCD encryption [Preview] az aks create \ --name --resource-group --assign-identity --enable-azure-keyvault-kms --azure-keyvault-kms-key-id
53. Resources https://github.com/mohmdnofal/aks-best- practices https://docs.microsoft.com/en-us/azure/aks https://docs.microsoft.com/en-us/azure/aks/best-practices https://github.com/Azure/AKS/projects/1 https://releases.aks.azure.com/webpage/index.html
54. Thank you!