AzureBootcamp2022: Private Networking with Kubernetes - Why? How? by Andrew Craig and Pascal Zanitti

This session is one of the sessions of Azure Bootcamp Switzerland 2022.
www.azurebootcamp.ch

Based on real-world experiences - Hear about building K8s clusters with secure azure private networking. Learn how to overcome challenges around private networking in Azure, how DNS works in such a scenario and what this means to your current and future deployments. After the session, we happily deploy some beer to the attendees at the itnetX booth 😉!
🙂 ANDREW CRAIG ⚡️Azure Azure @ itnetX
🙂 PASCAL ZANITTI ⚡️ Cloud Developer @ itnetX

Check out Andrew at: https://www.linkedin.com/in/andrewdcraig/
Check out Pascal at: https://www.linkedin.com/in/pascal-zanitti/

Transcript

1. 1 Private Networking with Kubernetes. Why? How?
2. 2 Based on real-world experiences - Hear about building K8s clusters with secure azure private networking. Learn how to overcome challenges around private networking in Azure, how DNS works in such a scenario and what this means to your current and future deployments. After the session, we happily deploy some beer to the attendees at the itnetX booth 😉!
3. 3 Agenda 1. Why Private Networking? 2. Kubernetes 3. Practical Experiences 4. Discussion about beer
4. 4 1. Private Networking
5. 5 Why Private Networking? o Security Considerations o Access Azure PaaS Services over a private endpoint in your virtual network. o Azure Backbone network o No public internet exposure o Access services within virtual network, allow peering and on-premises connection with ExpressRoute or VPN o Plenty of supported PaaS services o Microsoft recommends? o APL Geo-redundancy o You can use APL to deliver private link services o Legal Compliance o APL helps prevent data leaks Only exposes instances of services 5
6. 6 VNET Injection o AKA Virtual Network integration AKA private access o Integrate supported services with VNETs o Usually dedicated shared subnets (delegation) o Some PaaS services have special subnet requirements o Private VNET addressing o No internet access o Check list of supported azure services (AKS, no ACR) 6
7. 7 VNET Service Endpoints o Supports services with shared architecture o Integrate supported services with VNETs o Uses private IP address from the virtual network but still mapped to public address o Uses Azure backbone o Connect services to the virtual network, peered networks, on-premise o Secure the service itself or use NSGs o Check list of supported azure services (no AKS, ACR in preview) o Only VNET no on-premise o FREE!! 7
8. 8 Azure Private Link o Private Endpoint and Private Link service o Private Endpoint is just what it sounds like o A network interface with a private IP address from the virtual network o Connect services to the virtual network, peered networks, on-premise, other Private Link services o Client-initiated connections with approvals o Must be in same region as network, not as service o Private-link resource o Destination target of the endpoint (see list of supported resource types) o Sub-resources (file, blob) require separate endpoints o With Azure Link Service access is controlled by approval workflows and network access is validated by the platform o Services provide Firewall o Availability (ACR, AKS) 8
9. 9 Azure Private Link service o Private Link service is the reference o Can point a private endpoint at this service to expose private service provisioned behind standard load balancers 9
10. 10 Private DNS resolution o Configure DNS to resolve the private endpoint address o Override existing Azure resource DNS configurations o The network interface contains the information required for DNS Configuration o Configure with hub network o Use a private DNS zone (recommended) privatelink.database.windows.net (privatelink.postgres.database.azure.com) o For on-premise workloads you need a DNS forwarder o Peered networks need to have access to the private DNS records 10
11. 11 11
12. 12 Kubernetes
13. 13 Private Networking Considerations o From where should the AKS be accessible? o The AKS resource in the portal can be viewed from anywhere and some settings may be changed (like nodepools, given enough privileges) o Kubernetes Resources settings in the portal can only be viewed when accessing the portal from a linked VNET (also applies to kubectl) o Ways to access AKS for debugging o Bastion Host -> Jump Box VM -> Jump Box Pod -> Kubectl Creating a Simple Kubernetes Debug Pod: Premium Ubuntu Sleep Pod YAML|downey.io o From peered VNET 13
14. 14 AKS Prerequisites and Considerations o AKS creates a resource group with a generic name if not explicitly specified o The private DNS Zone resides in this group and needs to be linked to other VNETs o Watch out for missing DNS entries in private DNS zone (Overview Pane) o The Private Link service is supported on Standard Azure Load Balancer only. o To use a custom DNS server, add the Azure DNS IP 168.63.129.16 as the upstream DNS server in the custom DNS server. o Create a private Azure Kubernetes Service cluster - Azure Kubernetes Service | Microsoft Docs 14
15. 15 Sample Architecture 15 VNET azure container registry PostgreSQL flexible server privatelink.postgres.database.azure.com Kubernetes cluster
16. 16 PostgreSQL Flexible Server - Private o Networking overview - Azure Database for PostgreSQL - Flexible Server | Microsoft Docs o Connect from Azure resources in the same virtual network to your flexible server by using private IP addresses o Requires Azure CNI networking type for AKS or Egress configuration o IP Whitelisting automated o Flexible Server public/private network access can NOT be toggled o Requires delegated subnet o More Details Private Networking Patterns in Azure Database for Postgres Flexible Server - Microsoft Tech Community o Restore: Great to create copies of resources before deployments or migrations via restore feature 16
17. 17 Sample Application o As a sample application the Voting App from Microsoft was used o Kubernetes on Azure tutorial - Prepare an application - Azure Kubernetes Service | Microsoft Docs o We adjusted it to write results to the Postgre Database (and revolve around beer) o The AKS was deployed using Azure CNI as networking type so Pods can connect to the private Postgre instance o It can be accessed under: o http://20.126.205.89/ 17
18. 18 Example extended Architecture 18
19. 19 Extended Architecture – Fully private Environment 19 Private AKS cluster (uses Private Link) Private link ACR (Premium SKU) privatelink.database.windows.net privatelink.azurecr.io privatelink.<region>.azmk8s.io VNET Peered VNET WWW On Premise Site2Site tunnel Express Route DNS Forwarding
20. 20 Tiering AKS o Premium Tier needed for AKS as well as ACR to use private networking 20
21. 21 ACR Integration Considerations o Private ACR requires Premium Tier o Linking ACR to AKS requires high privileges o az aks update -n <aks> --attach-acr <acr> o “ERROR: Could not create a role assignment for ACR. Are you an Owner on this subscription?” o Could not create a role assignment for ACR. Are you an Owner on this subscription? · Issue #1517 · Azure/AKS · GitHub o Policies can also bite you at this step o Appservices have quirks with the WEBSITE_PULL_IMAGE_OVER_VNET setting needed o Limitation: App Service Private Endpoint pulling images to ACR with private Endpoint · Issue #88077 · MicrosoftDocs/azure- docs · GitHub o Also think about how images get pushed to the ACR o Like an external software development company pushing new versions (need peering) o Or “pull” new images via pipeline o ACR public/private networking access can be toggled o Set up private endpoint with private link - Azure Container Registry | Microsoft Docs 21
22. 22 Pipeline Considerations o In a fully private environment, the Microsoft-hosted agents can not be used o Self-hosted agents are mandatory o Maintaining, updating, backups need to be kept in mind o Identity for Infra-Pipeline might need higher privileges for operations like ACR integration o IaC Code gets more complicated with private endpoints and private links 22
23. 23 Practical experiences
24. 24 Challenges with private networking o Some Resources support switching public/private access (ACR with private link), some don’t (PostgreSQL flexible Server with VNET Integration) o Workflow in Dev Environment might be impacted -> Talk to your developers o Or result in different infrastructure in Dev environment o Jump pods for SSH tunnels also possible o One-time Migrations might need additional effort due to inaccessibility o Need for additional peering o NSG not supported for private endpoints o Public preview available though o https://docs.microsoft.com/en-us/azure/private-link/disable-private-endpoint-network-policy 24
25. 25 DB Migrations in a private environment example o Example: The DB on a private Postgres flexible Server needs to be migrated from one Region to a new Resource in a different Region o Challenge: You can not use global VNET peering due to regulations 25
26. 26 CSI Driver o Use the Azure Key Vault Provider for Secrets Store CSI Driver for Azure Kubernetes Service secrets - Azure Kubernetes Service | Microsoft Docs o Mounts secrets, keys, and certificates to a pod by using a CSI volume o Distribution of certificates can be a requirement in private environments with private DNS o Syncs with Kubernetes secrets o az aks enable-addons --addons azure-keyvault-secrets-provider --name myAKSCluster --resource-group myResourceGroup o kubectl exec busybox-secrets-store-inline -- ls /mnt/secrets-store/ o Note: When the Azure Key Vault Provider for Secrets Store CSI Driver is enabled, it updates the pod mount and the Kubernetes secret that's defined in the secretObjects field of SecretProviderClass. It does so by polling for changes periodically, based on the rotation poll interval you've defined. The default rotation poll interval is 2 minutes. 26
27. 27 Maintenance Mode for Kubernetes Applications o Think about maintenance modes for applications running on Kubernetes o Example: During the deployment of a new version in Prod some DB migrations need to run o Challenge: During the migration no DB changes through the app should be possible o “Easy way” to down an application is to delete the deployment which also deletes pods o Downside: Application frontend becomes unresponsive/fails o “One better way” point to a maintenance page o There are many different ways to achieve this, depending on the ingress setup and application 27
28. 28 Discussion (suggestions) o How can we achieve the same with enough security to keep ITSec happy? o Is private networking as cloudy as our lovely unfiltered Hoppy Pilsner? o Do Azure Private Endpoints leave you bitter like a West Coast style IPA? 28
29. 29 Resources o Private Networking Patterns in Azure Database for Postgres Flexible Server - Microsoft Tech Community o Networking overview - Azure Database for PostgreSQL - Flexible Server | Microsoft Docs o Kubernetes on Azure tutorial - Prepare an application - Azure Kubernetes Service | Microsoft Docs o Create a private Azure Kubernetes Service cluster - Azure Kubernetes Service | Microsoft Docs o Virtual network for Azure services | Microsoft Docs o Azure virtual network service endpoints | Microsoft Docs o Azure Private Link availability | Microsoft Docs o What is a private endpoint? | Microsoft Docs 29
30. 30 Industriestrasse 46 8152 Glattbrugg Contact itnetX Zurich itnetX Berne itnetX Lausanne Kirchgasse 22 3097 Liebefeld c/o Swisscom Avenue de Provence 4 1007 Lausanne info@itnetX.ch | www.itnetX.ch + 41 848 99 00 00