$30 off During Our Annual Pro Sale. View Details »

AzureBootcamp2022: Private Networking with Kubernetes - Why? How? by Andrew Craig and Pascal Zanitti

AzureBootcamp2022: Private Networking with Kubernetes - Why? How? by Andrew Craig and Pascal Zanitti

This session is one of the sessions of Azure Bootcamp Switzerland 2022.
www.azurebootcamp.ch

Based on real-world experiences - Hear about building K8s clusters with secure azure private networking. Learn how to overcome challenges around private networking in Azure, how DNS works in such a scenario and what this means to your current and future deployments. After the session, we happily deploy some beer to the attendees at the itnetX booth 😉!
🙂 ANDREW CRAIG ⚡️Azure Azure @ itnetX
🙂 PASCAL ZANITTI ⚡️ Cloud Developer @ itnetX

Check out Andrew at: https://www.linkedin.com/in/andrewdcraig/
Check out Pascal at: https://www.linkedin.com/in/pascal-zanitti/

Azure Zurich User Group
PRO

May 10, 2022
Tweet

More Decks by Azure Zurich User Group

Other Decks in Technology

Transcript

  1. 1 Private Networking with Kubernetes. Why? How?

  2. 2 Based on real-world experiences - Hear about building K8s

    clusters with secure azure private networking. Learn how to overcome challenges around private networking in Azure, how DNS works in such a scenario and what this means to your current and future deployments. After the session, we happily deploy some beer to the attendees at the itnetX booth 😉!
  3. 3 Agenda 1. Why Private Networking? 2. Kubernetes 3. Practical

    Experiences 4. Discussion about beer
  4. 4 1. Private Networking

  5. 5 Why Private Networking? o Security Considerations o Access Azure

    PaaS Services over a private endpoint in your virtual network. o Azure Backbone network o No public internet exposure o Access services within virtual network, allow peering and on-premises connection with ExpressRoute or VPN o Plenty of supported PaaS services o Microsoft recommends? o APL Geo-redundancy o You can use APL to deliver private link services o Legal Compliance o APL helps prevent data leaks Only exposes instances of services 5
  6. 6 VNET Injection o AKA Virtual Network integration AKA private

    access o Integrate supported services with VNETs o Usually dedicated shared subnets (delegation) o Some PaaS services have special subnet requirements o Private VNET addressing o No internet access o Check list of supported azure services (AKS, no ACR) 6
  7. 7 VNET Service Endpoints o Supports services with shared architecture

    o Integrate supported services with VNETs o Uses private IP address from the virtual network but still mapped to public address o Uses Azure backbone o Connect services to the virtual network, peered networks, on-premise o Secure the service itself or use NSGs o Check list of supported azure services (no AKS, ACR in preview) o Only VNET no on-premise o FREE!! 7
  8. 8 Azure Private Link o Private Endpoint and Private Link

    service o Private Endpoint is just what it sounds like o A network interface with a private IP address from the virtual network o Connect services to the virtual network, peered networks, on-premise, other Private Link services o Client-initiated connections with approvals o Must be in same region as network, not as service o Private-link resource o Destination target of the endpoint (see list of supported resource types) o Sub-resources (file, blob) require separate endpoints o With Azure Link Service access is controlled by approval workflows and network access is validated by the platform o Services provide Firewall o Availability (ACR, AKS) 8
  9. 9 Azure Private Link service o Private Link service is

    the reference o Can point a private endpoint at this service to expose private service provisioned behind standard load balancers 9
  10. 10 Private DNS resolution o Configure DNS to resolve the

    private endpoint address o Override existing Azure resource DNS configurations o The network interface contains the information required for DNS Configuration o Configure with hub network o Use a private DNS zone (recommended) privatelink.database.windows.net (privatelink.postgres.database.azure.com) o For on-premise workloads you need a DNS forwarder o Peered networks need to have access to the private DNS records 10
  11. 11 11

  12. 12 Kubernetes

  13. 13 Private Networking Considerations o From where should the AKS

    be accessible? o The AKS resource in the portal can be viewed from anywhere and some settings may be changed (like nodepools, given enough privileges) o Kubernetes Resources settings in the portal can only be viewed when accessing the portal from a linked VNET (also applies to kubectl) o Ways to access AKS for debugging o Bastion Host -> Jump Box VM -> Jump Box Pod -> Kubectl Creating a Simple Kubernetes Debug Pod: Premium Ubuntu Sleep Pod YAML|downey.io o From peered VNET 13
  14. 14 AKS Prerequisites and Considerations o AKS creates a resource

    group with a generic name if not explicitly specified o The private DNS Zone resides in this group and needs to be linked to other VNETs o Watch out for missing DNS entries in private DNS zone (Overview Pane) o The Private Link service is supported on Standard Azure Load Balancer only. o To use a custom DNS server, add the Azure DNS IP 168.63.129.16 as the upstream DNS server in the custom DNS server. o Create a private Azure Kubernetes Service cluster - Azure Kubernetes Service | Microsoft Docs 14
  15. 15 Sample Architecture 15 VNET azure container registry PostgreSQL flexible

    server privatelink.postgres.database.azure.com Kubernetes cluster
  16. 16 PostgreSQL Flexible Server - Private o Networking overview -

    Azure Database for PostgreSQL - Flexible Server | Microsoft Docs o Connect from Azure resources in the same virtual network to your flexible server by using private IP addresses o Requires Azure CNI networking type for AKS or Egress configuration o IP Whitelisting automated o Flexible Server public/private network access can NOT be toggled o Requires delegated subnet o More Details Private Networking Patterns in Azure Database for Postgres Flexible Server - Microsoft Tech Community o Restore: Great to create copies of resources before deployments or migrations via restore feature 16
  17. 17 Sample Application o As a sample application the Voting

    App from Microsoft was used o Kubernetes on Azure tutorial - Prepare an application - Azure Kubernetes Service | Microsoft Docs o We adjusted it to write results to the Postgre Database (and revolve around beer) o The AKS was deployed using Azure CNI as networking type so Pods can connect to the private Postgre instance o It can be accessed under: o http://20.126.205.89/ 17
  18. 18 Example extended Architecture 18

  19. 19 Extended Architecture – Fully private Environment 19 Private AKS

    cluster (uses Private Link) Private link ACR (Premium SKU) privatelink.database.windows.net privatelink.azurecr.io privatelink.<region>.azmk8s.io VNET Peered VNET WWW On Premise Site2Site tunnel Express Route DNS Forwarding
  20. 20 Tiering AKS o Premium Tier needed for AKS as

    well as ACR to use private networking 20
  21. 21 ACR Integration Considerations o Private ACR requires Premium Tier

    o Linking ACR to AKS requires high privileges o az aks update -n <aks> --attach-acr <acr> o “ERROR: Could not create a role assignment for ACR. Are you an Owner on this subscription?” o Could not create a role assignment for ACR. Are you an Owner on this subscription? · Issue #1517 · Azure/AKS · GitHub o Policies can also bite you at this step o Appservices have quirks with the WEBSITE_PULL_IMAGE_OVER_VNET setting needed o Limitation: App Service Private Endpoint pulling images to ACR with private Endpoint · Issue #88077 · MicrosoftDocs/azure- docs · GitHub o Also think about how images get pushed to the ACR o Like an external software development company pushing new versions (need peering) o Or “pull” new images via pipeline o ACR public/private networking access can be toggled o Set up private endpoint with private link - Azure Container Registry | Microsoft Docs 21
  22. 22 Pipeline Considerations o In a fully private environment, the

    Microsoft-hosted agents can not be used o Self-hosted agents are mandatory o Maintaining, updating, backups need to be kept in mind o Identity for Infra-Pipeline might need higher privileges for operations like ACR integration o IaC Code gets more complicated with private endpoints and private links 22
  23. 23 Practical experiences

  24. 24 Challenges with private networking o Some Resources support switching

    public/private access (ACR with private link), some don’t (PostgreSQL flexible Server with VNET Integration) o Workflow in Dev Environment might be impacted -> Talk to your developers o Or result in different infrastructure in Dev environment o Jump pods for SSH tunnels also possible o One-time Migrations might need additional effort due to inaccessibility o Need for additional peering o NSG not supported for private endpoints o Public preview available though o https://docs.microsoft.com/en-us/azure/private-link/disable-private-endpoint-network-policy 24
  25. 25 DB Migrations in a private environment example o Example:

    The DB on a private Postgres flexible Server needs to be migrated from one Region to a new Resource in a different Region o Challenge: You can not use global VNET peering due to regulations 25
  26. 26 CSI Driver o Use the Azure Key Vault Provider

    for Secrets Store CSI Driver for Azure Kubernetes Service secrets - Azure Kubernetes Service | Microsoft Docs o Mounts secrets, keys, and certificates to a pod by using a CSI volume o Distribution of certificates can be a requirement in private environments with private DNS o Syncs with Kubernetes secrets o az aks enable-addons --addons azure-keyvault-secrets-provider --name myAKSCluster --resource-group myResourceGroup o kubectl exec busybox-secrets-store-inline -- ls /mnt/secrets-store/ o Note: When the Azure Key Vault Provider for Secrets Store CSI Driver is enabled, it updates the pod mount and the Kubernetes secret that's defined in the secretObjects field of SecretProviderClass. It does so by polling for changes periodically, based on the rotation poll interval you've defined. The default rotation poll interval is 2 minutes. 26
  27. 27 Maintenance Mode for Kubernetes Applications o Think about maintenance

    modes for applications running on Kubernetes o Example: During the deployment of a new version in Prod some DB migrations need to run o Challenge: During the migration no DB changes through the app should be possible o “Easy way” to down an application is to delete the deployment which also deletes pods o Downside: Application frontend becomes unresponsive/fails o “One better way” point to a maintenance page o There are many different ways to achieve this, depending on the ingress setup and application 27
  28. 28 Discussion (suggestions) o How can we achieve the same

    with enough security to keep ITSec happy? o Is private networking as cloudy as our lovely unfiltered Hoppy Pilsner? o Do Azure Private Endpoints leave you bitter like a West Coast style IPA? 28
  29. 29 Resources o Private Networking Patterns in Azure Database for

    Postgres Flexible Server - Microsoft Tech Community o Networking overview - Azure Database for PostgreSQL - Flexible Server | Microsoft Docs o Kubernetes on Azure tutorial - Prepare an application - Azure Kubernetes Service | Microsoft Docs o Create a private Azure Kubernetes Service cluster - Azure Kubernetes Service | Microsoft Docs o Virtual network for Azure services | Microsoft Docs o Azure virtual network service endpoints | Microsoft Docs o Azure Private Link availability | Microsoft Docs o What is a private endpoint? | Microsoft Docs 29
  30. 30 Industriestrasse 46 8152 Glattbrugg Contact itnetX Zurich itnetX Berne

    itnetX Lausanne Kirchgasse 22 3097 Liebefeld c/o Swisscom Avenue de Provence 4 1007 Lausanne info@itnetX.ch | www.itnetX.ch + 41 848 99 00 00
  31. 31