Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AzureBootcamp2022: Private Networking with Kubernetes - Why? How? by Andrew Craig and Pascal Zanitti

AzureBootcamp2022: Private Networking with Kubernetes - Why? How? by Andrew Craig and Pascal Zanitti

This session is one of the sessions of Azure Bootcamp Switzerland 2022.
www.azurebootcamp.ch

Based on real-world experiences - Hear about building K8s clusters with secure azure private networking. Learn how to overcome challenges around private networking in Azure, how DNS works in such a scenario and what this means to your current and future deployments. After the session, we happily deploy some beer to the attendees at the itnetX booth 😉!
🙂 ANDREW CRAIG ⚡️Azure Azure @ itnetX
🙂 PASCAL ZANITTI ⚡️ Cloud Developer @ itnetX

Check out Andrew at: https://www.linkedin.com/in/andrewdcraig/
Check out Pascal at: https://www.linkedin.com/in/pascal-zanitti/

More Decks by Azure Zurich User Group

Other Decks in Technology

Transcript

  1. 1
    Private Networking with Kubernetes. Why? How?

    View full-size slide

  2. 2
    Based on real-world experiences - Hear
    about building K8s clusters with secure
    azure private networking. Learn how to
    overcome challenges around private
    networking in Azure, how DNS works
    in such a scenario and what this means
    to your current and future deployments.
    After the session, we happily deploy
    some beer to the attendees at the itnetX
    booth 😉!

    View full-size slide

  3. 3
    Agenda
    1. Why Private Networking?
    2. Kubernetes
    3. Practical Experiences
    4. Discussion about beer

    View full-size slide

  4. 4
    1. Private Networking

    View full-size slide

  5. 5
    Why Private Networking?
    o Security Considerations
    o Access Azure PaaS Services over a private endpoint in your virtual network.
    o Azure Backbone network
    o No public internet exposure
    o Access services within virtual network, allow peering and on-premises connection with ExpressRoute or VPN
    o Plenty of supported PaaS services
    o Microsoft recommends?
    o APL Geo-redundancy
    o You can use APL to deliver private link services
    o Legal Compliance
    o APL helps prevent data leaks
    Only exposes instances of services
    5

    View full-size slide

  6. 6
    VNET Injection
    o AKA Virtual Network
    integration AKA private access
    o Integrate supported services
    with VNETs
    o Usually dedicated shared
    subnets (delegation)
    o Some PaaS services have
    special subnet requirements
    o Private VNET addressing
    o No internet access
    o Check list of supported azure
    services (AKS, no ACR)
    6

    View full-size slide

  7. 7
    VNET Service Endpoints
    o Supports services with shared architecture
    o Integrate supported services with VNETs
    o Uses private IP address from the virtual
    network but still mapped to public address
    o Uses Azure backbone
    o Connect services to the virtual network,
    peered networks, on-premise
    o Secure the service itself or use NSGs
    o Check list of supported azure services (no
    AKS, ACR in preview)
    o Only VNET no on-premise
    o FREE!!
    7

    View full-size slide

  8. 8
    Azure Private Link
    o Private Endpoint and Private Link service
    o Private Endpoint is just what it sounds like
    o A network interface with a private IP
    address from the virtual network
    o Connect services to the virtual network,
    peered networks, on-premise, other
    Private Link services
    o Client-initiated connections with approvals
    o Must be in same region as network, not
    as service
    o Private-link resource
    o Destination target of the endpoint (see list
    of supported resource types)
    o Sub-resources (file, blob) require separate
    endpoints
    o With Azure Link Service access is
    controlled by approval workflows and
    network access is validated by the
    platform
    o Services provide Firewall
    o Availability (ACR, AKS)
    8

    View full-size slide

  9. 9
    Azure Private Link
    service
    o Private Link service is the
    reference
    o Can point a private endpoint at
    this service to expose private
    service provisioned behind
    standard load balancers
    9

    View full-size slide

  10. 10
    Private DNS resolution
    o Configure DNS to resolve the private endpoint address
    o Override existing Azure resource DNS configurations
    o The network interface contains the information required for DNS Configuration
    o Configure with hub network
    o Use a private DNS zone (recommended) privatelink.database.windows.net
    (privatelink.postgres.database.azure.com)
    o For on-premise workloads you need a DNS forwarder
    o Peered networks need to have access to the private DNS records
    10

    View full-size slide

  11. 12
    Kubernetes

    View full-size slide

  12. 13
    Private Networking Considerations
    o From where should the AKS be accessible?
    o The AKS resource in the portal can be viewed from anywhere and some settings may be changed (like nodepools, given
    enough privileges)
    o Kubernetes Resources settings in the portal can only be viewed when accessing the portal from a linked VNET (also
    applies to kubectl)
    o Ways to access AKS for debugging
    o Bastion Host -> Jump Box VM -> Jump Box Pod -> Kubectl
    Creating a Simple Kubernetes Debug Pod: Premium Ubuntu Sleep Pod YAML|downey.io
    o From peered VNET
    13

    View full-size slide

  13. 14
    AKS Prerequisites and Considerations
    o AKS creates a resource group with a generic name if not explicitly specified
    o The private DNS Zone resides in this group and needs to be linked to other VNETs
    o Watch out for missing DNS entries in private DNS zone (Overview Pane)
    o The Private Link service is supported on Standard Azure Load Balancer only.
    o To use a custom DNS server, add the Azure DNS IP 168.63.129.16 as the upstream DNS server in the custom
    DNS server.
    o Create a private Azure Kubernetes Service cluster - Azure Kubernetes Service | Microsoft Docs
    14

    View full-size slide

  14. 15
    Sample Architecture
    15
    VNET
    azure container registry
    PostgreSQL
    flexible server
    privatelink.postgres.database.azure.com
    Kubernetes cluster

    View full-size slide

  15. 16
    PostgreSQL Flexible Server - Private
    o Networking overview - Azure Database for
    PostgreSQL - Flexible Server | Microsoft Docs
    o Connect from Azure resources in the same virtual
    network to your flexible server by using private IP
    addresses
    o Requires Azure CNI networking type for AKS
    or Egress configuration
    o IP Whitelisting automated
    o Flexible Server public/private network access can NOT
    be toggled
    o Requires delegated subnet
    o More Details
    Private Networking Patterns in Azure Database for
    Postgres Flexible Server - Microsoft Tech Community
    o Restore: Great to create copies of resources before
    deployments or migrations via restore feature
    16

    View full-size slide

  16. 17
    Sample Application
    o As a sample application the Voting App from
    Microsoft was used
    o Kubernetes on Azure tutorial - Prepare an
    application - Azure Kubernetes Service |
    Microsoft Docs
    o We adjusted it to write results to the Postgre
    Database (and revolve around beer)
    o The AKS was deployed using Azure CNI as
    networking type so Pods can connect to the
    private Postgre instance
    o It can be accessed under:
    o http://20.126.205.89/
    17

    View full-size slide

  17. 18
    Example extended Architecture
    18

    View full-size slide

  18. 19
    Extended Architecture – Fully private Environment
    19
    Private AKS cluster
    (uses Private Link)
    Private link ACR
    (Premium SKU)
    privatelink.database.windows.net
    privatelink.azurecr.io
    privatelink..azmk8s.io
    VNET
    Peered VNET
    WWW
    On Premise
    Site2Site tunnel
    Express
    Route
    DNS Forwarding

    View full-size slide

  19. 20
    Tiering AKS
    o Premium Tier needed for AKS as well as ACR to use private networking
    20

    View full-size slide

  20. 21
    ACR Integration Considerations
    o Private ACR requires Premium Tier
    o Linking ACR to AKS requires high privileges
    o az aks update -n --attach-acr
    o “ERROR: Could not create a role assignment for ACR. Are you an Owner on this subscription?”
    o Could not create a role assignment for ACR. Are you an Owner on this subscription? · Issue #1517 · Azure/AKS · GitHub
    o Policies can also bite you at this step
    o Appservices have quirks with the WEBSITE_PULL_IMAGE_OVER_VNET setting needed
    o Limitation: App Service Private Endpoint pulling images to ACR with private Endpoint · Issue #88077 · MicrosoftDocs/azure-
    docs · GitHub
    o Also think about how images get pushed to the ACR
    o Like an external software development company pushing new versions (need peering)
    o Or “pull” new images via pipeline
    o ACR public/private networking access can be toggled
    o Set up private endpoint with private link - Azure Container Registry | Microsoft Docs
    21

    View full-size slide

  21. 22
    Pipeline Considerations
    o In a fully private environment, the Microsoft-hosted agents can not be used
    o Self-hosted agents are mandatory
    o Maintaining, updating, backups need to be kept in mind
    o Identity for Infra-Pipeline might need higher privileges for operations like ACR integration
    o IaC Code gets more complicated with private endpoints and private links
    22

    View full-size slide

  22. 23
    Practical experiences

    View full-size slide

  23. 24
    Challenges with private networking
    o Some Resources support switching public/private access (ACR with private link), some don’t (PostgreSQL
    flexible Server with VNET Integration)
    o Workflow in Dev Environment might be impacted -> Talk to your developers
    o Or result in different infrastructure in Dev environment
    o Jump pods for SSH tunnels also possible
    o One-time Migrations might need additional effort due to inaccessibility
    o Need for additional peering
    o NSG not supported for private endpoints
    o Public preview available though
    o https://docs.microsoft.com/en-us/azure/private-link/disable-private-endpoint-network-policy
    24

    View full-size slide

  24. 25
    DB Migrations in a private environment example
    o Example: The DB on a private Postgres
    flexible Server needs to be migrated from
    one Region to a new Resource in a different
    Region
    o Challenge: You can not use global VNET
    peering due to regulations
    25

    View full-size slide

  25. 26
    CSI Driver
    o Use the Azure Key Vault Provider for Secrets Store CSI Driver for Azure Kubernetes Service secrets - Azure
    Kubernetes Service | Microsoft Docs
    o Mounts secrets, keys, and certificates to a pod by using a CSI volume
    o Distribution of certificates can be a requirement in private environments with private DNS
    o Syncs with Kubernetes secrets
    o az aks enable-addons --addons azure-keyvault-secrets-provider --name myAKSCluster --resource-group
    myResourceGroup
    o kubectl exec busybox-secrets-store-inline -- ls /mnt/secrets-store/
    o Note: When the Azure Key Vault Provider for Secrets Store CSI Driver is enabled, it updates the pod mount and
    the Kubernetes secret that's defined in the secretObjects field of SecretProviderClass. It does so by polling for
    changes periodically, based on the rotation poll interval you've defined. The default rotation poll interval is 2
    minutes.
    26

    View full-size slide

  26. 27
    Maintenance Mode for Kubernetes Applications
    o Think about maintenance modes for applications running on Kubernetes
    o Example: During the deployment of a new version in Prod some DB migrations need to run
    o Challenge: During the migration no DB changes through the app should be possible
    o “Easy way” to down an application is to delete the deployment which also deletes pods
    o Downside: Application frontend becomes unresponsive/fails
    o “One better way” point to a maintenance page
    o There are many different ways to achieve this, depending on the ingress setup and application
    27

    View full-size slide

  27. 28
    Discussion (suggestions)
    o How can we achieve the same with enough security to keep ITSec happy?
    o Is private networking as cloudy as our lovely unfiltered Hoppy Pilsner?
    o Do Azure Private Endpoints leave you bitter like a West Coast style IPA?
    28

    View full-size slide

  28. 29
    Resources
    o Private Networking Patterns in Azure Database for Postgres Flexible Server - Microsoft Tech Community
    o Networking overview - Azure Database for PostgreSQL - Flexible Server | Microsoft Docs
    o Kubernetes on Azure tutorial - Prepare an application - Azure Kubernetes Service | Microsoft Docs
    o Create a private Azure Kubernetes Service cluster - Azure Kubernetes Service | Microsoft Docs
    o Virtual network for Azure services | Microsoft Docs
    o Azure virtual network service endpoints | Microsoft Docs
    o Azure Private Link availability | Microsoft Docs
    o What is a private endpoint? | Microsoft Docs
    29

    View full-size slide

  29. 30
    Industriestrasse 46
    8152 Glattbrugg
    Contact
    itnetX Zurich itnetX Berne itnetX Lausanne
    Kirchgasse 22
    3097 Liebefeld
    c/o Swisscom
    Avenue de Provence 4
    1007 Lausanne
    [email protected] | www.itnetX.ch
    + 41 848 99 00 00

    View full-size slide