Securing AI Apps on Azure: Deploying an AI App to a Private Network on Azure Matthew Gotteiner Azure AI Search aka.ms/securing-vnet-slides Anthony Shaw Python Cloud Advocacy @anthonypjshaw

What are we securing?

RAG solution: Demo aka.ms/ragchat

RAG solution: Architecture aka.ms/ragchat

Configuring a virtual network

Securely networked architecture Use subnets for further isolation: Move all resources into a virtual network: See full PR in https://github.com/Azure-Samples/azure-search-openai-demo/pull/864/files aka.ms/ragchat/private

VNet configuration in Bicep module vnet './core/networking/vnet.bicep' = { name: 'vnet' params: { subnets: [ { name: 'app-int-subnet' properties: { addressPrefix: '10.0.3.0/24' privateEndpointNetworkPolicies: 'Enabled' privateLinkServiceNetworkPolicies: 'Enabled' delegations: [ { id: appServicePlan.id name: appServicePlan.name properties: { serviceName: 'Microsoft.Web/serverFarms' }}] } { name: 'backend-subnet' properties: { addressPrefix: '10.0.1.0/24' privateEndpointNetworkPolicies: 'Enabled' privateLinkServiceNetworkPolicies: 'Enabled' } } .... Creates a subnet for: 1. App Service app 2. Backend services Different rules can be applied to each subnet. See full Bicep in: aka.ms/ragchat infra/network-isolation.bicep

Private endpoints and DNS zones privatelink.openai.azure.com cog-gvzpdyppfabnc.openai.azure.com Private endpoint Private DNS Zone The resource URL remains the same, no app code change needed!

Private endpoints in Bicep module dnsZones './core/networking/private-dns- zones.bicep' = [for privateEndpointConnection in privateEndpointConnections: { name: '${privateEndpointConnection.groupId}-dnszone' params: { dnsZoneName: privateEndpointConnection.dnsZoneName tags: tags virtualNetworkName: vnetName } }] module privateEndpoints './core/networking/private- endpoint.bicep' = [for privateEndpointInfo in flatten(privateEndpointInfo): { name: '${privateEndpointInfo.name}-privateendpoint' params: { location: location name: '${privateEndpointInfo.name}${abbrs.privateEndpoint}${res ourceToken}' tags: tags subnetId: vnetPeSubnetName serviceId: privateEndpointInfo.resourceId groupIds: [ privateEndpointInfo.groupId ] dnsZoneId: dnsZones[privateEndpointInfo.dnsZoneIndex].outputs.id } dependsOn: [ dnsZones ] }] Create private DNS zones and endpoints for: • Azure Blob Storage • Azure OpenAI • Azure AI Search • Azure App Service The endpoint for the service remains the same! No changes to backend code are needed. See full Bicep in: aka.ms/ragchat infra/private-endpoints.bicep

Demo: RAG app in VNet

Azure networking options

Deliver Monitor Protect Connect & Extend Modernizing your network Azure Networking services • Virtual Networking • Azure Peering Service • Azure Virtual WAN • DNS • VPN • ExpressRoute • IPv6 in Azure • Azure Firewall & Firewall Manager • Azure PrivateLink • Azure Bastian • Azure Web Application Firewall • Internet Analyzer • Azure Monitor for Networking • Application Gateway • Azure Front Door (& CDN) • Azure Traffic Manager • Software Load Balancer • Virtual Network NAT • Microsoft Global Network (Optical, L3, Subsea, Peering) • Azure Physical Network (RNG, Metro, Datacenter) • Azure Orbital (Comms, Earth Obs) Infrastructure • Edge Zones (Public, Telecom) How we position our services

Azure Virtual Network Build an isolated and highly-secure environment for your applications • Optionally connect to on-premises datacenters for a hybrid infrastructure that you control. • Bring your own IP addresses and DNS servers. • Secure your connections with an IPsec VPN or ExpressRoute. • Get granular control over traffic between subnets. • Create sophisticated network topologies using virtual appliances. Your private network in the cloud Network Security Group (NSG) Action Name Source Destination Port Allow WebRule Internet WebServers 80,443(HTTP) Allow AppRule1 Web1 App1 443 (HTTPS) Allow DbRule1 App1 Db1 1443 (MSSQL) Allow AppRule2 Web2 App2 443 (HTTPS) Allow DbRule2 App2 Db2 1443 (MSSQL) Deny Deny all inbound Any Any Any NSG Hub VNet App2 VNet App1 VNet DMZ BE NVA Transit Gateway Transit UDR

Azure Virtual Network Peering Seamlessly connect virtual networks within the same region or across regions, sending traffic over the Microsoft private network. • Enforce privacy for your global resources on multiple virtual networks. • Transfer data between virtual networks across Azure subscriptions, Azure Microsoft Entra ID tenants, deployment models, and Azure regions. • Include on-premises networks and leverage Azure high-speed networking in hybrid solutions. Securely connect one or more virtual networks

Azure Network Security Groups (NSG) Azure network security groups can automatically allow or deny traffic Contains security rules NSG security rules are evaluated by priority using five information points

Azure Bastion Connect your RDP and SSH sessions directly in the Azure Portal using a single click experience Log into your Azure virtual machines and avoid public Internet exposure using SSH and RDP with private IP addresses only Integrate and traverse existing firewalls and security perimeter using a modern HTML5 based web client and standard SSL ports Use your SSH keys for authentication when logging into your Azure virtual machines Private and fully managed RDP and SSH access to your virtual machines Azure VM Azure VM Azure VM NSG Private IP Port: 3389/22 Remote protocol (RDP, SSH) Azure Bastion Azure BastionSubnet Virtual Network NSG 443, Internet TLS TLS Azure portal Target VM subnet(s)

Azure Web Application Firewall (WAF) with Front Door Web application protection at network edge 1 Scalable, highly available, Low latency service provided at network edge 2 Easy setup with managed ruleset (OWASP TOP 10) and custom rules 3 Bot protection using threat intelligence-based filtering (preview) 4 Global insights 5 Built-in DDoS protection 6 Azure Front Door provides built-in CDN capabilities 7 Cost efficient: Pay as you go WAF blocks malicious attacks Front Door routes legitimate traffic to backends Backend allow Front Door access only AKS On-Premise Other cloud Azure Regions Web Servers Azure Web App Azure Edge Public Internet Application Gateway

Decision Tree on Load Balancing & Network Acceleration PaaS (App Service, Functions) Web application? (HTTP/HTTPS) Internet facing application? Global / Deployed in multiple regions? Do you require SSL offload or application- layer processing per request? Hosting - PaaS, laaS, AKS Global / Deployed in multiple regions? No Yes AKS laaS (VMs) No No Yes Yes Yes Yes No Internet facing application? No Do you require performance acceleration? No Yes START Azure Load Balancer Traffic Manager + Azure Load Balancer Application Gateway Azure Front Door + Application Gateway Azure Front Door Azure Front Door + Application Gateway ingress controller Azure Front Door + Azure Load Balancer Application Gateway Azure Front Door Yes

Azure Front Door Define, manage, and monitor global routing of your web traffic by optimizing for best performance and enabling instant failover for high availability. • Global HA, BCDR - Enable fast-failover for regional services, microservices at the Edge with active path monitoring • Security at the Edge - Stop threats where they come from at the Edge with DDoS protection and customizable WAF • Faster apps - Reduce latency and increase throughput for apps by offloading SSL at the Edge and accelerating requests • Deliver and scale global web apps with an http(s) load balancer • Integrated static content caching • Global app dashboard, service insights Fast, reliable cloud content delivery service with intelligent threat protection Azure Region Media Services Ingest, Encoding, Personalization, Security, Analytics Storage Ingest Microsoft Global Network Edge Location Unmetered egress

Decision Tree on Private Network Connectivity Integration with existing SDWAN environment? Custom/No high availability configuration? Encryption Over Internet? Need to connect to PAAS /Public services privately? Private endpoints/Private link service Need to exchange data using private IP address? Need native system routing? Native build in high availability? Encryption Over Internet? Follow the guidelines on the network connectivity ExpressRoute Azure VPN SDWAN Tunnel Predictable throughput? SLA required? High Volume data transfer? Avoid connection over public Internet? Yes No Yes No No Yes Yes Yes Yes START Internet facing application?

Azure ExpressRoute Experience predictable performance with a faster, private connection to Azure Use Azure ExpressRoute to create private connection between Azure datacenter and your on-premises or co-location infrastructure • Predictable network performance and lower latencies than public internet connection • Bandwidth up to 10/100 Gbps Supported • MacSec support • Enterprise-grade resiliency with availability SLA • ExpressRoute Direct, Local, and Global Reach • Over 200 ExpressRoute partners, including Satellite providers Customer’s Network Partner Edge Primary Connection Microsoft Edge Microsoft Peering for Office 365, Dynamics 365, Azure public services (public IPs) Azure Private Peering for Virtual Networks Secondary Connection ExpressRoute Circuit

Azure VPN Gateway Connecting your infrastructure to the cloud A virtual network gateway that sends encrypted traffic between an Azure virtual network and on- premises over the public Internet, or between Azure virtual networks. • Deploy and access resources within your VPN • Supports multiple platforms, protocols, and authentication mechanisms • Configure once to access multiple resources • Scale on-prem VPN by connecting it to Azure Internet VPN-connected site VPN-connected site Point-2 site users Microsoft backbone Spoke Spoke Hub Internet edge Internet edge Internet edge Azure region VPN gateway On-premises and legacy apps

Continuous security

Defender for Cloud aka.ms/enable-defender • Security alerts: Detects DDOS, suspicious logins, etc. • Security posture: Audits Azure resources and their settings • Workload protections: Scans for known vulnerabilities in SQL, container images, etc. • Data security: Scans stored data for PII and sensitive data • Regulatory compliance: Ensure compliance with benchmarks.

Defender recommendations: RAG without VNet 10 recommendations for azure-search-openai-demo, non-private deployment:

Defender recommendations: RAG with VNet 10 recommendations for azure-search-openai-demo, private deployment:

GitHub actions for security recommendations Use ps-rule action on your Bicep to auto-scan IaC for security issues - name: Run PSRule analysis uses: microsoft/ps-rule@v2.9.0 with: modules: PSRule.Rules.Azure baseline: Azure.Pillar.Security inputPath: infra/*.test.bicep outputFormat: Sarif outputPath: reports/ps-rule-results.sarif summary: true continue-on-error: true env: PSRULE_CONFIGURATION_AZURE_BICEP_FILE_EXPANSION: 'true' PSRULE_CONFIGURATION_AZURE_BICEP_FILE_EXPANSION_TIMEOUT: '30' - name: Upload results to security tab uses: github/codeql-action/upload-sarif@v3 with: sarif_file: reports/ps-rule-results.sarif github.com/microsoft/ps-rule

GitHub actions for security recommendations: Results https://github.com/Azure-Samples/azure-search-openai-demo/actions/runs/9378324878

Next steps

Try our samples and learn more! Azure OpenAI + AI Search + Entra + MSAL + App Service Built-in Auth aka.ms/ragchat Find more AI samples at: aka.ms/azai Java JavaScript Python .NET OpenAI Assistants Fine-tuning ...and more! Azure Well-Architected Framework https://learn.microsoft.com/azure/well-architected/ Azure Verified Modules https://azure.github.io/Azure-Verified-Modules/

Securing AI Apps on Azure Date Topic Speakers July 2 5-6PM UTC Using Keyless Auth with Azure AI Services Marlene Mhangami Pamela Fox July 8 5-6PM UTC Add User Login to AI Apps using Built-in Auth James Casey Pamela Fox July 9 7-8PM UTC Add User Login to AI Apps using MSAL SDK Ray Luo Pamela Fox July 10 7-8PM UTC Handling User Auth for a SPA App on Azure Matt Gotteiner July 17 7-8PM UTC Data Access Control for AI RAG Apps on Azure Matt Gotteiner Pamela Fox July 25 11PM-12PM Deploying an AI App to a Private Network on Azure Matt Gotteiner Anthony Shaw https://aka.ms/S-1355