Kubernetes on Azure Michel Hubert May 2023 

Content ## Introduction ## Kubernetes ## Why Managed Kubernetes? ## Azure Kubernetes Service (AKS) is the best choice for a managed Kubernetes service ## Kubernetes on Azure: A world class ecosystem for an enterprise-grade platform ## Key Benefits ## Customer stories ## Resources ## Optional sections ## All customer stories ## Microsoft in the community ## what is a container? ## Kubernetes 101 ## Top scenarios ## Product deep dive 

Test your Kubernetes adoption knowledge How many of you are using Kubernetes for container orchestration? A. 42% B. 66% C. 73% D. 88% E. 95% Of those of you using Kubernetes, how many are running workloads in production? A. 45% B. 55% C. 74% D. 85% E. 100% All statistics from Red Hat. “Kubernetes adoption, security, and market trends report 2021.” July 14, 2021. Report. 

Why Managed Kubernetes? 

Kubernetes: the de facto container orchestrator Portable Public, private, hybrid, multi-cloud Extensible Modular, pluggable, hookable, composable Self-healing Auto-placement, auto-restart, auto-replication, auto-scaling 88% of enterprises using or adopting Kubernetes Microsoft investing in Kubernetes • 74% of enterprises that use Kubernetes are running workloads in production • 74% of survey respondents have a DevSecOps initiative underway • At least 20% of enterprises that use Kubernetes leverage six different open-source security tools All statistics from Red Hat. “Kubernetes adoption, security, and market trends report 2021.” July 14, 2021. Report. 

The elements of high-performing Kubernetes Scheduling Affinity/ anti-affinity Health monitoring Failover Scaling Networking Service discovery Coordinated app upgrades Kubernetes 

Kubernetes 1. Kubernetes users communicate with API server and apply desired state 2. Control plane actively enforces desired state on agent nodes 3. Agent nodes support communication between containers 4. Agent nodes support communication from the Internet is complex api-server replication, namespace, serviceaccounts, etc. controller- manager scheduler etcd Control plane Agent node kube-proxy Container runtime Pods Pods Containers Containers Agent node kube-proxy Container runtime Pods Pods Containers Containers Internet Internet User kubelet kubelet Agent pools 5. Agent pools keep multiple agent nodes organized 

Managed api-server -controller- manager -scheduler etcd Control plane Container runtime Pods Pods Containers Containers Agent nodes Container runtime Pods Pods Containers Containers Azure managed control plane Kubernetes • Automated upgrades, patches • High reliability, availability • Easy, secure cluster scaling • Self-healing • API server monitoring • At no charge* *Higher SLA guarantees available as an optional uptime SLA paid feature handles the complexity for you 

Security How do you manage access to clusters? What about managing sensitive info? Developer tools What does the developer toolchain look like? Platform management Should we use a single cloud provider or multiple? Own our data center? Consider a hybrid setup? DevOps What processes do we need for a smooth operation? But… Managed is just a starting point Kubernetes Things to consider as you adopt management Kubernetes 

the most integrated experience for developing cloud native apps Security DevOps Platform mgmt. Developer tools on Azure Security • Azure Active Directory • Azure Policy • Azure Security Center • Azure Key Vault Platform management • Azure Kubernetes Service • Azure Red Hat OpenShift • Azure Arc-enabled Kubernetes Developer tools • IDE container support • Source code repository • Registry supporting Helm • Microservice debugging DevOps • Visual Studio Code • GitHub • Azure Pipelines • Azure Monitor Managed Kubernetes 

Community Platform mgmt. Security DevOps Developer tools Community • CNAB for packaging distributed applications • Dapr for building event-driven, resilient distributed applications • Helm for managing Kubernetes applications • KEDA for event-driven autoscaling • Virtual Kubelet for connecting Kubernetes to other APIs • Gatekeeper for enforcing policies in Kubernetes • VS Code Kubernetes Extensions for building Kubernetes applications • Open Service Mesh for securing and managing microservice applications • Containerd to manage the complete container lifecycle Managed Kuber in the Kubernetes community Microsoft Azure 

• Deep expertise due to leading and contributing to OSS projects • Building solutions to address tough cloud native problems • Invested in the success of Kubernetes • Committed to ensuring compatibility with OSS • Committed to portability regardless of tools you adopt Top contributor to open-source projects on GitHub since 20161 100% increase of employee contributors on GitHub since 20161 29+ open source-related organizations and initiatives where Microsoft is a member2 Microsoft in the community 1Open Source Contributor Index (OSCI) 2Various sources from Wikipedia. Microsoft is either a founding member, joining member, contributing member, and/or sponsor. Managed Kuber in the Kubernetes community Microsoft Azure 

Azure Kubernetes Service is the most advanced Kubernetes platform Azure Kubernetes Service z Built-in best practices Multi-layer security Operational efficiency Unified management Kubernetes innovations are built together with the community Kubernetes Azure Service 

Kubernetes on Azure Enterprise-grade by design Built-in best practices Proactive and actionable recommendations and support from experts, based on knowledge from thousands of enterprise engagements Multi-layer security Hardened security and layers of isolation across compute resources, data, and networking Operational efficiency Automated provisioning, repair, monitoring, and scaling gets you up and running quickly and minimizes infrastructure maintenance Unified management Consistent configuration and governance across on premises, multi cloud, multi- cluster, hybrid, and edge Built for enterprises World-class developer tools and a broad ecosystem to meet the diverse needs of enterprises 

Azure Kubernetes Service (AKS) is the best choice for a managed Kubernetes service 

Built-in best practices Multi-layer security Operational efficiency Unified management Built-in best practices • Based on knowledge from thousands of customer engagements • Proactive and actionable recommendations from Azure Advisor • Proactively improve performance, availability, and security • Intelligent, self-diagnostic portal-based experience • Self-service troubleshooting with proven tools For more information, see Enterprise-scale for AKS 

Multi-layer security Operational efficiency Unified management Multi-layer security • Enforce compliance rules with Azure Policy • Identity and access control using Azure Active Directory • Encrypt using your own keys, stored in Azure Key Vault • Gain unmatched security management with Azure Security Center integration • Interact securely with Kubernetes API server using Azure Private Link Built-in best practices 

Operational efficiency Operational efficiency Multi-layer security Unified management Built-in best practices • Elastically add compute capacity with serverless Kubernetes • Higher availability using redundancies across availability zones • Paired region deployment for disaster recovery • Real-time personalized recommendations with Azure Advisor • Detailed insights via Azure Monitor • Financially-backed opt-in Service Level Agreement (SLA) • Workload specific cost savings options Azure Kubernetes Service Microservices Availability Auto scaling Auto repair Auto upgrade Trusted Advisor Pods Virtual node  Monitor Disaster recovery 

Unified management Unified management • Central inventory and monitoring of assets running anywhere • Consistently apply policies & role-based-access- controls (RBAC) • Deploy resources using GitOps-based workflow • Use Flux operator for automatic sync Multi-layer security Operational efficiency Built-in best practices Identity RBAC Monitoring Policy Azure Kubernetes Service Developer GitHub repo Flux operator commit sync apply/delete Azure Arc Kubernetes Azure Stack On-premises Multi-cloud Edge 

Bringing you the enterprise expertise Best practices from of enterprise engagements 1000s on-call support backed by Kubernetes certified experts 24x7x365 Security experts focused on your data security and privacy 3,500 Built-in Best practices Enterprise support Multi-layer Security Available in more regions than any other cloud provider 

Azure Kubernetes Service momentum The fastest growing compute service on Azure Trusted by thousands of customers 

We support every workload scenario across industries Machine Learning + Artificial Intelligence Internet of Things (IoT) High-performance Computing (HPC) Microservices Automotive Financial Services Healthcare Manufacturing 

Resources 

Best support for your enterprise needs Container Adoption Best Practices aka.ms/adopt/containers Learning path aka.ms/LearnKubernetes What is Kubernetes aka.ms/k8sLearning Hear from experts aka.ms/AKS/videos Case studies aka.ms/AKS/casestudy See what’s new aka.ms/k8sroadmap Try for free aka.ms/AKS/trial Kubernetes on Azure aka.ms/K8sonAzure 

© Copyright Microsoft Corporation. All rights reserved. Thank you. 

Kubernetes 101 

Kubernetes: deploy and manage containerized workloads/services at scale Portable Public, private, hybrid, multi-cloud Kubernetes is… Extensible Modular, pluggable, hookable, composable Self-healing Auto-placement, auto-restart, auto-replication, auto-scaling Flexible Facilitates both declarative configuration and automation Enterprise-ready Supported by a large, and growing ecosystem 

Kubernetes is complex 1. Control Plane: manages the agent nodes and the pods in the cluster • api-server: front end of the Kubernetes control plane; exposes Kubernetes API • controller-manager: runs the controller processes • scheduler: tracks newly created pods and selects node to run them on • etcd: stores the state of the cluster (config, running workloads status, etc.) 2. Agent nodes: run your application workloads • Pods: a collection of containers co- located on a single machine • kube-proxy: a network proxy that runs on each node in a cluster • kubelet: agent that runs on each node in a cluster; ensures containers are running in a pod • Containers: software responsible for running containers api-server replication, namespace, serviceaccounts, etc. controller- manager scheduler etcd Control plane Agent node kube-proxy Container runtime Pods Pods Containers Containers Agent node kube-proxy Container runtime Pods Pods Containers Containers Internet Internet User kubelet kubelet Agent pools 1 2 

How does Kubernetes work? User Kubernetes Service Pods Containers Pods Containers Pods Containers Customer VMs YAML Cloud load balancer api-server Control plane IP address Kubernetes Internet Developer Pods Containers   Liveness check Readiness check • Problem is detected in one of the pods and a new pod is created as replacement Image • Liveness check and readiness check are performed on the new pod • Load balancer starts directing incoming traffic to the newly created pod • Grace period provided for existing users • Problematic pod is terminated. Configuration is fully restored. Scenario: Maintain configuration while handling failure 

Benefits of Kubernetes Velocity Faster development cycle due to declarative configuration and immutability Self-healing Continuous action to maintain desired state leads to self-healing when issues arise Scalability Easier to scale manually or automatically Infrastructure abstraction Applications can be developed independent of the environment Declarative configuration Declare the desired state and have Kubernetes manage it for you Scheduling No need to schedule each container manually 

Top scenarios 

Top scenarios for Kubernetes on Azure Cost saving without refactoring your app Lift and shift to containers Agility Faster application development Microservices Automation Deliver code faster and securely at scale Secure DevOps Performance Low latency processing Machine learning Portability Build once, run anywhere IoT Analytics Real-time data collection and streaming Data streaming 

Machine learning Secure DevOps Lift and shift to containers App modernization without code changes • Speed application deployments by using container technology • Defend against infrastructure failures with container orchestration • Increase agility with continuous integration and continuous delivery Modernized application Modernized application Modernized application Container Registry Existing application Kubernetes cluster Managed Database CI/CD IoT Microservices Data streaming 

Lift and shift to containers App modernization without code changes Capabilities 1. Use Azure Container Registry to store container images and Helm charts for your modernized applications, replicated globally for low latency image serving 2. Integrate AKS with Azure Pipelines or other Kubernetes ecosystem tooling to enable continuous integration/continuous delivery (CI/CD) 3. Enhance security with Azure Active Directory and RBAC to control access to AKS resources Azure Container Registry Existing application Virtual network AKS Active Directory Azure Database for MySQL 1 2 3 CI/CD Pipelines Machine learning Secure DevOps IoT Microservices Data streaming 

Microservices Microservices: for faster app development • Independent deployments • Improved scale and resource utilization per service • Smaller, focused teams Monolithic APP APP APP Microservices Large, all-inclusive app Small, independent services Machine learning Secure DevOps IoT Lift and shift to containers Data streaming 

Microservices Microservices for faster app development Capabilities 1. Use Visual Studio Code to iteratively develop, test, and debug microservices targeted for AKS clusters. 2. Azure Pipelines has native integration with Helm and helps simplifying continuous integration/continuous delivery (CI/CD) 3. Virtual node—a Virtual Kubelet implementation—allows fast scaling of services for unpredictable traffic. 4. Azure Monitor provides a single pane of glass for monitoring over app telemetry, cluster-to-container level health analytics. Machine learning Secure DevOps IoT Lift and shift to containers Data streaming https://github.com/Microsoft/SmartHotel360- AKS-DevSpaces-Demo Inner loop Source code control Azure Container Registry CI/CD Pipelines Auto- build Azure Monitor Test Debug Visual Studio Code AKS dev cluster AKS production cluster Pods Container instances Pods 1 2 3 4 

Secure DevOps Secure DevOps • Deliver code faster with Kubernetes and CI/CD • Accelerate the feedback loop with constant monitoring • Balance speed and security with continuous security and deep traceability Source code Build Pipelines Release Pipelines Kubernetes cluster Continuous Integration Continuous Delivery Deployment strategies Monitor & logging Monitor Iterate Machine learning IoT Lift and shift to containers Microservices Data streaming 

Secure DevOps Secure DevOps Capabilities 1. Developers rapidly iterate, test, and debug different parts of an application together in the same Kubernetes cluster 2. Code is merged into a GitHub repository, after which automated builds and tests are run by Azure Pipelines 5. Container image is pushed to Azure Container Registry 4. Kubernetes clusters are provisioned using tools like Helm charts that define the desired state of app resources and configurations 6. Cluster operators define policies in Azure Policy to govern deployments to the AKS cluster 3. Release pipeline automatically executes pre-defined deployment strategy with each code change 7. Azure Policy audits requests from the pipeline at the AKS control plane level 8. App telemetry, container health monitoring, and real-time log analytics are obtained using Azure Monitor 9. Insights used to address issues and fed into next sprint plans Machine learning IoT Lift and shift to containers Microservices Real-time log analytics Release 3 Release N Release 3 Release 2 Release 1 Azure Pipelines CI/CD Pipelines Helm chart Azure Monitor Azure Policy 3 6 AKS production cluster Release 3 App telemetry Container health Data streaming 9 Inner loop Azure Container Registry Container image 2 5 7 8 Source code control sample code { iterating.with.team // in one // isolated environment } 1 Test Debug Visual Studio Code  4 AKS dev cluster 7 

Machine learning Data scientist in a box • Quick deployment and high availability • Low latency data processing • Consistent environment across test, control and production IoT Lift and shift to containers Microservices Secure DevOps How to deploy ML Model Registry IDE Monitoring Cloud service Heavy Edge Light Edge Developer Cognitive Services ML Experimentation Data from Azure Blob or Cosmos DB Model creation/ retraining ML Image Registry Data scientist Register model Create scoring files and dependencies Create and register image Deploy image Improve model Data streaming 

Machine learning Data scientist in a box IoT Lift and shift to containers Microservices Secure DevOps Data streaming Capabilities 1. Data Scientist creates the model 2. Register Model using the Model Registry 3. Register Image using the Image Registry (the Azure Container Registry) 4. Deploy the Image to cloud or to edge devices using AKS 5. Monitor and use models—you can monitor input, output, and other relevant data from your model. Developers can query the model for insights for apps App Developer Data scientist Create/ retain model Register model Create scoring files and dependencies Create and register image w/ ACR Monitor Cloud Heavy Edge Light Edge Deploy image w/ AKS 1 2 3 4 5 

IoT Scalable Internet of Things solutions • Portable code, runs anywhere • Elastic scalability and manageability • Quick deployment and high availability AKS Database for MySQL Azure Cosmos DB SQL Database IoT Hub IoT Edge devices IoT Edge Connector Lift and shift to containers Microservices Secure DevOps Machine learning Data streaming 

IoT Scalable Internet of Things solutions Capabilities 1. Azure IoT Edge encrypts data and send to Azure, which then decrypts the data and send to storage 2. Kubelets serve as the translator between cloud and Edge 3. IoT Edge Provider in kubelet redirects containers to IoT Edge and extend AKS cluster to target millions of edge devices 4. Consistent update, manage, and monitoring as one unit in AKS using single pod definition Azure IoT Edge Compress Encrypt Send to Cloud Azure Kubernetes cluster Node Docker container Docker container Node Docker container Docker container kubelet Docker container Docker containers Decrypt Decompress Send to Storage 1 2 3 4 Lift and shift to containers Microservices Secure DevOps Machine learning Data streaming kubelet IoT Edge Provider 

Data streaming Data streaming • Real-time data gathered and streamed to AKS • Collected data analyzed and insights generated almost instantly • Data stored and available for deeper analysis by data scientists Azure Cosmos DB HDInsight Apache Kafka Cache for Redis Database for PostgreSQL AKS IoT sensor API Management Storage Analysis Lift and shift to containers Microservices Secure DevOps Machine learning IoT 

Data streaming Lift and shift to containers Microservices Secure DevOps Machine learning IoT Data streaming Capabilities 1. Sensor data is generated and streamed to Azure API Management 2. AKS cluster runs microservices that are deployed as containers behind a service mesh; containers are built using a DevOps process and stored in Azure Container Registry 3. Ingest service stores data in an Azure Cosmos DB 4. Asynchronously, the analysis service receives the data and streams it to Apache Kafka and Azure HDInsight 5. Data scientists can analyze the big data for use in machine learning models using Splunk 6. Data is processed by the processing service, which stores the result in Azure Database for PostgreSQL and caches the data in an Azure Cache for Redis 7. A web app running in Azure App Service is used to visualize the results AKS Azure Cosmos DB HDInsight Apache Kafka Cold path Hot path IoT sensor API Management Service Mesh Interface Service Mesh Ingest service Analysis service Processing service 1 2 Cache for Redis Database for PostgreSQL App Service 7 3 Asynchronous 4 6 Splunk 5 GitHub CI/CD Pipelines ACR 

Product deep dive 

For Infrastructure and Operations managers 

Schedule pods over private tunnel API server Controller Manager Scheduler etcd Store Cloud Controller Self-managed master node(s) Customer VMs App/ workload definition User Docker Pods Docker Pods Docker Pods Docker Pods Docker Pods Kubernetes API endpoint Azure managed control plane Increase operational efficiency Focus on your containers and code, not the plumbing of them Responsibilities DIY with Kubernetes Managed Kubernetes on Azure Containerization Application iteration, debugging CI/CD Provisioning, upgrades, patches Reliability availability Scaling Monitoring and logging Customer Microsoft 

Increased operational efficiency Highly available, reliable service with serverless scaling • Easily provision fully managed clusters with automatically configured monitoring capabilities based on Prometheus • Real-time personalized recommendations to optimize your AKS deployments with Azure Advisor integration • Elastically add compute capacity with serverless Kubernetes in seconds without worrying about managing the infrastructure. • Higher availability using redundancies across availability zones, protecting applications from datacenter failures User AKS prod cluster Microservices Availability Auto scaling Auto repair Auto upgrade Trusted Advisor Teleport Monitor Pods Virtual node  

Azure makes Kubernetes easier Manage and operate Kubernetes with ease Task The Old Way With Azure Create a cluster Provision network and VMs Install dozens of system components including etcd Create and install certificates Register agent nodes with control plane az aks create Upgrade a cluster Upgrade your master nodes Cordon/drain and upgrade Agent nodes individually az aks upgrade Scale a cluster Provision new VMs Install system components Register nodes with API server az aks scale 

Serverless Kubernetes using AKS virtual nodes • Elastically provision compute capacity in seconds • No infrastructure to manage • Built on open sourced Virtual Kubelet technology, donated to the Cloud Native Computing Foundation (CNCF) Node Pods Node Pods Kubernetes control plane Azure Container Instances (ACI) Pods Virtual node 

Horizontal Pod Autoscaler The horizontal pod autoscaler (HPA) uses the Metrics Server in a Kubernetes cluster to monitor the resource demand of pods. If a service needs more resources, the number of pods is automatically increased to meet the demand. 1. HPA obtains resource metrics and compares them to user-specified threshold 2. HPA evaluates whether user specified threshold is met or not 3. HPA increases/decreases the replicas based on the specified threshold 4. The Deployment controller adjusts the deployment based on increase/decrease in replicas Horizontal Pod Autoscaler Deployment ReplicaSet Metrics Server Collects metrics from all nodes NodeX Grabs metrics replicas++ replicas-- Node1 Node2 Pod Pod Kubelet cAdvisor Pod Kubelet cAdvisor Collects metrics from all containers on the node 1 2 3 4 

Cluster Autoscaler The cluster autoscaler watches for pods that can't be scheduled on nodes because of resource constraints. The cluster then automatically increases the number of nodes. 1. HPA obtains resource metrics and compares them to user-specified threshold 2. HPA evaluates whether user specified threshold is met or not 3. HPA increases/decreases the replicas based on the specified threshold 4. The Deployment controller adjusts the deployment based on increase/decrease in replicas Additional nodes needed Pods are in pending state Pod Pod Node Pod Pod Node Pod Pod AKS cluster Cluster Autoscaler Azure 1 2 3 Node is granted 4 Pending pods are scheduled 

Service Mesh Interface (SMI) SMI defines a set of APIs that can be implemented by individual mesh providers. Service meshes and tools can either integrate directly with SMI or an adapter can consume SMI and drive native mesh APIs. • Standard interface for service mesh on Kubernetes • Basic feature set to address most common scenarios • Extensible to support new features as they become widely available Apps Tooling Ecosystem …and more Service Mesh Interface Routing Telemetry Policy Kubernetes 

Azure Disk Storage Azure Blob and Data Lake Storage Azure File Storage High, performance, durable block storage for Azure Virtual Machines Simple, secure, and serverless enterprise-grade cloud file shares Massively scalable and secure object storage and data lake Enteprise file storage, powered by NetApp Workloads Databases, bigdata, cache, CI/CD Shared/user workspace, CMS, databases, AI/ML Analytics on data lake Analytics, HPC, Custom apps currently using NetApp Access protocol SCSI SMB, NFS v4.1 (preview) Blobfuse, NFS v3.0 (preview) NFS v3.0, NFS v4.1 Model Static, Dynamic Static, Dynamic Static, Dynamic Static, Dynamic SKUs Standard HDD, Standard SSD, Premium SSD, Ultra (v1.20) Standard HDD, Premium SSD Standard HDD, Premium SSD Standard, Premium, Ultra Access modes ReadWriteOnly (RWO), ReadWriteMany (RWX) (v1.20) RWO, RWX RWO, RWX RWO, RWX Container type Linux, Windows Linux, Windows, ACI Linux Linux Redundancy Locally-redundant storage (LRS), Zone-redundant storage (ZRS) - coming soon LRS, ZRS, Geo-redundant storage (GRS), Read-access geo- redundant storage (RAGRS) LRS, ZRS, GRS, RAGRS Single-zone Azure NetApp Files via Trident Azure Storage options https://docs.microsoft.com/bs-latn-ba/azure/aks/concepts-storage 

For developers and DevOps specialists 

Accelerate containerized development an iterative Kubernetes development experience for teams with integrated CI/CD • Native containers and Kubernetes support in Visual Studio Code • Private container registry with Helm support • Develop and test Kubernetes apps without mocking up dependencies • Effective code merge, containerization, and CI/CD pipeline with automated tasks in a few clicks • Pre-configured canary deployment strategy • In-depth build and delivery process review and integration testing Release 3 Release N Release 3 Release 2 Release 1 Azure Pipelines CI/CD Pipelines Inner loop Azure Container Registry Helm chart Container image Source code control sample code { iterating.with.team // in one // isolated environment } Test Debug Visual Studio Code  AKS production cluster Release 3 AKS dev cluster 

Azure makes Kubernetes easier Accelerate containerized application development Task The Old Way With Azure Inner loop development Set up a local dev environment using Minikube Determine the transitive closure of your dependencies Identify behavior of dependencies for key test cases Stub out dependent services with expected behavior Make local changes, check-in, and hope things work Validate with application logs Use Visual Studio Code to run and debug services locally while connected to existing services and dependencies without having to mock them Set up a CI/CD pipeline and deploy to Kubernetes Create Git repo Create a build pipeline Create a container registry Create a Kubernetes cluster Configure build pipeline to push to container registry Configure build pipeline to deploy to Kubernetes Define and set up deployment strategy Store source code on GitHub, then create a project on Azure Pipelines with Kubernetes/AKS as a target Make container images available for deployment worldwide Create a container registry in every region Configure build pipeline with multiple endpoints Loop through all regions and push following build Create an Azure Container Registry with geo-replication Push your image to a single endpoint Track health with consolidated cluster and application logs Choose a logging solution Deploy log stack in your cluster or provision a service Configure and deploy a logging agent onto all nodes Checkbox enable monitoring with centralized tracking of logging and analytics 

Unmatched agility Powered by automation and integration with familiar tools 1. Automatically containerize and scaffold any applications directly from IDE 2. Auto-build to a secure container registry 3. Rapidly iterate, test, and debug microservices 4. A few clicks to a full CI/CD pipeline and pre-configured deployment strategy 5. Built-in monitoring and logging to get full visibility of container health and app telemetry App Container Container Registry Production environment 1 2 3 4 5 Monitoring and logging 

Azure Pipelines for AKS 1. As part of the CI, developers check in their code to a central repository, like GitHub; Azure Pipelines automatically builds application binaries, runs unit test, and pushes container image into a registry 2. Developers then deploy the application to a testing environment and run integration test as part of the CD workflow 3. Developers can review which pod is running which container image, what source code is built into an image, and what tests are run against each image at any point of time 4. For production deployment, Azure Pipelines automatically executes pre-defined deployment strategy and progressively rolls out application to an AKS cluster 5. Enable app telemetry, container health monitoring, and real-time log analytics; insights used to address issues and feed into next sprint plans Build Pipelines Release Pipelines AKS cluster Continuous Integration Continuous Delivery Deploy strategies Azure Monitor Monitor Iterate Deep traceability Source code Pod Container image Source Repository 1 2 3 4 5 

GitHub Actions for Kubernetes on Azure 1. Authenticate and login securely to an Azure subscription 2. Set the target AKS cluster 3. Create Kubernetes secret objects to manage sensitive information 4. Connect to the Kubernetes cluster and deploy manifests, etc. Action k8s-create-secret Action docker-login Action aks-set-context Action k8s-deploy 1 2 3 4 

Azure Container Registry geo-replication Push image to a single registry and ACR takes care of geographical replication, including local notifications. 1. US-based developer commits codes to build container image 2. Image is pushed to the nearest Azure Container Registry (ACR) region based on DNS 3. Geographical webhook triggers deployment to East US 4. ACR geo-replicates to configured regions 5. Geographical webhook triggers deployment to West Europe 6. Both AKS clusters pull from contoso.azurecr.io Developer Container image contoso.azurecr.io East US West Europe contoso.azurecr.io contoso.azurecr.io/app:v1 AKS CD ACR contoso.azurecr.io/app:v1 AKS CD ACR Geo-Replication 1 2 3 4 5 6 6 

Kubernetes-based event-driven auto-scaling (KEDA) Open-source component jointly built by Microsoft and RedHat • Event-driven container creation & scaling Allows containers to “scale to zero” until an event comes in, which will then create the container and process the event, resulting in more efficient utilization and reduced costs • Native triggers support Containers can consume events directly from the event source, instead of routing events through HTTP • Can be used in any Kubernetes service This includes in the cloud (e.g., AKS, EKS, GKE, etc.) or on-premises with OpenShift—any Kubernetes workload that requires scaling by events instead of traditional CPU or memory scaling can leverage this component. Kubernetes cluster External trigger source KEDA AKS cluster Scaler Controller Metrics adapter 

Azure Monitor for containers 1. Get detailed insights about your workloads with Azure Monitor 3. See graphical insights about clusters 2. Filter for details about nodes, controllers, and containers 4. Pull events and logs for detailed activity analysis Azure Monitor for containers Visualization Insights Monitor & analyze Response Native alerting with integration to issue management and ITSM tools Monitor and analyze Kubernetes and container deployment performance, events, health, and logs Provide insights with cluster health rollup view Visualize overall health and performance from cluster to containers with drilldowns and filters Cloud native experience for Azure Monitor with Prometheus integration Azure Kubernetes Service Azure Pipelines Observability Observe live container logs and Kubernetes event log on container deployment status Virtual node Prometheus 4 1 3 2 

AKS diagnostics • Faster resolution of common issues with an intelligent, self-diagnostic experience right in the portal • Cluster-specific observations • Recommended actions for troubleshooting Intelligent detectors based on AKS-specific telemetry Recommended actions for troubleshooting Cluster-specific observations Zero configuration and zero cost <\> User Azure portal AKS diagnostics Sample diagnostics web portal Azure backend telemetry AKS production cluster Node 2 Node 1 

Azure Monitor for containers Configuration management scenario 1. Deploy Azure Arc for Kubernetes agent 2. Azure Arc agent registers cluster with ARM 3. Cluster operator applies cluster configuration via ARM 4. Configuration agent picks up configuration and syncs state from git repo 5. Configuration agent informs Azure policy of status 6. Cluster operator or application developer pushes changes via GitHub Cluster Connect RP Cluster operator Azure Resource Manager Cluster Config RP Azure Policy GitHub Config agent Azure Arc agent Cluster operator/ Application dev 1 3 5 4 2 6 Kubernetes on-prem 

overview API-driven development with Kubernetes: Putting API at the center of the development process to clearly separate app accessibility and app logic API defines how internal world communicates with outside world • External interface to the world • Formalizes parameters for internal and external user access • Allows definition and enforcement of policies, like security and usage • Provides abstraction of the underlying details • Enables decoupling of interface development from logic development • Acts as proxy for app logic Kubernetes provides app orchestration environment and scalability needs • Simplifies migration and modernization • Enables developers to focus on app logic • Provides orchestration and scalability across apps and services Kubernetes platform Kubernetes Security, governance, identity Infrastructure automation Infrastructure Accessibility Logic Cloud app Website Mobile app B2B partner API 

API-driven development with Kubernetes: architecture Putting API at the center of the development process to clearly separate app accessibility and app logic 1. API is defined by API developers and published via the API Management portal 2. Application developers define the microservices and associated logic and deploy to Kubernetes 3. API users (internal and/or external) use the API developer portal to learn about the API and use them in their applications 4. Applications access APIs via the API Gateway 5. API Gateway, after ensuring the API request meets security and other policies e.g. throttling, forwards the request to service running in Kubernetes Kubernetes platform Kubernetes Security, governance, identity Infrastructure automation Infrastructure Accessibility Logic API API developers API users App developers Management Portal Developer Portal Gateway 1 2 3 4 5 

API-driven development with Kubernetes: benefits Putting API at the center of the development process to clearly separate app accessibility and app logic Benefits of using API with Azure • Create API gateway and developer portal in minutes • Publish APIs easily for internal or external use • Manage, secure, optimize all your APIs in one place • Connect to back-end services anywhere Enabling technologies • Broad support for technologies to fit your migration, modernization, transformation, and API needs • Extensive infrastructure and services to simplify security, compliance, and standardization • Refined management plane to ease the task of development and management • Support for multi-cloud and hybrid* Accessibility Logic API Management Portal Developer Portal Gateway *Map illustration represents existing and future availability for Azure. Map is not all-inclusive. Kubernetes On-prem, cloud, or hybrid Security, governance, identity Infrastructure automation 

Security & Networking 

Build on an enterprise-grade, secure platform Control access through AAD and RBAC Get runtime vulnerability scanning and auditing through Azure Security Center Put guardrails in your development process with Azure Policy Secure network communications with VNET and network policy Gain automated threat protection and best practice recommendations for Kubernetes clusters 

Identity Use familiar tools like AAD for fine-grained identity and access control to Kubernetes resources from cluster to containers AKS with RBAC Storage SQL Database Cosmos DB VNet Node Node Pod Pod AAD Pod Identity Key Vault Active Directory Active Directory Synced identity 

Identity and access management through AAD and RBAC 1. A developer authenticates to the AAD token issuance endpoint and requests an access token 2. The AAD token issuance endpoint issues the access token 3. The access token is used to authenticate to the secured resource 4. Data from the secured resource is returned to the web application Azure delivers a streamlined identity and access management solution with Azure Active Directory (AAD) and Azure Kubernetes Services (AKS) AKS Azure Active Directory Token Token 1 2 3 4 Developer 

Pod identity 1. Kubernetes operator defines an identity map for K8s service accounts 2. Node Managed Identity (NMI) watches for mapping reaction and syncs to Managed Service Identify (MSI) 3. Developer creates a pod with a service account, and pod uses standard Azure SDK to fetch a token bound to MSI 4. Pod uses access token to consume other Azure services; services validate token Kubernetes Kubernetes controller Azure MSI Azure Identity Binding Active Directory Pod Identity NMI + EMSI Pod Token Azure SQL Server 1 2 3 4 Developer <\> 

Image Security Your private registry, with built-in Helm chart support, only deploys validated images and can be automatically geo-replicated to the data center close to where your users are Developer CI/CD Pipelines Azure Container Registry Azure Kubernetes Service Image scanning Fail Pass Vulnerability scanning Actionable recommendations Admin 

Networking Secure your Kubernetes workloads with virtual network and policy-driven communication paths between resources Kubernetes cluster: Azure VNET Agent node Pods Containers kubelet Control plane Internal Load Balancer Ingress Controller Agent node Pods Containers kubelet … Namespace External DNS Private cluster App Gateway Egress lockdown 

Secure network communications with VNET and CNI 1. Uses Azure subnet for both your containers and cluster VMs 2. Allows for connectivity to existing Azure services in the same VNet 3. Use Express Route to connect to on- premises infrastructure 4. Use VNet peering to connect to other VNets 5. Connect AKS cluster securely and privately to other Azure resources using VNet endpoints AKS VNet integration works seamlessly with your existing network infrastructure AKS subnet Backend services subnet Azure VNet A On-premises infrastructure Enterprise system Other peered VNets VNet peering Azure Express Route AKS cluster SQL Server 1 2 3 4 Service Endpoint 5 Azure SQL PaaS DB 

Governance Cloud Architect Developer Cluster-1  Cluster-2 Cluster-3 AKS Azure Policy Cluster-3  Cluster-2 Cluster-1 Compliance reports Assigns a policy across clusters Real-time enforcement of policy and feedback Compliance reports for the entire environment, with pod-level granularity Dynamically enforce guardrails defined in Azure Policy across multiple clusters—nodes, pods, and even container images can be tracked and validated at the time of deployment or as part of CI/CD workflows 

Azure Pipelines build audit & enforcement using Azure Policy 1. Cloud architect assigns a policy across clusters; policy can be set to block non- compliance (deny) or generate non- compliance warnings (audit) 2. Developer makes code change that kicks off a build on Azure Pipelines 3. Azure Pipelines evaluates the request for policy compliance 4. If policy is set to deny, Azure Pipelines rejects the build attempt if any non- compliance is identified 5. If policy is set to audit, a non-compliance event is logged and the build is allowed to proceed Cloud Architect Developer Cluster-1 Cluster-2 Cluster-3 AKS Azure Policy 1 CI/CD Pipelines Pass Fail Deny policy 2 3 5 4 Yes No Compliance check 

Azure Policy for clusters 1. Cloud architect assigns a deployment policy across cluster(s) 2. Developer uses standard Kubernetes API to deploy to the cluster 3. Real-time deployment enforcement (acceptance/denial) provided to developer based on policy 4. Cloud architect obtains compliance report for the entire environment and can drill down to individual pod level Cloud Architect Developer Cluster-1  Cluster-2 Cluster-3 AKS Azure Policy Cluster-3  Cluster-2 Cluster-1 Compliance reports 1 2 4 3 

Threat protection Automated threat detection and best practices recommendation for Kubernetes clusters using advanced analytics from Azure Security Center Cluster Cluster Cluster Azure Security Center Continuous discovery of managed AKS instances Actionable recommendations for security best practices Detect threats across AKS nodes and clusters using advanced analytics Azure Kubernetes Service 

AKS Support in Azure Security Center 1. For managed subscriptions, each new AKS cluster and node are discovered in ASC 2. ASC monitors AKS cluster for security misconfigurations and provides actionable recommendations for compliance with security best practices 3. ASC continuously analyzes AKS for potential threats based on: a. Raw security events such as network data and process creation b. Kubernetes log audit Azure Security Center Continuous discovery of managed AKS instances Actionable recommendations for security best practices Detect threats across AKS nodes and clusters using advanced analytics Azure Kubernetes Service AKS security configuration API Server Control Plane Agent nodes Node1 Container runtime Security center Node2 Container runtime Security center Node3 Container runtime Security center Verified by Security Center Audit log Raw security events 1 2 a b …and reports any threats and malicious activity detected (e.g., “API requests to your cluster from a suspicious IP was detected”) 

Security summary 1. Image and container level security • AAD authenticated Container registry access • ACR image scanning and content trust for image validation 2. Node and cluster level security • Automatic security patching nightly • Nodes deployed in private virtual network subnet w/o public addresses • Network policy to secure communication paths between namespaces (and nodes) • Pod Security Policies using Gatekeeper • K8s RBAC and AAD for authentication • Threat protection on nodes 3. Pod level security • Pod level control using AAD Pod Identity • Pod Security Context 4. Workload level security • Azure Role-based Access Control (RBAC) & security policy groups • Secure access to resources & services (e.g. Azure Key Vault) via Pod Identity • Storage Encryption • App Gateway with WAF to protect against threats and intrusions AKS with RBAC Developer Azure Container Registry Kubernetes Admin Azure Storage SQL Database Cosmos DB Internal User Internal Load Balancer External User External Load Balancer Azure VNet Node Node Pod Pod AAD Pod Identity Ingress Controller Encrypted Storage 1 3 2 4 Azure Key Vault Ingress Controller App Gateway External DNS Active Directory 

Run anything, anywhere 

Run anything, anywhere Your choice of… Container Linux Windows Region 35+ regions worldwide Environment IoT Edge Public clouds Azure Stack Azure Government Private data centers 

Azure Kubernetes Service (AKS) support for Windows Server Containers Now you can get the best of managed Kubernetes for all your workloads whether they’re in Windows, Linux, or both • Lift and shift Windows applications to run on AKS • Seamlessly manage Windows and Linux applications through a single unified API • Mix Windows and Linux applications in the same Kubernetes cluster—with consistent monitoring experience and deployment pipelines 

Azure Hybrid Innovation anywhere with Azure Management | Security + Identity | App + Data Services | Dev Tools + DevOps Azure IoT Any edge device Azure Arc Any datacenter, any cloud Integrated systems Azure Stack Microsoft Azure Microsoft Azure 

Azure Arc Bring Azure services and management to any infrastructure Run Azure data services anywhere Extend Azure management across your environments Adopt cloud practices on-premises Implement Azure security anywhere Azure Arc is a set of technologies that extends Azure management and enables Azure services to run across on-premises, multi-cloud, and edge 

Azure Arc | Customer use cases Organize and govern across environments Get Kubernetes clusters and servers that are sprawling across clouds, datacenters and edge under control by centrally organizing and governing from a single place At-scale Kubernetes app management Deploy and manage Kubernetes applications at scale across environments using DevOps techniques. Ensure that applications are deployed and configured consistently from source control, at scale Run data services anywhere Deploy and manage data services where you need it for latency or compliance reasons. Always use the most current technology and seamlessly manage and secure your data assets across on-premises, clouds and edge Multi-cloud Datacenter & hosted 

Azure Arc-enabled Kubernetes clusters • Central inventory and monitoring of the sprawling assets running anywhere from on-premises to edge • Consistently apply policies, role-based- access-controls (RBAC) for at-scale governance • Deploy Kubernetes resources to all clusters using a GitOps-based workflow Identity RBAC Monitoring Policy Azure Arc Kubernetes Azure Stack On-premises Multi-cloud Edge 

Azure application services Event Grid Logic Apps App Service Functions API Management Azure | On-premises | AWS | GCP Run your apps, anywhere Accelerate development with turnkey services Gain the productivity of PaaS with the control of Kubernetes Run your apps anywhere with Azure Arc