AKS-Series 4 [Video]: AKS - More than just a Managed Kubernetes by Nico Meisenzahl

Link to recording: https://youtu.be/8ayRKBkhCYM
Link to meetup: https://www.meetup.com/de-DE/Microsoft-Azure-Zurich-User-Group/events/283143131/

You are already running Kubernetes-based workloads on Azure Kubernetes Service and want to get more out of it?

This is your session! In this talk, Nico will show you all the nice features you get with AKS besides just Kubernetes. Learn all the details about the available add-ons, integrations, and toolsets to integrate and secure your Kubernetes-based applications.

Furthermore, Nico will give a preview of potential new features from the AKS roadmap.

Bio:
Nico Meisenzahl works as Senior Cloud & DevOps Consultant at white duck. As an elected Microsoft MVP and GitLab Hero, his current passion is for topics around Cloud Native and Kubernetes. Nico is a frequent speaker at conferences, user group events and Meetups in Europe and the United States.

Links:
https://meisenzahl.org
https://twitter.com/nmeisenzahl
https://www.linkedin.com/in/nicomeisenzahl

Transcript

1. Azure Kubernetes Service – more than just a managed Kubernetes Microsoft Azure Zürich User Group, March 2022
2. Nico Meisenzahl • Cloud Solution Architect at white duck • Microsoft MVP, GitLab Hero • Cloud Native, Kubernetes & Azure © white duck GmbH 2022 Phone: +49 8031 230159 0 Email: nico.meisenzahl@whiteduck.de Twitter: @nmeisenzahl LinkedIn: https://www.linkedin.com/in/nicomeisenzahl Blog: https://meisenzahl.org
3. Agenda • Azure Kubernetes Service – a managed K8s • AKS features (my picks) • AKS add-ons & extensions • further resources © white duck GmbH 2022
4. AKS – A MANAGED K8S © white duck GmbH 2022
5. Azure Kubernetes Service “Deploy and scale containers on managed Kubernetes” “Deploy and manage containerized applications more easily with a fully managed Kubernetes service” “Build on an enterprise-grade, more secure foundation” © white duck GmbH 2022 https://azure.microsoft.com/services/kubernetes-service
6. A managed K8s, but … • what you will get out of the box • Kubernetes à great flexibility also introduces complexity! • a fully managed control plane • worker nodes you need to care about • fully managed Kubernetes is possible • not enabled by • can cause issues (you must be ahead of all changes) • addons / integrations required © white duck GmbH 2022
7. Fast changing world • AKS/Kubernetes is a fast changing world • integrations/features evolve quickly and need to be implemented on an ongoing basis • fire and forget is not an option • you will need a team to operate your clusters • if you are not able to provide this, AKS/Kubernetes is not an option for you à Azure Container Apps (preview) might help © white duck GmbH 2022
8. That said, AKS … • is the best choice if you require Kubernetes • can help you a lot and make your life much easier • perfectly integrated with other Azure services • provides you with useful open-source integrations © white duck GmbH 2022
9. AKS FEATURES (MY PICKS) © white duck GmbH 2022
10. Private AKS • expose API Server via Private Link into an internal subnet • expose services into an internal subnet using internal Load Balancer • access Azure PaaS services via Private Link endpoints • Container Registry • Storage services (Storage Account, Databases, …) • can introduce some complexity due to networking and DNS • there will be an updated version (v2) in the future which reduces the complexity © white duck GmbH 2022
11. Azure AD integration • assign IAM to Azure AD user's identity or directory group membership • integrated with the Azure Portal and CLI • allows to disable local cluster-admin account • can be assigned via Azure Roles or Kubernetes Roles/RoleBindings • support for Group Managed Service Accounts (GMSA) for your Windows nodes (preview) • https://docs.microsoft.com/azure/aks/managed-aad © white duck GmbH 2022
12. Azure AD Pod Identity (preview) • assigns Azure AD identities to Pods to leverage Azure resource that depends on AAD as an identity provider • e.g., securely talk with databases or Storage Accounts without injecting secrets and connection strings • no code changes required (relies on the default credentials) • will not leave preview! • the successor will be Azure AD Workload Identity • same outcome, new implementation © white duck GmbH 2022
13. Azure AD Workload Identity (preview) • successor of Azure AD Pod Identity • implements known-issues and learnings • removes scale and performance issues • supports Kubernetes clusters hosted in any cloud or on- premises • supports both Linux and Windows workloads • removes the need for CRDs and pods that intercept Instance Metadata Service (IMDS) traffic © white duck GmbH 2022
14. Azure AD Workload Identity © white duck GmbH 2022
15. Auto-upgrade & node upgrade • AKS can automatically upgrade clusters and nodes • there are different upgrade channels • none, patch, stable, rapid, node-image • https://docs.microsoft.com/azure/aks/upgrade-cluster#set-auto-upgrade- channel • manifests & API calls need to stay up-to-date for stable/rapid • do not miss to define a maintenance windows (preview, currently best-effort only) • node auto-repair • AKS automatically try to fix node issues if node is “NotReady” • steps are reboot, reimage, recreate • https://docs.microsoft.com/azure/aks/node-auto-repair © white duck GmbH 2022
16. Autoscaling & Spot instances • Cluster Autoscaler allows node scalling (on a node pool level) • support for Azure Spot VMs (on a node pool level) • take advantage of unused capacity at a significant cost savings • Virtual Node interation via ACI © white duck GmbH 2022
17. Integrated Storage • AKS integrates with Azure Disk (incl. Ultra Disk) and Azure Files • REST and network based storage should be prefered where possible • stateless workload will make your life much easier • Azure HPC Cache and NFS (Storage Account) can be integrated via Kubernetes-native NFS • Azure Backup for AKS PVs (private preview) © white duck GmbH 2022
18. AKS and CSI • Azure Disk and Azure Files are supported by CSI since AKS 1.21 • CSI (Container Storage Interface) is the future of storage integration and will replace the in-tree implementation soon • CSI brings you many advantages • ZRS and ReadWriteMany support for Azure Disk • Kubernetes-native integrations for Volume snapshots, resizing and cloning • https://medium.com/01001101/azure-kubernetes-service-next-level-persistent- storage-with-azure-disk-csi-driver-c5a04ac775c1 • you will have to migrate existing clusters to use CSI • https://docs.microsoft.com/azure/aks/csi-storage-drivers#migrating- custom-in-tree-storage-classes-to-csi © white duck GmbH 2022
19. Azure Event Grid integration (preview) • Azure Events Grid now supports AKS as a source • allows to subscribing to AKS events for further integration • preview, and early stage • so far following events are supported • new Kubernetes version upgrade availability • new Node image version upgrade availability • https://docs.microsoft.com/azure/aks/quickstart-event-grid © white duck GmbH 2022
20. Microsoft Defender for Containers • environment hardening • provides visibility into misconfigurations and guidelines • vulnerability assessment • vulnerability assessment images after build, when stored in ACR and running in AKS • runtime protection • threat protection for clusters and Linux nodes generates security alerts for suspicious activities • why? • https://github.com/nmeisenzahl/hijack-kubernetes © white duck GmbH 2022
21. Microsoft Defender for Containers © white duck GmbH 2022
22. Microsoft Defender for Containers • upgrade Defender if you previously used it to get the latest features • Microsoft Defender for Kubernetes • Microsoft Defender for Containers Registries • also supports non-Azure environments (via Azure Arc) • Amazon Elastic Kubernetes Service (EKS) • Google Kubernetes Engine (GKE) • self-hosted CNCF-certified Kubernetes © white duck GmbH 2022
23. Confidential computing • allows you to protect your sensitive data while it's in use • allow user-level as well as OS code to define/use private regions of memory • based on Intel SGX (Software Guard Extensions) • requires DCsv2 VMs • supporting confidential containers out of the box • application is loaded in the trusted boundary (enclave) • https://docs.microsoft.com/azure/defender-for- cloud/defender-for-containers-introduction © white duck GmbH 2022
24. Enclave aware containers • are supported via the Open Enclave SDK • container development involves untrusted and trusted parts to the container application © white duck GmbH 2022
25. Uptime SLA • AKS is available with two tiers • free tier (default) • fewer replicas and limited resources for the control plane • paid tier packed by SLA • guaranteeing 99.95% (99.9% for non-AZ) • why? • I have seen issues with free tier in “smaller” regions due to lower prioritization of requests © white duck GmbH 2022
26. AKS ADD-ONS & EXTENSIONS © white duck GmbH 2022
27. Add-ons and Extenions • add-ons and extensions allowing to extend/integrate AKS with Azure services and open-source projects • are integrated with the Azure Resource Manager • easy to use © white duck GmbH 2022
28. AKS Add-ons • fully managed and supported by Azure • fixes are applied automatically on a weekly basis • minor/major changes are implemented via AKS updates • part of the Azure RM AKS resource provider • limited configuration options • https://docs.microsoft.com/azure/aks/integrations#add- ons © white duck GmbH 2022
29. AKS Extenions • relatively new with AKS • still on preview • already know concept from Azure Arc • easy integration • installation and lifecycle management via Azure tooling (API, CLI, …) • build on top of Helm Charts (but abstracted) • not managed nor automatically updated • separate resource provider within the Azure RM • therefore not yet supported in all IaC Tools (e.g. Terraform) • https://docs.microsoft.com/azure/aks/cluster-extensions © white duck GmbH 2022
30. Add-On: Container Insights • entry point for logs and metrics & diagnostic data • integrates with Azure Portal • provides out-of-the-box workbooks and KQL queries • supports Prometheus endpoint scrapping • Azure Managed Grafana (currently private preview) • integrates via AKS data source • https://docs.microsoft.com/azure/azure- monitor/containers/container-insights-overview © white duck GmbH 2022
31. Add-On: Container Insights © white duck GmbH 2022
32. Add-On: Virtual Node • rapidly scale container workloads • no cluster autoscaler / node provisioning required • can also be useful for batch/job workload with special requirements (e.g., GPU) • https://docs.microsoft.com/azure/a ks/virtual-nodes © white duck GmbH 2022
33. Add-On: Azure Policy • integrates AKS with Azure Policies • based on Open Policy Agent Gatekeeper • can be enforced or audited • compliance across clusters © white duck GmbH 2022
34. Add-On: Azure Policy • use built-in definitions to base-level security • pod security baseline standards for Linux-based workloads • pod security restricted standards for Linux-based workloads • apply custom policies for your use-cases (preview) • https://docs.microsoft.com/azure/governance/policy/conce pts/policy-for-kubernetes © white duck GmbH 2022
35. Add-On: Application Gateway Ingress Controller • integrates Azure Application Gateway as an ingress controller (managed Ingress) © white duck GmbH 2022
36. Add-On: Application Gateway Ingress Controller • supports URL-based routing, cookie-based affinity, WAF, end-to-end TLS, … • TLS certificates can be served by Kubernetes secrets (Cert-Manager) • add-on is more limited than Helm deployment • https://docs.microsoft.com/azure/application- gateway/ingress-controller-overview © white duck GmbH 2022
37. Add-On: HTTP Application Routing • quick development option to spin up an Ingress Controller • not intended for production • spins up • Nginx Ingress Controller • External-DNS Controller (watching Ingress resources) • Azure DNS Zone • https://docs.microsoft.com/azure/aks/http-application- routing © white duck GmbH 2022
38. Add-On: Open Service Mesh • managed service mesh based on Open Service Mesh • lightweight service mesh implementing Service Mesh Interface • helps you with • service to service mTLS • traffic shifting (A/B, canary) • access control policies • monitoring and instrumentation • https://docs.microsoft.com/azure/aks/open-service-mesh- about © white duck GmbH 2022
39. Add-On: Azure Keyvault Secrets Provider • inject secret, certificates and keys into container workload without storing them outside of Azure Key Vault • based on Container Storage Interface • injection is done via volumes • can also be synced with Kubernetes secrets (and then inject via env) • https://docs.microsoft.com/azure/aks/csi- secrets-store-driver © white duck GmbH 2022
40. Extension: GitOps (preview) • abstracted GitOps setup based on Flux • already known from Azure Arc • integrated via ARM à no need to ”talk” to K8s directly • GitOps? • check out Azure Rosenheim Meetup for further details • https://github.com/whiteducksoftware/azure-meetup-rosenheim • https://docs.microsoft.com/azure/azure- arc/kubernetes/conceptual-gitops-flux2 © white duck GmbH 2022
41. Extension: Dapr (preview) • a portable, event-driven, runtime for building distributed applications across cloud and edge • https://docs.microsoft.com/azure/aks/dapr © white duck GmbH 2022
42. Extension: Azure ML (preview) • use AKS to train, inference, and manage machine learning models in Azure Machine Learning • Azure ML extension will deploy an Azure Machine Learning agent • https://docs.microsoft.com/azure/machine-learning/how- to-attach-arc-kubernetes © white duck GmbH 2022
43. Extension: KEDA (preview soon) • not yet available as extension • Kubernetes event-driven autoscaling • scale to zero • scale based on various events • scale-based on events from • Application Insights, Azure Monitor • Azure Blob, Azure Storage Queue • Azure Event Hub, Azure Service Bus • and many more © white duck GmbH 2022
44. FURTHER RESOURCES © white duck GmbH 2022
45. Get involved • AKS office hours (bi-weekly call) • https://github.com/Azure/aks-gbb-officehours • AKS release notes • https://github.com/Azure/AKS/releases • AKS Roadmap • https://github.com/Azure/AKS/projects/1 • Stack Overflow AKS tag • https://stackoverflow.com/questions/tagged/azure-aks © white duck GmbH 2022
46. More details • AKS docs • https://docs.microsoft.com/azure/aks • AKS Reference Architecture • https://docs.microsoft.com/azure/architecture/reference- architectures/containers/aks-start-here • AKS checklist • https://www.the-aks-checklist.com © white duck GmbH 2022
47. Questions? • Slides: https://www.slideshare.net/nmeisenzahl © white duck GmbH 2022 Phone: +49 8031 230159 0 Email: nico.meisenzahl@whiteduck.de Twitter: @nmeisenzahl LinkedIn: https://www.linkedin.com/in/nicomeisenzahl Blog: https://meisenzahl.org