Presented in the CNCF Webinar series
What’s New in Kubernetes 1.21
View Slide
© 2020 Cloud Native Computing Foundation2Anna Jung1.21 Enhancements Lead@antheajungPresentersDivya Mohan1.21 CommunicationsLead & Moderator@Divya_Mohan02Nabarun Pal1.21 Release Lead@theonlynabarun
© 2020 Cloud Native Computing Foundation3Agenda★ 1.22 Release Updates★ 1.21 Highlights★ SIG Updates★ Q&A
1.22 Release Updates
© 2020 Cloud Native Computing Foundation5Overview★ 1.22 Release Timeline○ Start Date: 26th of April 2021■ Enhancements Freeze: 13th of May 2021 23:59 PDT■ Code Freeze: 8th of July 2021 18:00 PDT○ Target Release Date: 4th of August 2021★ Kubernetes Release Cadence has changed to 3 releasesper year○ Roughly one Minor release every 4 months○ Enhance determinism and reduce risk
1.21 Highlights
© 2020 Cloud Native Computing Foundation7Kubernetes 1.21 - Power to the CommunityLogo Credits: Aravind Sekar / Behance
© 2020 Cloud Native Computing Foundation8Overview★ 51 total enhancements tracked in 1.21○ 13 Stable Enhancements○ 15 Graduating to Beta○ 21 Introduced Alpha features○ 2 Deprecations
© 2020 Cloud Native Computing Foundation9★ CronJobs graduate to Stable★ Immutable Secrets and ConfigMaps to Stable★ IPv4/IPv6 dual-stack support★ Graceful Node ShutdownMajor Themes!
© 2020 Cloud Native Computing Foundation10★ PersistentVolume Health Monitor★ Reducing Kubernetes Build Maintenance★ PodSecurityPolicy Deprecation★ TopologyKey Deprecation(more) Major Themes!
SIG Updates
API MACHINERY
© 2020 Cloud Native Computing Foundation13Efficient watch resumption after kube-apiserver reboot● Avoid tons of relists during kube-apiserversrolling upgrades● Avoid different instances ofkube-apiserver stuck with watchcachesynced to different resource versions forextended period of timeTracking IssueEnhancement ProposalStatus: StableStatus: Beta
© 2020 Cloud Native Computing Foundation14Apply for client-go's typed clients• Introduces a type-safe programmaticway to call server side apply fromclient-go• Client-go bundles a set of Applyconfigurations• Clears the path for server side apply to goGATracking IssueEnhancement ProposalStatus: Beta
© 2020 Cloud Native Computing Foundation15Immutable label selectors for all namespaces• Introduces a reserved label“kubernetes.io/metadata.name” whichwill be set to name of the namespace• Adds the ability to select namespaces byname reliably using traditional labelselector methods.Tracking IssueEnhancement ProposalStatus: Beta
APPS
© 2020 Cloud Native Computing Foundation17CronJobs graduate to Stable● CronJobs graduated to Beta inKubernetes 1.21● The old controllers and feature flags havebeen removed Tracking IssueEnhancement ProposalStatus: BetaStatus: Stable
© 2020 Cloud Native Computing Foundation18Graduate PodDisruptionBudget (PDB) to stable● Makes PDBs mutable● Address performance issues in the PDBControllerTracking IssueEnhancement ProposalStatus: BetaStatus: Stable
© 2020 Cloud Native Computing Foundation19TTL After Finished Controller● Finished resources like `Jobs` and `Pods`can accumulate in a cluster over time ifthey are not cleaned periodically● Make it easy to for the users to specify atime-based clean up mechanism forthemTracking IssueEnhancement ProposalStatus: StableStatus: Beta
© 2020 Cloud Native Computing Foundation20Random Pod Selection on ReplicaSet Downscale● Implements a randomized algorithm tochoose Pods to be killed on a ReplicaSetdownscale event● Also keeps into consideration the existingheuristics Tracking IssueEnhancement ProposalStatus: BetaStatus: Alpha
© 2020 Cloud Native Computing Foundation21Indexed Job● Provides users with support to run massivelyparallel programs● The Pods running can talk to each other withthe addition of a Headless ServiceTracking IssueEnhancement ProposalStatus: BetaStatus: Alpha
© 2020 Cloud Native Computing Foundation22Suspend Jobs● Adds a `suspend` boolean field to Jobspecification● It allows to suspend and resume jobs● Useful for preserving existing Jobmetadata like successful or failedcompletionsTracking IssueEnhancement ProposalStatus: StableStatus: Alpha
© 2020 Cloud Native Computing Foundation23ReplicaSet Pod Deletion Cost● Influence the order of Pod deletion ondownscale events● `controller.kubernetes.io/pod-deletion-cost`can be provided as an annotation● Pods with Lower pod deletion cost will bedeleted firstTracking IssueEnhancement ProposalStatus: StableStatus: Alpha
AUTH
© 2020 Cloud Native Computing Foundation25Pod Security Policy● Deprecation starts in 1.21● Planned to be removed in 1.25● Replacement being worked on Tracking IssueEnhancement ProposalStatus: DeprecatedDeprecation Blog
© 2020 Cloud Native Computing Foundation26External client-go credential providers● Allow out-of-tree implementationcredential providers in client-go● Ensure that credentials can be rotatedwithout restarting clients● Eventually make client-go vendor neutralby deprecating `gcp` and `azure`authentication optionsTracking IssueEnhancement ProposalStatus: StableStatus: Beta
© 2020 Cloud Native Computing Foundation27Bound Service Account Tokens: separate RootCAConfigMapfrom BoundServiceAccountTokenVolume● Audience of issued JWTs would be bound● Auto-configured service account tokens inpods use projected tokensTracking IssueEnhancement ProposalStatus: StableStatus: Beta
© 2020 Cloud Native Computing Foundation28Bound Service Account Tokens: RootCAConfigMap to GA● Publishes a `kube-root-ca.crt` ConfigMap toevery namespace● This ConfigMap contains a CA bundle used forverifying connections to the kube-apiserver Tracking IssueEnhancement ProposalStatus: Stable
© 2020 Cloud Native Computing Foundation29Service Account signing key retrieval● Allow authorized systems to discover theinformation they need to authenticateKubernetes Service Account tokens● Eventual goal is to make the Kubernetes APIServer OIDC compatibleTracking IssueEnhancement ProposalStatus: Stable
CLI
© 2020 Cloud Native Computing Foundation31Include kubectl command metadata in http request headersAllows cluster admins to use this information fortelemetry and debuggingTracking IssueEnhancement ProposalStatus: StableStatus: Alpha
© 2020 Cloud Native Computing Foundation32Default container behaviorKubectl commands can consume this informationto decide the container to operate onTracking IssueEnhancement ProposalStatus: StableStatus: Alpha
Cloud Provider
© 2020 Cloud Native Computing Foundation34Leader Migration for Controller Managers● Enables HA migration of in-tree to out-of-treecloud providers● Defines a set of guidelines and processes Tracking IssueEnhancement ProposalStatus: StableStatus: Alpha
INSTRUMENTATION
© 2020 Cloud Native Computing Foundation36Metrics Stability Enhancement● Deprecation lifecycle is in placeto better handle deprecation ofstable metrics● Deprecation notice in thedescription text (Deprecated fromx.y) and a warning logTracking IssueEnhancement ProposalStatus: Stable
© 2020 Cloud Native Computing Foundation37Structured logging● Structured logging available forKubeletTracking IssueEnhancement ProposalStatus: Alpha
© 2020 Cloud Native Computing Foundation38Expose metrics about resource requests and limitsthat represent the pod model● The `kube-scheduler` exposesoptional metrics that reports therequested resources and thedesired limits of all running pods Tracking IssueEnhancement ProposalStatus: Beta
© 2020 Cloud Native Computing Foundation39Defend against logging secrets via static analysis● Static analysis to be used duringtesting to prevent various types ofsensitive information from leakingvia logs Tracking IssueEnhancement ProposalStatus: Beta
© 2020 Cloud Native Computing Foundation40Metric cardinality enforcement● Turn off metrics to mitigate issuewhere metrics causes memoryleaks● Turn off metrics using`--disabled-metrics`● Set allow-list of label value formetrics using `--allow-label-value`Status: AlphaTracking IssueEnhancement Proposal
NETWORK
© 2020 Cloud Native Computing Foundation42Add IPv4/IPv6 dual-stack support● Dual stack mode to supportassigning both IPv4 and IPv6enabled by default Tracking IssueEnhancement ProposalStatus: Beta
© 2020 Cloud Native Computing Foundation43EndpointSlice API● In the v1 API, `topology` field wasremoved in favor of thededicated fields `nodeName`and `zone`● The Endpoints controller addsannotation to indicate overcapacity for an Endpointsresource with more than 1000endpointsTracking IssueEnhancement ProposalStatus: Stable
© 2020 Cloud Native Computing Foundation44Service Type=LoadBalancer Class● Option to specify the class of aload balancer implementation for`LoadBalancer` type of Service● Introduces field`service.spec.loadBalancerClass`in ServiceStatus: AlphaTracking IssueEnhancement Proposal
© 2020 Cloud Native Computing Foundation45NetworkPolicy port range● Option to enable`NetworkPolicyEndpoint` to targeta range of ports instead of asingle port when setting a networkpolicy● Introduces field `endPort` inNetworkPolicyStatus: AlphaTracking IssueEnhancement Proposal
© 2020 Cloud Native Computing Foundation46Service Internal Traffic Policy● Introduce a new field`spec.internalTrafficPolicy` inService that kube-proxy uses tofilter the endpoint it routes● When set to `Cluster` or missing, allendpoints are considered● When set to `Local`, only nodelocal endpoints are consideredStatus: AlphaTracking IssueEnhancement Proposal
© 2020 Cloud Native Computing Foundation47Block service ExternalIPs via admission● Allow users to disable the`externalIPs` feature of Servicesvia `DenyServiceExternalIPs`admission control● Blocks deployment of anyresource that uses `externalIPs`fieldTracking IssueEnhancement ProposalStatus: Stable
© 2020 Cloud Native Computing Foundation48Namespace Scoped Ingress Class Parameters● IngressClass parameters areNamespace scoped resourcesStatus: AlphaTracking IssueEnhancement Proposal
© 2020 Cloud Native Computing Foundation49Topology Aware Hints● Provide hints to Clustercomponents like kube-proxy toinfluence how traffic to is routedby keeping traffic within the zoneit originated from● Activate feature by settingannotation`service.kubernetes.io/topology-aware-hints` to `auto`Status: AlphaTracking IssueEnhancement Proposal
© 2020 Cloud Native Computing Foundation50Topology aware routing of services● Alpha topologyKeys API is nowdeprecated in favor of topologyaware hintsStatus: DeprecatedTracking IssueEnhancement Proposal
NODE
© 2020 Cloud Native Computing Foundation52Add sysctl support● Support for Linux sysctl interface totune OS parameters for deployedPods● Beta since 1.11, now stable Tracking IssueEnhancement ProposalStatus: Stable
© 2020 Cloud Native Computing Foundation53Provide RunAsGroup feature for Containers in aPod● Support `runAsGroup` field insidethe `securityContext` field in aPod● Beta since 1.14, now stable Tracking IssueEnhancement ProposalStatus: Stable
© 2020 Cloud Native Computing Foundation54Memory Manager● New component in Kubeletecosystem to guarantee memoryallocation for pods in theGuaranteed QoS class● single-NUMA and multi-NUMAallocation strategiesStatus: AlphaTracking IssueEnhancement Proposal
© 2020 Cloud Native Computing Foundation55Graceful node shutdown● `GracefulNodeShutdown`enabled by default● Kubelet detects node systemshutdown and gracefullyterminates pods running on thenodeTracking IssueEnhancement ProposalStatus: Beta
© 2020 Cloud Native Computing Foundation56Add downward API support for hugepages● Pods are able to fetch informationon their hugepage requests andlimits via the downward API● Supported if all workers in thecluster are min 1.20 versionTracking IssueEnhancement ProposalStatus: Beta
© 2020 Cloud Native Computing Foundation57Remove cAdvisor json metrics from the Kubelet● Deprecated since 1.18● Removed permanentlyTracking IssueEnhancement ProposalStatus: Stable
© 2020 Cloud Native Computing Foundation58Add configurable grace period to probes● Introduce probe-level`terminationGracePeriodSeconds`● Override the pod-level`terminationGracePeriodSeconds`for liveness or startup termination,and will be ignored for readinessprobesStatus: AlphaTracking IssueEnhancement Proposal
© 2020 Cloud Native Computing Foundation59Extend podresources API to report allocatableresources● Addition to Kubelet pod resourcesendpoint to allow third partyconsumers to learn about thecompute resources allocated to aPod● Introduces`GetAllocatableResources`endpointStatus: AlphaTracking IssueEnhancement Proposal
© 2020 Cloud Native Computing Foundation60CRIContainerLogRotation● Enables container log rotation forContainer Runtime Interface (CRI)container runtime● Beta since 1.11, now stable Tracking IssueEnhancement ProposalStatus: Stable
SCHEDULING
© 2020 Cloud Native Computing Foundation62Honor Nominated node during the new schedulingcycle● Define a preferred node to speedup scheduling● Introduce a new field`.status.nomindatedNodeName`in Pod Tracking IssueEnhancement ProposalStatus: Alpha
© 2020 Cloud Native Computing Foundation63Namespace selector for pod affinity● Introduces `namespaceSelector`to allow setting namespacesdynamically for affinity term● Introduces`CrossNamespacePodAffinity`that limits which namespaces areallows to have pods with affinityterms that cross namespacesTracking IssueEnhancement ProposalStatus: Alpha
STORAGE
© 2020 Cloud Native Computing Foundation65Immutable Secrets and ConfigMaps● Protects against inadvertent updates toSecrets and ConfigMaps● Kubelet doesn’t poll for such Secrets andConfigMaps resulting in performanceimprovements Tracking IssueEnhancement ProposalStatus: BetaStatus: Stable
© 2020 Cloud Native Computing Foundation66Persistent Volume Health Monitor• Improves UX of handling underlyingstorage issues in PersistentVolumes• Early signal for impending storage failureevents preventing serious problems Tracking IssueEnhancement ProposalStatus: BetaStatus: Alpha
© 2020 Cloud Native Computing Foundation67Storage Capacity Constraints for Pod Scheduling• Prevents Pod creation getting stuck dueto unavailability of requested storage• Schedule pods to nodes where therequested storage capacity is available Tracking IssueEnhancement ProposalStatus: Beta
© 2020 Cloud Native Computing Foundation68Generic Ephemeral Inline Volumes● Extend Kubernetes with CSI drivers thatprovide light-weight, local volumes● New volume source, the so-calledEphemeralVolumeSource contains allfields that are needed to create avolume claim● The Pod is the owner of the volume claim,if the pod gets deleted the garbagecollector deletes also the volumeStatus: AlphaTracking IssueEnhancement ProposalStatus: Beta
© 2020 Cloud Native Computing Foundation69Prioritizing nodes based on volume capacity● Optimize Volume resource usage● Schedules pods on nodes where theavailable capacity is close to requestedcapacityStatus: AlphaTracking IssueEnhancement Proposal
© 2020 Cloud Native Computing Foundation70Azure file in-tree to CSI driver migrationIf you have the Azure File CSIDriver, you can turn on the featuregate CSIMigrationAzureFile toenable the sameStatus: AlphaTracking IssueEnhancement ProposalStatus: Beta
© 2020 Cloud Native Computing Foundation71Service Account Token for CSI Driver• Allow CSI driver to requestaudience-bounded service accounttokens of pods from kubelet toNodePublishVolume.• Provide an option to re-executeNodePublishVolume in a best-effortmanner.Status: AlphaTracking IssueEnhancement ProposalStatus: Beta
TESTING
© 2020 Cloud Native Computing Foundation73Reducing Kubernetes Build Maintenance● The project used to maintain multiplebuild systems● CI processes using Bazel moved to `makebuild`● Bazel based build and related tooling areremoved Tracking IssueEnhancement ProposalStatus: Stable
Release Team Shadow Program
© 2020 Cloud Native Computing Foundation75Release Team Shadow Program★ Release Team Roles○ Release Team Lead○ Enhancements○ CI Signal○ Bug Triage○ Docs○ Release Notes○ Communications★ 1 lead : 3 - 5 shadows★ ~4 months // weekly workload varies depending on team★ Release Team Shadows Github repo
Questions?
Thank You