Table of contents:
- Introduction
- Hybrid Cloud Experience and Red Hat OpenShift Platform Plus
- Managed Cloud Services
- Telco and Edge
- Core, Platform and Developer Tools and Services
Apps Cloud Native Microservices ISV Packaged Apps Physical Virtual Private cloud Public cloud Red Hat Enterprise Linux Edge cloud Red Hat OpenShift Red Hat Open Hybrid Cloud Data, Analytics & AI/ML Enabling any application, on any infrastructure, in any location
7 Available as self-managed platform or fully managed cloud service Red Hat OpenShift Dedicated2 Red Hat OpenShift service on Amazon Web Services1 Microsoft Azure Red Hat OpenShift Red Hat OpenShift on IBM Cloud1 Managed Red Hat OpenShift services Self-managed Red Hat OpenShift On public cloud, or on-premises on physical or virtual infrastructure3 Source: 1 In preview as of 1/1/2021. Also available as Red Hat OpenShift Dedicated managed service running on user-supplied AWS infrastructure. 2 Red Hat managed service running on user-supplied GCP infrastructure 3 See docs.openshift.com for supported infrastructure options and configurations Start quickly, we manage it for you Cloud managed You manage it, for control and flexibility Customer managed
MANAGED CLOUD SERVICES Self-managed clusters and applications Foundations for Managed Services and Telco and Edge HYBRID CLOUD EXPERIENCE OpenShift as a (SRE) Managed Service Managed (SRE) Application, Data and Management Services Unified Experience Security Everywhere Platform Consistency TELCO & EDGE 5G CORE and 5G RAN Near edge and Far edge From and to the edge 10 Applications in hybrid clouds and clusters
18 months life • Even releases are designated as EUS • A new EUS to EUS upgrade experience • OpenShift 4 EUS be available to both standard and premium support • 3 OCP releases per year (same as Kubernetes) Link to the Red Hat Blog - https://cloud.redhat.com/blog/time-is-on-your-side-a-change-to-the-openshift-4-lifecycle Red Hat OpenShift Container Platform Life Cycle Policy - https://access.redhat.com/support/policy/updates/openshift Changes to OpenShift Minor Release (4.y) Life Cycle
Customers desire “regionality” for these tools This is a big shift in thinking: 1. Embrace the Hub—an infrastructure cluster—as the unit of regionality to run OpenShift Plus. 2. Update deployment patterns to reflect Hub and spoke OpenShift clusters HyperShift ACM ACS Quay
B Multi-cluster layer Cluster A Node Node Node Pod Pod Pod Node Node Node Pod Pod Pod Ingress/Router Multicluster management Observability ⠇Discovery ⠇Policy ⠇Compliance ⠇Configuration ⠇Workloads Machine Pool’s tuning/hardware offload config Machine Pool’s tuning/hardware offload config Ingress/Router Standardized tools for your 1st and 100th cluster East/West IPsec Multicluster security Kube native declarative security | DevSecOps Container registry Container Builds ⠇Security Scanning ⠇Geo Replication Global Ingress/Egress | Global LB | Service Mesh Federation Multi-cluster Storage Hybrid Cloud and OpenShift Platform Plus
for Kubernetes Cosigned manifests and secrets management enable faster application delivery with security throughout the supply chain. Security Everywhere Deploy single, compact, multinode, remote worker nodes, cloud services, and HyperShifted clusters from hierarchical tiers of management hubs. Platform Consistency Reduced Exposure and Risk Increased Developer Productivity Leverage a single console experience from 1 cluster to thousands to deliver applications consistently across cloud services and on premises environments. Unified Experience Reduced Total Cost of Ownership Hybrid Cloud and OpenShift Platform Plus
17 Gateway API Platform-Native Load Balancing Ingress Controller Physical Virtual Private cloud Public cloud Managed cloud Edge Node Node Node Pod Pod Pod OVN OVS ▸ Unified traffic handling so you configure all your traffic the same way ▸ OVN for advanced traffic workloads ▸ IPv6 single/dual for scale ▸ eBPF for policy, traffic control, tooling, debugging, observability ▸ BGP-advertised services (FRR) ▸ Observability for improved understanding ▸ Multi-NIC support to align host networking ▸ HW Offload (OVS, IPsec, ...) for performance ▸ ▸ Any supported platform – add or swap easily, hybrid scenarios ▸ Flexibility to use native traffic distribution and filtering (e.g. WAF) for optimal performance ▸ Your traffic, your way: L4-L7, Envoy, by-pass Hybrid Cloud and OpenShift Platform Plus Istio Ingress Internet Submariner
async replication Easily add regional Disaster Recovery OpenShift and ACM common console for all shared file (RWX), block (RWO), and object storage classes NFS Kerberos mounts Secret Store CSI CSI Resize Transfer PVC/Snapshots between namespace CSI Ephemeral volumes Expansion of stateful sets Cloud providers CSIs CSI Migration from in-tree CSI Standardization OpenShift Multi-Cluster Storage 18 OpenShift Data Foundation Security Everywhere Platform Consistency Unified Experience Hybrid Cloud and OpenShift Platform Plus OpenShift Storage Consistent data foundation capabilities and experience for users and workloads: on-premises, in the cloud, and at the edge Standardization & Reduced Total Cost of Ownership Reduced Risk & Increased Business Continuity Increased Developer & Admin Productivity
Enable teams to remediate issues more effectively Identify risk indicators across expanded use cases Security Everywhere Provide consistent security data across the OpenShift and Kubernetes ecosystem Enable teams to scale policy workflows in a repeatable way Platform Consistency Innovate with confidence by bridging the skill gap Reduce complexity to focus resources Accelerate operationalization with managed services. Improve feedback loops, and create a shared languages for security and development teams Unified Experience Break cross functional barriers to reduce cost Hybrid Cloud and OpenShift Platform Plus
beyond container base images (Java / Go packages) Trust & verify with signatures Security Everywhere Geo-replication on all platforms via the Quay operator Consistent consumption experience with pull-thru caching of external registries Platform Consistency Remediate security risk before production Hybrid content distribution Visual consistency with a completely new UI Integration of quay.io into console.redhat.com Unified Experience Consistent UX from self-managed to hosted Hybrid Cloud and OpenShift Platform Plus
Support to extend using remote write for storage and platform monitoring for OpenShift Workloads Correlation Consistency Extensible visualization flexibility enabling Dashboards or OpenShift Console Visualization Across Cluster Workloads. Log Exploration Tools Visualization Flexibility OpenShift Long & Short-Term Ingest Metrics Storage Optimized API Experience in OpenShift Console Enhancements to distinguish between Workload Monitoring & User Defined Projects to monitor flexible Hybrid Workloads and Applications Simplified Hybrid Observability OpenShift Console Optimized for Hybrid Workload Monitoring Hybrid Cloud and OpenShift Platform Plus
compliance requires governance of traffic in, around, and out of networks. Security Everywhere Developers and administrators require a common understanding of their traffic within and across cluster boundaries. Platform Consistency Network Policy and Governance Network Traffic Flow and Topology Whether one cluster or one hundred, developers and cluster administrators require seamless connectivity across applications. Unified Experience Network Traffic Metrics and Tracing
Pod Pod IP Networking Service Discovery and Load Balancing NetworkPolicy Pod Pod Pod Pod IP Networking Service Discovery and Load Balancing NetworkPolicy Networking Service Discovery Security 24 Kubernetes cluster networking with Submariner Submariner provides cross-cluster network infrastructure for OpenShift by extending the well-known Kubernetes networking objects
Low CAPEX and OPEX costs (bundling of CPs + CP as pods) Central Management of CPs (Easy operation & maintenance) Multi-arch support (e.g. CP x86, workers ARM) Network & Trust segmentation Decoupled Lifecycle of Control Planes & Workers Fast cluster bootstrapping (Control Plane as Pods, no master nodes) Economic Conserve resources Swift Mixed Iaas For CP and Workers (True Hybrid) Fleet-level Product Manager: Adel Zaalouk
The complexities of 5G CORE Integration or Migration with 4G Core CNF certification Legacy Integration PAO, NUMA Awareness, Topology Scheduler Advanced Scheduling for Enhanced Performance New CPUs, NICs, SmartNICs, GPUs, FPGA/ASIC, crypto engines NextGen Hardware Simplify mgmt Convergence of workloads on to a common platform Optimal resource utilization with enhanced performance Agile Infrastructure with the latest Hardware [Efficient, Scale, TCO]
API Kube API server Kubelet Kube scheduler Topology aware scheduler plugin Kubelet NFD- topology -updater Pod Pod resources API Kubernetes Control plane poll Pod Pod NFD NODE Telco 5G Core and Edge
services including Networking, Storage, AI/ML in a separate cluster on ARM cores in the NIC. Tenant workloads in x86 cluster SmartNICs Crypto engines support inline IPsec and TLS offload Programmable FPGAs and GPUs with 5G Core and RAN acceleration (GTP, FEC) Operators to manage Accelerators Accelerators Isolation of Tenant and Infrastructure cluster Accelerators to optimize resource usage OVN flow offload with Programmable FPGAs or ASICs Offload services: firewall, load balancer, QoS and Egress OVN Hardware Offload High Performance Networking w/ services Telco 5G Core and Edge
RAN Different edge sites can vary in network connectivity, space, and power/cooling Variability Need to deploy and manage hundreds to thousands sites and nodes Scale Nodes are tuned so that RAN realtime workloads can leverage advanced timing and hardware accelerations. Appliance like Performance Small Footprint and Optimized infrastructure Ease of Management through ACM and ZTP (RAN) Technology Evolution Telco 5G Core and Edge
2k SNO nodes provisioned and managed by RHACM • Policy Driven Upgrades Define groups of SNOs that can be upgraded independent of each other for more granular multi-cluster management • ZTP Everything DU, C-RAN Hub, CU, Hub Cluster, additional infrastructure (image repository, NBDE Server, DHCP Server, etc…) Aimed at regional distributed on-prem disconnected deployment. Enabling customer’s automated path from uninstalled infrastructure to application running on an OpenShift cluster. ZTP - Zero Touch Provisioning DU - Distributed Unit (5G RAN) Site Plan Manifests in Git Existing Infrastructure (Regional Data Center) Site 1 - DU Site 2 - DU Site 3 - DU CU Pool S S W W S W W W DU Remote Worker Nodes Single Node OpenShift Three Node Cluster Telco 5G Core and Edge CU - Central Unit (5G RAN) - future
- Cell Site Router (CSR) GMC - Grandmaster Clock BC - Boundary Clock OC - Ordinary Clock (GMC) NIC RU RU RU 32 • OCP Node as an Ordinary Clock or as a Boundary Clock with PTP (Timing & phase) and SyncE ( Frequency) • [O-RAN Approved] Low-latency, Node-local Event Bus w/ PTP Events and sidecar image for easy CNF (vDU) consumption RHEL CoreOS / OpenShift DU Workload RH Provided Event Bus Sidecar Red Hat PTP SW Stack (PTP Operator, ptp4l, phc2sys, …) PTP Events Event Bus PTP Events System Clock PTP/SyncE Telco 5G Core and Edge
Far Edge BIOS Hardware Red Hat CoreOS Red Hat OpenShift Telco Workload Enable application pods to set a required power performance profile • Extend PAO to allow the user to define a set of CPUs to offline if not needed • Tiered performance pools Develop automation to tune nodes for power savings prior to Zero Touch Provisioning Default all cores to lower power state at start-up Telco 5G Core and Edge
you Red Hat OpenShift Dedicated Red Hat OpenShift Everywhere A consistent platform no matter how or where you run Red Hat OpenShift Cloud Services Red Hat OpenShift Container Platform On public cloud, or on-premises on physical or virtual infrastructure You manage it, for control and flexibility 35 Red Hat OpenShift Service on AWS Azure Red Hat OpenShift Red Hat OpenShift on IBM Cloud Cloud native offerings jointly managed by Red Hat and Cloud Provider Managed by Red Hat
OpenShift Achieve compliance with more industry certifications such as HIPAA as and Gov certifications like FedRAMP HIGH Security Everywhere If it runs on OCP it should run on Managed OpenShift Platform Consistency Offer more flexibility in the kinds of workloads that can be run Reducing the barriers to adopting Managed OpenShift Allow users to create all Managed OpenShift clusters from one single location Unified Experience Simplicity of operations
OpenShift Support BYO Key for KMS and enable EBS encryption Security Everywhere Only run the platform when you need it. Pause it (and payments) when you don’t. Platform Efficiency Enable further security options for our sensitive customers Reduces the barriers to adopting Managed OpenShift Allow customers more options when choosing worker nodes to address many different workloads or budgets. Spot instances, GPU, Wavelength, AMD, dedicated Expanded choice Meet the customer where they are
cloud native solutions: KMS, DNS, LB • Cert-manager and improved lifecycle management of certificates Consistency • A shift to self-driven control plane with automated scaling, backups and DR of the control plane • Ability to customize RHCOS Experience Enables Hybrid Cloud and accelerates projects More choice and flexibility to meet standards and compliance • Enable Arm • More IBM P/Z innovations • Mixed CPU chip architecture • DPU/IPU integration with unique architectural approach Platform Enable new workloads and reduce TCO Core, Platform and Developer Tools
operator catalogs. Streamlined disconnected registry mirror. Factory installs of OpenShift for reduced startup time. Clusters can install with optional operators at day0. Global operator model with granular permission management and automatic failure recovery. Expanded functionality for Operators: reusable libraries, custom scaffolding, additional languages (e.g. Java) and underlying libraries support. Scaffold build pipelines in a git-friendly way. Namespace-scoped Helm repositories. Tested and certified Helm Charts from partners. Helm CLI support in oc client. Specialized scheduler for next generation workloads on Openshift. Secondary scheduler operator to onboard new schedulers. Multicluster Application Dispatcher operator to prioritize, queue and dispatch jobs to multiple clusters. Enabling Workloads on OpenShift 43 Operators and Helm Specialized Schedulers Deploy AI/ML or HPC workloads Core, Platform and Developer Tools More functionality out of the box Disconnected Customers receive updates faster
VLANs and static IPs. No DHCP required. Advanced network config via IPI on day 1 and day 2. Mix bare metal and VM nodes. Virtualized control plane and physical workers. Expand non-bare metal clusters with bare metal nodes. Hybrid Clusters Bootable Installer Advanced Host Network Config Infrastructure adapted to your network Reduced footprint and optimized resources Faster onboarding of platforms Bootable ephemeral installer. Create cluster zero easily. For on-prem or any cloud provider. Metal3 Core, Platform and Developer Tools
views for health-metrics and Insights on specific Kata Containers components. Node Feature Discovery Health Metrics Quickly Identify whether your cluster nodes/environment are eligible for the installation of the Kata runtime. Runtime Admission Control Isolate your untrusted workloads during admission. Enforce or exempt workloads to/from running in sandboxed runtimes (e.g., Kata Containers). SR-IOV with DPDK Enables running Cloud-Native Network Functions (CNFs) with sensitive network requirements. Swift Root-Cause Analysis Low-friction Installs Lower Time To Kata (TTK) Accelerate your Data-Plane with Tight Isolation Focus Less on HOW to Isolate, More on WHAT to Isolate Core, Platform and Developer Tools
move to Containerd as the runtime, and CSI for storage, thus future proofing consistency and application portability for Windows Platform Consistency Health Management of Windows Nodes with self healing will allow for better resiliency of the Windows nodes (e.g. recovering from a Kubelet crash) Unified Experience Core, Platform and Developer Tools Bring Your Own Host GA Announcement: Link
47 Managed Cluster Managed Cluster Managed Cluster OCP: OpenShift Container Platform OPP: OpenShift Platform Plus • ACM is just the start. ACS, Quay, Log Mgmt, and others will integrate with the OCP Console via dynamic plugins in 2022 • Dynamic plugins will enable partners & customers to create their own native integrations Management Hub Cluster • OCP will update to a fleet experience when the management hub is enabled • New lightweight multi-cluster operator enhances OCP screens via dynamic plugins • Fleet-wide auth for managed clusters Security Everywhere Platform Consistency Unified Experience New Hub/Managed Cluster Intelligence Unified OpenShift Platform Plus UX Core, Platform and Developer Tools Quay: Container Registry ODF: OpenShift Data Foundation ACM: Advanced Cluster Management ACS: Advanced Cluster Security
encryption Multi-Tenancy Security Everywhere Serverless part of the OpenShift Default deployment for stateless workload Creation of apps in “cluster agnostic” environment Platform Consistency Reduced exposure and risk Increased productivity Integration for platform services, Elevated Serverless Function experience. Event sources to cover the breadth of applicability Unified Experience Enhance developer experience for Event Driven solutions Core, Platform and Developer Tools
and manage service-level policies consistently across a zero-trust multi-tenant, environment. Security Everywhere A consistent platform with Istio service mesh across clusters, cloud providers, regions, and infrastructure types. Platform Consistency Reduced exposure and risk across your network Reduce complexity with a consistent platform experience A platform integrated service mesh - including operator installation, observability and visualizations, networking, API management,and more. Unified Experience Save time - solving integrations for you! Core, Platform and Developer Tools
Compliance Operator and Advanced Cluster Security Least privilege principles Security Everywhere More Public Cloud & Bare Metal providers vGPU support SNO resource optimizations Telco VNF validation program Warm migration (RHV) and basic migration (OSP) Platform Consistency Enhanced security compliance of VM OpenShift Everywhere Core, Platform and Developer Tools Improved visualization of individual VMs Overall resource utilization and intelligent diagnostics Data protection (via OADP) Disaster Recovery (via ACM) Unified Experience Manage and protect VMs at Scale
adoption leads to take informed decisions and make the migration and modernization process measurable and predictable Gather Insight Fully integrated toolkit leveraging multiple Open Source tools with a seamless user experience Extended Scope Reduce risks Provide value on each stage of adoption Help organizations safely migrate and modernize their application portfolio to leverage OpenShift Migration Guidance Ease OpenShift adoption Core, Platform and Developer Tools
migrations from on-premise to cloud solutions: ARO and ROSA Migrating from on-premise to cloud Supporting in-place migrations of your existing storage to OpenShift Data Foundation Storage migration Always improving and reducing the effort of migrating your applications at scale to OpenShift 4. Migrating from OCP3 to OCP4 Adopt with ease Red Hat's latest OpenShift technologies Core, Platform and Developer Tools
Experience and OpenShift Plus • Red Hat Advanced Cluster Management • Multi-Cluster Gateway for Ingress and Egress • OpenShift Multi-Cluster Storage • Red Hat Advanced Cluster Security • Red Hat Quay • Network Observability • Observability • HyperShift Managed Services Telco and Edge 57 Core, Platform and Developer Tools • Installation Updates and Provider Integration • Compute • Enabling Workloads on OpenShift • OpenShift on Bare Metal • OpenShift Sandboxed Containers • Windows • OpenShift Console • OpenShift CI/CD & GitOps • OpenShift Serverless • OpenShift Service Mesh • OpenShift Virtualization • Migration Toolkit for Applications • Migration Toolkit for Containers
Christian Stark, Bradd Weidenbenner, Sho Weimer ADVANCED CLUSTER MANAGEMENT - UPDATED Nov 29 2021 RHACM Roadmap Near Term (3 months) Mid Term (6 months) Long Term (9+ months) ACM • Scalability target: 2K Single Node OpenShift bare metal clusters (GA) • Central Infrastructure Management with Assisted Installer (GA) • Cluster lifecycle support: RHV, AWS Gov • Import and manage OpenShift on ARM (TP) • Discover non-OCP clusters from hyperscale clouds • Manage RHACM clusters from an Ansible Automation Platform • Submariner multicluster networking (GA) • ClusterPools, ClusterSets (GA) • Application Enhancements (Pre/Post hooks, SyncWaves support) • PolicySet definition for policy organization at scale • RHACM w/ RHACS (StackRox) integration (Phase 2 - User Experience) • OpenShift Data Foundation (aka OCS) with VolSync for Business Continuity (TP) • RHACM Hub DR backup and restore (GA) • Service Level Objectives (SLO) defined on the Grafana Observability dashboard (GA) ACM • Lifecycle HyperShift control plane and managed endpoints (TP) • Cluster lifecycle support: IBM Cloud, AWS China • Deploy/Destroy ROSA via CLI (TP) • Deploy/Destroy EKS via CLI (TP) • Multi-Hub architecture for edge management • Key and secret management via RHACM • Configuration Management Cloud Service launch • Tighter developer lifecycle integrations • Helm/Sigstore-Integration • Unified Hybrid Console • Support for PolicySets using PolicyGenerator • Ready-to-use Policies for certain Standards (e.g. NIST/HIPPA) • Integration with Integrity Shield (GA) • Provide a Scheduling Feature for Policies • Collaboration with Kyverno in Security Governance • Fleet view for User Workload monitoring ACM • Scalability target: Approaching 2K Single Node OpenShift bare metal clusters (TP) • Central Infrastructure Management with Assisted Installer (TP) • RHACM hub on IBM Z and Power (GA) • Cluster lifecycle: Microsoft Azure Gov • Identity Configuration Management operator (TP) • FIPS ready • RHACM w/ RHACS (StackRox) integration (Phase 1 - Central and Sensors) (TP) • Generation of Alerts for Policy Violations • Console enhancements for policy management • Support for ArgoCD & GitOps ApplicationSets • Easily incorporate existing configuration (Kubernetes, Rego) as RHACM policies • RHACM Hub DR backup and restore (TP) • VolSync Integration for Business Critical applications (TP) • Cluster health metrics for non-OpenShift (EKS, GKE, AKS, IKS) clusters • Service Level Objectives (SLO) defined on the Grafana Observability dashboard (TP)
What’s Next Multicluster lifecycle management Policy driven governance and compliance Advanced application lifecycle management Multicluster observability for health and optimization • Reduce infrastructure costs using externalized control planes with HyperShift. • Deploy hybrid and infrastructure agnostic clusters at scale. • Deploy SRE-managed and self-managed clusters from a single source of truth. • Creation of PolicySets and issue examples of PolicySets. • Cluster governance cloud service. • Tight integration of ACS and ACM within OpenShift Platform Plus. • Consolidated application deployment approach across various dev tools. • Quickly observe application health and status from any deployment source. • Include cross-cluster PV replication for business critical workloads. • Provide the complete fleet view for cluster health metrics. • Enable operations teams to analyze metrics using preferred tooling.
Data Foundation Near Term Mid Term Long Term FEATURES Platforms CSI API FEATURES Platforms ` CSI API FEATURES Platforms CSI API • Maintain CSI API • Disaster Recovery ◦ Multi-Cluster Regional DR for block using ACM (TP) • Data Federation - Object data replication • KMS PV encryption with multi tenant authentication • IPv6 single/dual stack (DP) • IBM ROKS (GAed) • ARO self managed service (GAed) • RHV (GAed) • BM IPI (GA) • Maintain CSI API • ROSA & OpenShift Dedicated • Disaster Recovery ◦ Multi-Cluster Regional DR for block using ACM (GA) ◦ Metro-DR Multi Cluster with ACM (TP) • IPv6 and Multus (GA) • Access filesystem via S3 API • Single Node support (TP) • Support for IBM HPCS KMS (DP) • Maintain CSI API • BM deployment on any platform • Windows Nodes (TP) • Disaster Recovery ◦ Multi-Cluster Regional DR for File using ACM (GA) ◦ Metro-DR Multi Cluster with ACM (GA) • Single Node (GA) • Single Node HA (TP) • Support for COSI - Object API for K8s • Support KMS vendors via KMIP OPENSHIFT CONTAINER STORAGE
Red Hat Advanced Cluster Security Vision Security across the entire application lifecycle Enable advanced incident response and hardening capabilities to enable the most mature risk management programs Enable teams to shift security left with our continued in improving vulnerability management and compliance workflows Advanced security workflows First class support for the OpenShift platform across clouds and managed services and security use cases Best in class OpenShift support Reduce security program costs Improve cybersecurity programs by making recommendations that would have an outsized impact on an organization's security posture and exposing program metrics to showcase ROI Program Management Provide teams with the information to prioritize the issues that matter most in their environment Enable effective prioritization workflows Creating an open source community focused on enabling Kubernetes security will enable us to tap into innovation pools not previously available Our commitment to open source
Term Logging • Customer Preview for New Loki Logging to eventually replace Elasticsearch • Customer Preview for New Vector Collector for Multi-Thread Scaling to replace Fluentd single threaded collectors • CloudWatch and Loki support for Log Forwarding with support for authentication mechanisms for ROSA STS Authentication Tokens. • Aggregate multi-line stack traces for non-JSON type logs. Logging • Tech Preview Loki Logs and Vector Collector for greater scale and reliability. • Log Exploration Tool provided natively inside the OpenShift Console. • Support for forwarding logs to GCP and AWS. • Support for AWS Cloud Watch authentication mechanisms for ROSA Automated STS Authentication Tokens. • Tech Preview Hybrid Logging Operators for OSD and ROSA Logging • GA Loki Logging and Vector Collector as an alternative storage engine to Elasticsearch. • Support higher throughput and resource-efficiency for Vector Multi-Thread collecting. • Allow OpenShift tenants to configure log forwarding themselves. • Improve OpenShift Logging Managed Service Experience and add-on logging workflows for managed tenants 67 Monitoring • Forward metrics to a remote endpoint. • Improve Monitoring component-relevant alerting rules to be more meaningful and actionable. • Ability to restrict User Workload Monitoring to only watch defined namespaces. • Ability to configure Monitoring to forward fired alerts to an external, user-owned Alertmanager. Monitoring • Create new alerting rules based on platform-defined metrics. • Introduce view into the OpenShift Console to show which services we scrape metrics from. • Adopt cluster-wide TLS configuration. • Allow tenants to create their own individual routing configuration independently from the cluster-wide Alertmanager config. • Proxy support for Alertmanager. Monitoring • Enable Prometheus Operator deployments per namespace to enable metrics to specific namespaces. • Prometheus Operator for collecting application meticx • Allow users to configure HTTP probes against an API. • More flexibility on how admins will operate the Monitoring stack (start small, grow out). • Simplified support for defining important SLOs for your application. Product Manager: Shannon Wilber OpenShift 4.11+ Logging v5.3 Logging v5.4 Logging v5.5+ OpenShift 4.9 OpenShift 4.10
Long Term Platform Data Collection OpenTelemetry • Re-branding: Red Hat OpenShift distributed tracing (formerly known as Red Hat OpenShift Jaeger) • Red Hat OpenShift distributed tracing platform: Jaeger 1.26 • Red Hat Openshift distributed tracing data collection: OpenTelemetry Collector 0.33 (Tech Preview) • Runtimes Integration: OpenTelemetry Client Libraries Platform Data Collection OpenTelemetry • OpenTelemetry Collector (GA) • Multi-Cluster Capabilities • Distributed Tracing (Umbrella Operator) for installing platform and data collection seamless Platform Data Collection OpenTelemetry • Visualization on OpenShift Console • Auto-Instrumentation through Operator 68 Product Manager: Mauricio "Maltron" Leal (3-6 months) (6-9 months) (9 months +)
Mid Term (6-9 months) Long Term (9+ months) HyperShift ` HyperShift • HyperShift for OSD • HyperShift for ARO • HyperShift on OpenShift Appliance • Cost operator support • Integration with Cert-manager operator • HyperShift Multi-Arch control-planes • Management Clusters Auto-scaling • Cross management clusters scheduling. • OVN support • Azure Infra provider • Platform None Infra Provider • KubeVirt as Infra provider • Bare Metal infra provider • In-place upgrades • Compliance Operator support • HyperShift for ROSA • HyperShift integrations with OCM • HyperShift Multi-Arch NodePools • ACM hub of hubs backed by HyperShift Product Manager: Adel Zaalouk HyperShift • HyperShift Etcd Operator • OLM In the Control Plane • Console support for HyperShift • HA / Single Replica Modes • Independent CP & NodePool Upgrades • Multiple versions of the CP on mgmt cluster • Reverse tunneling with api-server proxy (konnectivity) • Dedicated STS Roles for CP components • Private Link to support Private Clusters on AWS • AWS as infra provider • ACM Hosted Cluster life cycle (AWS) • Cluster Auto-scaling • Multiple service publishing strategies • FIPS Compliance for Hosted Clusters • HyperShift Operator Tracing • Metrics for Cluster Creation + Resource Consumption
with IEEE 1588) Q4 2021 Q1 2022 Mid-Year 2022 Single NIC OC Single NIC BC Single NIC OC Single NIC BC PTP Robustness Enhancements OC Events to CNF Single NIC OC Single NIC BC PTP Robustness Enhancements OC Events to CNF BC Events to CNF Planned Projected OC Events to CNF Q4 2022 Single NIC OC Single NIC BC OC Events to CNF GM - Grandmaster BC - Boundary Clock OC - Ordinary Clock Completed BC (No HA) Multi NIC BC Events to CNF SyncE 2023 PTP Robustness Enhancements BC Events to CNF Best Master Selection GMC via NIC GNSS BC (No HA) Multi NIC Single NIC OC Single NIC BC OC Events to CNF SyncE PTP Robustness Enhancements BC Events to CNF Best Master Selection GMC via NIC GNSS BC (No HA) Multi NIC LinuxPTP 3.1 LinuxPTP 3.1 LinuxPTP 3.1 PM: Robert Love
and GCP Region parity with OCP • GPU support • Spot instances • AMD instances • Support dedicated cloud instances Compute OSD / ROSA • Cluster hibernation • CloudFormation support for ROSA • Terraform provider support • Ansible support • Cluster-wide HTTP(S) proxy during creation for existing VPC • Use OVN as default • Support for NLB in addition to CLB • Use preexisting Route53 when installing in existing VPC • Edit existing node labels and taints Infrastructure ARO • Azure Government region support (Preview) • Expanded instance type support • Spot Instance support • Azure Norway West • GPU Support ARO • Azure Portal Cluster Creation GUI • Installation Configurability (version, etc) • Azure AppLens Integration Managed OpenShift Roadmap
into OCM • OCM: Cluster AddOns • OCM: Provision ARO clusters through OCM • OCM: Manage Upgrades ARO OSD / ROSA • Expanded region support • Log forwarding • User Workload Monitoring including Alerting • ROSA: Output YAML from CL • ROSA: Annual agreements in AWS Console • Scheduled upgrades for AddOns • ROSA: AWS console integration • Support selecting AZ’s • ROSA: STS OCM Provisioning • Ensure that editing the node label or taint applies to all existing nodes. • ROSA: CLI commands to delete / list IAM resources • Allow customer to customize web console branding Platform Managed OpenShift Roadmap
Near Term (3-6 months) Mid Term (6-9 months) Long Term (9+ months) GENERAL PROVIDERS GENERAL ` PROVIDERS GENERAL PROVIDERS Alibaba Cloud • User-provisioned infrastructure support Azure: • Better documenting of credential permissions • User-managed keys Azure Disk Encryption Sets • UltraDisks support GCP: • Better documenting of credential permissions IBM Cloud: • User-provisioned infrastructure support VMware vSphere: • Multi-cluster deployment support (single vCenter) Red Hat OpenStack: • Support MetalLB with BGP • DCN improvements for Telco/NFV • Better scaling using Kuryr OVN Install: • Improved Disconnected Workflow Upgrade: • Release upgrade graph data as a container • ‘oc’ enhancement to display upgrade paths IBM Cloud: • Installer-provisioned infrastructure support Alibaba Cloud • Installer-provisioned infrastructure support Azure Stack Hub (on-premise): • Installer-provisioned infrastructure support Azure: • Document restricted network installation GCP: • Deploy OCP to a shared VPC VMware vSphere: • Thin provisioning for OS Disk • Use HW version 15 for the VMs Red Hat OpenStack: • Allow strict anti-affinity for servers at install time • Tech-Preview - Support OCP workloads on OSP DCN for Enterprise use-cases • OVS Hardware offload - UPI Install: • Customer managed external DNS for Cloud Providers Upgrade: • EUS to EUS upgrades Azure: • Support for Azure China cloud instance • User-defined infrastructure tags GCP: • User-defined infrastructure tags VMware vSphere: • CPU & Memory reservations Install: • MachineSet-managed control plane • Additional disk for ETCD • Include/exclude capabilities based on user selection 79 PMs: Marcos Entenza (AWS, Azure, GCP, IBM Cloud), Gaurav Singh (Alibaba), Maria Bracho (VMware), Peter Lauterbach (RHV), Ramon Acedo Rodriguez (BM), Anita Tragler (OSP), Duncan Hardie (IBM Z & Power)
4.x EUS Contingent on Testing & Validation Upgrade Control Plane to 4.(n+1) & Skip Compute nodes All nodes upgrade to 4.(n+2) All nodes upgrade to 4.x EUS All nodes running 4.x EUS EUS to EUS Upgrades ➔ Alerts when node/pod fails to drain ➔ Zone awareness during upgrades ➔ Targeted upgrade blocking OpenShift Installation Managed ROSA /ARO/OSD Self Managed Cluster Lifecycle API OpenShift Hive (& Assisted Installer Service) Cluster Lifecycle API HyperShift (Hosted Cluster, NodePool) Product Managers: Marcos Entenza Garcia, Ramon Acedo Rodriguez, Adel Zaalouk, Ju Lim, Tushar Katarki Azure Stack Hub More platforms Installation Upgrades Installation, Updates, and Provider Integration Enable Hybrid Cloud Simplify onboarding Mitigate risk Cluster & Fleet Management OCM ACM
months) Long Term (9 months +) Mid Term (6-9 months) Product Manager: Ramon Acedo Rodriguez ` Installation + Hardware Management + Networking • Hybrid Clusters: deploy Bare Metal Worker Nodes from OpenShift on non-bare metal platforms • Central Infrastructure Management (power management) from RHACM for Single Node OpenShift nodes • Document adding custom Redfish-based controllers for telco partners with specialized hardware • Network logging improvements dsafasdf Installation + Hardware Management + Networking ` Installation + Hardware Management + Networking • Central Infrastructure Management (power management) from RHACM for bare metal nodes in managed clusters • High Availability for SNO pairs • Node Health Check (non-Machine API-based health checks) • Bootable Installer: ephemeral installer for on-prem, agnostic and long tail of platforms (bare metal focused) Version: 2021-11-23 82 • Support for day-1 configuration of bonds, static IPs and VLANs on nodes • Kubernetes NMState GA for bare metal clusters • Get and set node BIOS attributes • Monitor Node Health from Workloads: powered by Metal3 and Redfish, API to subscribe to hardware events • TLS for virtual media
the installation experience of OpenShift 4 on OpenStack in the User-Provisioned Infrastructure (UPI) and Installer-Provisioned Infrastructure (IPI) installation workflows - Improvements involve addressing new use cases and simplifying the installation workflows Telco & Edge Focus - OpenStack is one of the most popular platforms in Telco. OpenShift on OpenStack is strategic for many of such customers, running VNFs and CNFs together, Distributed Compute Nodes or SR-IOV for containers OpenStack Bare Metal Integration - Mixed environments of virtual and bare metal instances to address use cases such as direct hardware access (NVMe, SR-IOV, GPU, FPGA) or performance-sensitive apps VNFS VNFS CNFs VNFS VNFS VNFs Standard hardware KUBERNETES-NATIVE INFRASTRUCTURE Product Manager: Ramon Acedo Rodriguez
OSP) Current OpenShift 4 on Red Hat OpenStack Platform Reference Architecture OCP 4.4 & OSP 13/16 85 https://www.openshift.com/blog/ocp-4-on-osp-ra-blog-post
Containers • Metrics for Kata stack. • Additional dashboards in OpenShift console for health metrics. • Node Feature discovery for new installs. • Additional Logging from QEMU, the operator, and the kata runtime. • Integrations of the sandboxed operator in CPaaS Sandboxed Containers • Smart admission control for kata runtime (with / without ACS) • Kata 3.0 compatibility - Part I • Update to QEMU 6.2 • Update to RHEL 8.6 with extensions • SR-IOV with DPDK support • Qualify sandboxed containers on SNO • Dev Flows Integration Sandboxed Containers • Use CRI-O stats (instead of cadvisor) for metrics • Support External Control Plane (HyperShift) topology with sandboxed containers. • Integrate sandboxed containers in the logging stack • Kata 3.0 compatibility part II Near Term (~4 months = 4.10) Mid Term (~7 months = 4.11) Long Term (9+ months = 4.11+) Product Manager: Adel Zaalouk
Term Kubernetes • Secondary Scheduler Operator • Technology preview of autoscaling based on custom metrics with KEDA • Ability to deploy alternative recommender in VPA • GPU Fabric Manager enablement for NVIDIA DGX A100 (8 x GPUs per physical node) Kubernetes • In-Place upgrade of VPA • Krew • Cgroup V2 • Special Resource Operator GA • GPU Dashboard • GPUDirect RDMA enablement for multi-node training Kubernetes • Multi dimensional POD autoscaler • Swap Support for containers • Behavior detection driven recommenders in VPA • Enabling AI/ML and HPC workload • NVIDIA GPU on ARM systems • Checkpoint/Restore In Userspace • Cache container images 87 Product Managers: Mark Russell, Tushar Katarki, Gaurav Singh, Erwan Gallen Container Host RHEL CoreOS and Machine Config Operator • Kdump for diagnosing kernel crashes to GA • AWS GovCloud image publishing • coreos-installer live ISO customization UX • Improved node drain documentation and alerting • Improved CA certificate handling • Support of kubelet tlsSecurityProfile cryptographic policy flag Windows • Support for Bring your own Host that should enable Windows Containers support in UPI for platforms such as vSphere, Bare metal etc Container Host RHEL CoreOS and Machine Config Operator • rpm-ostree RHCOS images in OCI container • Password-protecting grub menu via Ignition • Automated hotfix distribution Windows • Support for ContainerD runtime • Improved Logging/Monitoring & Storage • Support for more network plugins such as NSX 3.0 Container Host RHEL CoreOS and Machine Config Operator • Simpler customization • Custom content • RHEL 9 based RHCOS • Reboot policies Windows • Support for Hosted Platforms (ARO. AMRO, OSD) • Knative Automations, Service Mesh support • Policy enforcement using OPA • Deeper UI changes
Multi-Architecture Near Term (3-6 months) Mid Term (6-9 months) Long Term (9+ months) OPENSHIFT MULTI-ARCHITECTURE • Compliance Operator • Multiple NICs • OVNKube IPSec Support • Kubernetes nmstate operator • Multus Plugins (IPVAN, Bridge with VLAN, Static IPAM, Host Device) • Vertical pod autoscaler • Horizontal pod autoscaling - Memory Tech Preview IBM POWER • PowerVS Automated Installer (IPI) • HyperShift IBM POWER • PowerVS Pre Existing Infrastructure (UPI) • Migration toolkit • FIPS compliance IBM POWER • Future platform support (Z Next) • Compliance Operator • Multiple NICs • CryptoExpress (CEX) Adaptors • OVNKube IPSec Support • Kubernetes nmstate operator • Multus Plugins (IPVAN, Bridge with VLAN, Static IPAM, Host Device) • Vertical pod autoscaler • Horizontal pod autoscaling - Memory Tech Preview IBM Z • Enhance KVM UPI with Secure Execution • HyperShift IBM Z • Migration toolkit • zVPC Automated Installer (IPI) • FIPS compliance IBM Z
and Auth • Update Control Plane to 1.23 • Tech Preview of Cert-manager • Consume Group membership information from an idp (OIDC) API and Auth • Update Control Plane Kubernetes Version to 1.24 • GA of cert-manager • Pod Security Admission • Improved audit logging (login/login failure details) • Reconcile SCC • Kube KMS R&D API and Auth • Hierarchical namespaces, • Automated Group sync • Prevent brute force logins 90 etcd • Master Node Scaling and Recovery Behavior Matches That of Worker Nodes etcd • Disaster recovery & Automated backups • Etcd bump etcd • Etcd Auto tune • Separate PIOPS volume for etcd supported as a Day 1 operation • Etcd certs signed by Custom CA Q1 CY2022
Infrastructure Near Term (3-6 months) Mid Term (6-9 months) Long Term (9+ months) ` API • AWS: support for other network types (EFA) • AWS: mixed spot instance machine sets • GCP: support for pd-balanced disk type • Set subnet for “service type” load balancer • Migration from MachineAPI to CAPI • Managed Compute for Control Plane ◦ Compute autoscaling ◦ Vertical rolling updates • Cluster API Phase 1 (TP) PROVIDERS • GCP: Out of tree cloud provider (TP) • IBM: Out of tree cloud provider (GA) • Alibaba: Out of tree cloud provider (GA) • vSphere: Out of tree cloud provider (TP) API • Azure: enable accelerated networking • Azure: worker node ephemeral disks • Azure: availability set support • Azure: UltraSSD support • GCP: GPU support • Cloud Controller Manager Operator PROVIDERS • vSphere: Out of tree cloud provider (GA) • GCP: Out of tree cloud provider (GA) • Azure: Out of tree cloud providers (GA) • OpenStack: Out of tree cloud provider (GA) • AWS: Out of tree cloud provider (GA) • Nutanix: Cloud provider (GA) API • CAPI completion • Native scaling groups • Improved Metrics • Automatically spread across Availability Zones PROVIDERS • Equinix Metal: cloud provider OPENSHIFT CLUSTER INFRASTRUCTURE
GitOps Short Term Mid Term Long Term Builds • Shipwright Tech Preview • Buildpacks build strategy • Shared secret/configmaps across namespaces (Shared Resource CSI Driver) Builds • Auto-pruning builds • Shipwright in Dev Console • Volumes support in Shipwright builds • Build triggers Builds • Shipwright Builds GA • Shipwright custom tasks for Tekton • BuildConfig to Shipwright migration guide • Build dependency caching 94 Product Manager: Siamak Sadeghianfar Pipelines • Pipeline as code concurrency control • In-cluster Tekton Hub • Pipeline resource quota guidance • Tetkon Chains and task signing Pipelines • Pipeline concurrency control • Improve workspace user-experience • Unprivileged builds in pipelines • Extended pipeline history and log retention • Manual approval in pipelines • Tekton Bundle support • GitLab support in pipeline as code Pipelines • Tekton Task bundles • Pipeline templates in pipeline builder • Pipeline reuse in pipelines • Workspace templates • Scheduled pipelines • Repository metrics in Dev Console GitOps • ApplicationSets GA • kubeadmin and OCP group suppor with RH SSO • Guidance on OpenShift configurations • Guidance on secret management GitOps • HashiCorp Vault integration • Argo CD multi-tenancy alignment with k8s • Application CRs in any namespace • Application CRs service account impersonation • Argo CD Helm deployment enhancements • Improve support for custom plugins in Argo CD GitOps • Application dependencies • Image updater • Notifications • Cluster-wide Argo CD control plane
Roadmap Mid Term (4.10/4.11) Long Term (4.11+) Near Term (410) Product Manager: Daniel Messer HELM HELM HELM • GitHub action for Chart Verifier tool • ArtifactHub integration in Developer Console • Kubelinter integration with Chart Verifier • IDE Tooling • Security and Signature • Migration from Templates and Samples Operator to Helm Charts • Helm CLI updates • Enabled namespace scope Helm chart repositories • Best practices guides for Helm on OpenShift • Multi-Cluster Support
(3-6 months) Mid Term (6-9 months) Long Term (9 months+) Service Mesh • Internal improvements to increase release cadence - keeping closer to upstream Istio. • Kiali enhancements for large meshes and federation • More flexible integration with Network Policies • Service Mesh support on OpenShift Virtualization Service Mesh • Support for external services (VMs, bare metal) • Support for IPv6 • Service Mesh Command Line Support Service Mesh • Centrally managed multi-cluster service mesh • Service Mesh integration with Advanced Cluster Manager (ACM) 97 Product Manager: Jamie Longmuir / Naina Singh Serverless • Functions General Availability ◦ Node and TypeScript ◦ Local developer experience using podman/docker ◦ On cluster build • Functions Tech Preview ◦ Quarkus, Python, Go, Rust, Spring Boot • Knative Kafka Broker - Tech Preview • Red Hat Event Sink ◦ Camel-K sinks ◦ Kafka sinks • OpenShift Dedicated Support , ROSA ◦ Unmanaged Add-On • Managed Kafka support Serverless • Functions General Availability ◦ Additional runtimes • Functions Tech Preview ◦ IDE Experience ◦ Customizable language packs • Security Enhancements ◦ mTLS natively in Knative • Knative Kafka Broker - GA • Cold start improvements • Serverless workflow orchestration • Red Hat Event Sources ◦ Ceph ◦ Kogito ◦ Data Grid • Additional monitoring alerts and dashboard Serverless • Serverless Cost Model • Serverless on SNO • Default deployment for stateless workload • Security Enhancements ◦ End to end encryption ◦ Broker and channel authentication/authorization • Make existing deployments Serverless • Integrations ◦ Keda with Eventing ◦ 3scale API Gateway • Stateful functions • Event Streaming support OpenShift Serverless and Service Mesh
support • Velero Backup/Restore Tech Preview • Service Mesh • IPv6 single-stack • Virtualization resource usage visualization Core Platform • Single VM deeper statistics visualization • OADP (Velero) Backup/Restore GA with Ecosystem Partners • Import / Export virtual machine • Online snapshots • Real time virtual machine • App High Availability (Pacemaker and fencing) Core Platform • AI/ML and Remote Visualization • Compliance operator • Improve disconnected experience • ACS integrations • Non-privileged containers 98 Deployment Options • Single Node Openshift (TP) • Tech Preview Bare Metal IBM Public Cloud • MTV support for Warm Migration from RHV Deployment Options • Single Node Openshift (GA) • Tech Preview Bare Metal additional Cloud vendors • VNF Certification • MTV support for migration from OSP • OCP Virtualization as control plane for OSP Deployment Options • Public Cloud Bare Metal GA/Tech Preview contd. • ARM support Near Term 1H CY2022 Mid Term 2H CY2022 Long Term 2023+
Migration Toolkit for Applications Near Term (Q4’21 - MTA 5.2.1) Mid Term (Q2’22 - MTA 6.0) Long Term (Q3’22 - MTA 6.1) • User experience driven by the Windup Web console UX • First JBoss EAP 8 rules: replace javax imports with jakarta equivalents • First OpenJDK 8 to OpenJDK 11 rules contributed, more to follow • Updated set of rules for Quarkus extensions • Updated targets in all IDE plugin flavors • FEATURES INTEGRATIONS • Application Inventory as the driver for user experience • Seamless integration between tools • Windup web console UX merged into the Application Inventory • Enhanced RBAC UX • Application portfolio management ◦ Integration with Git, SVN and Maven • Application assessment ◦ Custom questionnaire management • Applications analysis ◦ Centralized database and API FEATURES • Tackle Application Inventory (GA) • Tackle Pathfinder (GA) • Windup (GA) INTEGRATIONS • Seamless integration of Move2kube and Tackle Test as part of the Migration waves flow UX • Automate the creation of migration assets on application repositories ◦ Deployment manifests (Tech Preview) ◦ Automatically generated tests (Tech Preview) • Introduce migration waves FEATURES • Move2kube (Tech preview) • Tackle Test (Tech preview) INTEGRATIONS Migration Toolkit for Applications
(4Q’21) Mid Term (1Q’22) Long Term (2Q’22+) CORE EXPERIENCE • Provide org-type tenancy • Enhance subscription emails to include feedback surveys • Deploy apps in Serverless mode • Create data science models with RHODS • Easily add RHOAM to apps • Interact with Red Hat team via public Slack channel • Generate activation codes for Red Hat events and customers for faster signup and co-location 100 CORE EXPERIENCE • Unify Sandbox clusters with RHODS, RHOAM, and all pre-configured operators • Enable Sandbox styled clusters for App Studio services and subscriber workloads • Enhance telemetry from Sandbox console to capture catalog items and subscriber email domain • Access web terminal in cluster • Access Dev Sandbox from console.redhat.com • Build and run apps that deliver data science experimentation models • Send records from SQL databases to Kafka streams CORE EXPERIENCE • Enable proxy-based accessibility to data from Sandbox • Run and manage Dev Sandbox on ROSA and ARO clusters, along with AWS and Microsoft • View vulnerability advisories about images deployed from Quay • One-click signup and deployment of source code from Github 100 Product Manager: Parag Dave
Manager: Serena Nichols, Mohit Suman, Steve Speicher Developer Tools CR Containers CR Workspaces odo CR Containers CR Workspaces ` odo CR Containers CR Workspaces odo • Builds v2 • Knative Serving • Functions • Additional runtime support via devfiles • Local dev support with podman • Updates to include 4.8 z-streams • Profile work for app consumption • Single node profile installer work • Integration with podman • Better telemetry/metrics around downloads and usage • Plug-in recommendations for better OOTB experience • Initial support of devfile v2 for interoperability • Improved support for Bitbucket and Gitlab • New dashboard and improved management of credentials and secrets • Pipeline / GitOps enable • Helm support • Transition to outer loop • Updates to include 4.8/9 GA bits • Improved consumption for podman/RHEL minimal guest image • Apply single node installer support to additional operators • Import/export app • Create helm chart • Knative Eventing 101 Near Term Mid Term Long Term • Update mechanism for crc binary • Update mechanism for embedded image • Further operator enablement of single node case, further resource consumption improvements • Co-editing and team collaboration • Better integration and support for outer-loop • Multi-cluster support • Pipelines and Serverless integration • Easier discoverability in OpenShift Developer Console • More easily able to share workspace definitions with teams • Improved support of IntellIJ IDE • Simpler user management
redhatlivestreaming
kaonavi
visional_engineering_and_design
ch1aki
kniino
yuyatakeyama
manarobot
bobtani
lycorptech_jp
infiniteloop_inc
sansantech
kongmingstrap
yasumuusan
ammeep
hannesfritz
bluesmoon
geoffreycrofte
rasmusluckow
yeseniaperezcruz
danielanewman
philnash
shlominoach
schacon
jcasabona
tanoku
