Red Hat AI - What’s New and What’s Next Series ▸ Cadence ･ Quarterly cadence (1st month of the Quarter) ･ Find the schedule, registration links and recordings at ･ red.ht/ai-whats-new-whats-next Any forward looking statements are subject to change Join our quarterly updates to stay ahead of the curve in AI
of AI solutions across hybrid-cloud environments Increase efficiency with fast, flexible and efficient inferencing Simplified and consistent experience for connecting models to data Flexibility and consistency when scaling AI across the hybrid cloud Accelerate Agentic AI deployments
Consistent and Comprehensive foundation Edge Private Cloud Physical Virtual Public Cloud * NVIDIA, AMD, Intel, Google TPU supported in Red Hat AI. AWS Inferentia/Neuron IBM AIU are on our roadmap Hardware Acceleration Gen AI model Inference on RHEL/Linux or OpenShift/Kube Integrated Gen AI Linux server appliance for inference Generative and Predictive AI platform for inference, training, tuning and GenAIOps
AI Platform ▸ GA distributed inference (llm-d) ▸ New validated and optimized models ▸ vLLM enhancements ▸ LLM Compressor GA ▸ Modular and extensible approach for: data ingestion, synthetic data generation, tuning, evaluations. ▸ RAG enhancements & partner integrations ▸ Continual Post Training Algorithm ▸ Feature Store GA Agentic AI ▸ Model catalog and registry GA ▸ Model as a Service provider enhancements and API Mgt integration ▸ GPU as a Service enhancements ▸ AI experiences: AI hub and gen AI studio ▸ Model Context Protocol support & MCP Server access in gen AI studio ▸ Llama Stack API integration Single platform to run any model, on any accelerator, on any cloud
scalable inference Enterprise GenAI inference platform Holistic approach to optimize and operationalize deployment and scaling of open-source LLMs GPU Instinct TPU Pod 2 Instinct Pod 1 GPU Pod 3 TPU Accelerators Inference Server Distributed Inference Framework Models AI Gateway/API Mgt Model as a Service
scalable inference Enterprise GenAI inference platform Holistic approach to optimize and operationalize deployment and scaling of open-source LLMs GPU Instinct TPU Pod 2 Instinct Pod 1 GPU Pod 3 TPU Accelerators Inference Server Distributed Inference Framework Models AI Gateway/API Mgt Model as a Service
available as GA in all the Red Hat AI platforms 10 Day-Zero Model Support V1 Unified Architecture Expanding Hardware Support Instinct MI35x series TPU architecture Neuron enablement Gaudi new hardware plugin GB200 and RTX PRO 6000 Connecting Models to the Hardware What’s New in vLLM 0.10.x: Next-Gen Hardware, Day-Zero Models, and V1 Unified Architecture Spyre Hardware plugin Interactive Demo: red.ht/vllm-interactive vLLM is available as GA in all the Red Hat AI platforms Hardware plugin for ATOM Hardware plugin with MACA Re-architecture of vLLM core components, including the scheduler, KV cache manager, worker, sampler, and API server. Multimodal in vLLM 0.10.1
scalable inference Enterprise GenAI inference platform Holistic approach to optimize and operationalize deployment and scaling of open-source LLMs GPU Instinct TPU Pod 2 Instinct Pod 1 GPU Pod 3 TPU Accelerators Inference Server Distributed Inference Framework Models AI Gateway/API Mgt Model as a Service
AI Model Validation Methodology Model Selection & Prioritization We prioritize models for validation based on a continuous analysis of the AI ecosystem, driven by several key inputs: ▸ Customer Demand ▸ OSS Market Leadership ▸ Strategic Portfolio Expansion Enterprise Packaging & Security Selected models are packaged as OCI artifacts and ModelCars. This enterprise-grade packaging is a critical step that enables: ▸ Security ▸ Lifecycle Management Performance & Accuracy Validation Each model is rigorously validated for performance across diverse hardware (NVIDIA, AMD, etc.) and various use cases. ▸ Performance Benchmarking ▸ Accuracy Evaluations Results Publication & Integration All generated data is aggregated and published to two key destinations to empower our teams and customers: ▸ Internal Sales Enablement UI (requires VPN) ▸ Customer-Facing Model Catalog (Coming to RHOAI in 3.0) A rigorous and transparent process to deliver trusted, enterprise-ready AI models
rigorous and transparent process to deliver trusted, enterprise-ready AI models Simplified Deployment ▸ Models packaged as ModelCars and OCI artifacts ▸ A wide variety of popular OSS and proprietary models available Enhanced Security ▸ Model Artifacts are scanned ▸ Model Artifacts have from a trusted source and are signed (future) Model Lifecycle Management ▸ Acts as a single source of truth for the entire model lifecycle. ▸ Enables robust versioning and simplifies promoting models from development to staging and production with GitOps Models in the Red Hat AI Container Registry will be surfaced in the RHOAI model catalog from 3.0
Eval Integration Enable benchmarking of AI models that operate across multiple input/output modalities — such as text, images, audio, etc. — within a unified, standardized framework. Expand Model Validation tooling (GuideLLM/J-Benchmark) to include accuracy evaluation OOTB. Multi-Model (Agentic) Benchmarking Benchmarking across chains of models, not just individual components. This enables real-world evaluation of multi-model pipelines — capturing end-to-end latency, quality, and cost. Model Signing / Security Sign and secure models that we validate to ensure to our customers they are scanned for vulnerabilities, safe to deploy and can be deployed with enterprise-grade security standards. Future work items include Provenance and AI SBOM scans (Coming 1H2026). 15 Benchmark-as-a- Service (BaaS) in RHOAI Provide users with a simple, self-service UI within RHOAI that allows them to benchmark models on demand. They can either upload their own models or select one from the Catalog. In a later phase, this tool will integrate model evaluation and expanded benchmarking use cases. Our roadmap for multi-modal and agentic benchmarking to ensure the highest quality models Q42025 1H2026 1H2026 1H2026
to maximize performance for specific model, use case and hardware. auto-tune-vLLM is a distributed hyperparameter optimization framework that uses Ray and Optuna to systematically find the best vLLM configurations. This moves beyond default settings to unlock significant performance gains tailored to your models and hardware. ▸ GitHub: openshift-psap/auto-tuning-vllm ▸ Core Technologies: vLLM, GuideLLM, Optuna, Ray Key Features ▸ 🚀 Distributed Optimization: Scale across multiple GPUs and nodes using Ray. ▸ 🎯 Flexible Backends: Run locally or on Ray clusters. ▸ 📊 Rich Benchmarking: Built-in GuideLLM support + custom benchmark providers. ▸ 🗄 Centralized Storage: PostgreSQL for trials, metrics, and logs. ▸ ⚙ Easy Configuration: YAML-based study and parameter configuration. ▸ 📈 Multi-Objective: Support for throughput vs latency trade-offs. ▸ 🔧 Extensible: Plug-in system for custom benchmarks. ▸ Immediate Focus: R&D - gather proof points ◦ Generate configs for key validated models to test i. gpt-oss 120B ii. Kimi K2 INT4 iii. Nemotron-9B-v2 ◦ Gather feedback on ux, instillation, + perf ▸ Long-Term Roadmap: [Subject to change] ◦ Run against all validated models moving forward ◦ Integrate into RHOAI via Workbench Images and RHAIIS via CLI. What’s New (Upstream Only) What’s Next
and speculative decoding to optimize your models for inference Advanced Quantization Expansion LLM Compressor ▸ What’s New ･ RHAIIS Image: LLM Compressor included in model-opt-toolkit image. ･ New Formats & Schemes: MxFp4 and W4A8 (with GPTQ/AWQ). non-uniform support coming soon. ･ Extended Quantization Support: KV Cache Quantization ･ Deeper vLLM Integration: Making transforms, QuIP, and SpinQuant first-class citizens. ▸ What’s Next ･ Smarter Quantization: Tools for sensitivity analysis (e.g., kurtosis) ･ Training & Scaling: QAT, QLora, Axolotl integration, RL pipelines, and Multi-GPU/Nodel support. Productizing Speculative Decoding Speculators ▸ What’s New ･ Making Speculators a core pathway in vLLM for large LLMs. ･ Supporting more speculators models ･ LLama 70B ･ Qwen 30B ･ Llama4 ･ Generically expanding verifier support for Eagle3 Models ･ Llama4 ･ DeepSeek v3 ▸ What’s Next ･ Productizing training support in speculators ･ Supporting online data generation training pathways Community, Docs, & Outreach LLM Compressor + Speculators ▸ What’s New ･ Major expansion of tutorials, general docs, and feature blogs. ･ Active community engagement: "First Good Issues," release recaps, and joining vLLM Office Hours ▸ What’s Next ･ Standardized/integrated benchmarking with GuideLLM. ･ Standardized/integrated accuracy evaluation with LM-Eval-Harness. Interactive demo: red.ht/llm-compress-interactive LLM Compressor will be GA on RHAIIS 3.2.3 + will be GA via Workbench Images and Runtime Images for AI Pipelines in RHOAI 2.25
scalable inference Enterprise GenAI inference platform Holistic approach to optimize and operationalize deployment and scaling of open-source LLMs GPU Instinct TPU Pod 2 Instinct Pod 1 GPU Pod 3 TPU Accelerators Inference Server Distributed Inference Framework Models AI Gateway/API Mgt Model as a Service
by Red Hat, Google, NVIDIA, AMD, Hugging Face, and many more • Kubernetes-Native Architecture for simple deployment and management of GenAI models • Optimized GenAI Inference to accelerate LLM’s and MoE • Intelligent Resource Utilization to reduce inference costs • High Performance and Scalability to meet demanding Service Level Objectives (SLOs). • Supported on Heterogeneous Hardware like NVIDIA and AMD GPUs (and many more to come in the future) 20 Maximize GPU utilization and deliver on your SLOs with distributed inference Request Inference gateway Validation Prompt logging Prefill Scheduler Decode KV cache GPUs Node Node GPU GPU GPU
scalable inference Enterprise GenAI inference platform Holistic approach to optimize and operationalize deployment and scaling of open-source LLMs GPU Instinct TPU Pod 2 Instinct Pod 1 GPU Pod 3 TPU Accelerators Inference Server Distributed Inference Framework Models AI Gateway/API Mgt Model as a Service
audience ▸ IT serves common models centrally ▪ Generative AI focus, applicable to any model ▪ Centralized pool of hardware ▪ Platform Engineering for AI ▸ Models available through the RHOAI console ▸ Developers consume models, build AI applications ▪ For end users (private assistants, etc) ▪ To improve products or services through AI ▸ Shared Resource business model keeps costs down Fast, flexible and scalable inference Model-as-a-Service in OpenShift AI Model as a service Developers End users AI Gateway Available Models AI Engineer GPUs utilization End user apps Development environments ITOps ITOps MaaS will be in Dev Preview on RHOAI 3.0 Empower developers and end-users with centralized, cost-effective access to AI models
3.0 RHOAI NEXT Q4 2025 H1 2026 Dev Preview Release ▸ MaaS Control Plane Foundation: Core infrastructure for centralized model serving around k8s-native gateways ▸ RBAC for Models: Role-based access control with fine-grained permissions ▸ Integrates models served through Kserve, which includes llm-d ▸ Apply runtime policies to ensure fair sharing of model access and prevent starvation. ▸ Chargeback and Showback: Per-model cost tracking, usage analytics, and chargeback automation ▸ Developer Self-Service: Enhanced "Available Assets" interface for model discovery and key management, ▸ Integrated AI Gateway based on Red Hat Connectivity Link (included with RHOAI for MaaS usage) Hidden behind a feature flag for admins who want to try MaaS GA Release ▸ Platform Admin UI: Complete governance interface with quota management, applying policies, and monitoring ▸ Advanced policies: Extend policy engine capabilities to enable enterprises to granularly manage their models’ runtime policy and ensure QoS. ▸ External models management: Serve and apply policies to external models through RHOAI MaaS. ▸ TBD: Partner integrations for AI Gateway based on customer demand.
(Kubernetes Event Driven Autoscaling) (Technical Preview) ▸ Pause models to release resources without the need to recreate deployments to restart ▸ Display of runtime versions in the RHOAI dashboard ▸ ML Server now certified and tested ▸ Serving from PVCs ▸ llm-d as a GenAI model Serving platform (Developer Preview) ▸ KEDA UI and GA release ▸ More fully supported OOTB runtimes for traditional AI ▸ Models as a Service (MaaS) What’s New What’s Next Enhanced model serving capabilities for both generative and traditional AI
InstructLab Connecting models to data 30 Training hub Data processing Generate high-quality data, with dynamic parameters, run-time visibility, and multilingual support Synthetic data Generation hub An algorithm-focused interface for common llm training, continual learning, and reinforcement learning techniques Simplifies document processing and parsing into AI-readable data for model customization and RAG applications Evaluations Simplifies the distributed execution of Evaluation jobs from popular eval frameworks or tasks A modular and extensible toolkit for rapid experimentation for connecting models to data
training hub Red Hat AI Python Index Validated Notebooks Workbench AI Pipelines & Components Experimentation flow Production flow pip install <sdg hub> git clone repo Build your own SDG flow :) Guided examples with AI Pipelines & Components Use docling, SDG, Training Hub packages directly from Red Hat AI Python Index and seamlessly move from experimentation to production with our integrated model customization workflows Reusable workflows (notebooks, pipeline components) provided for model customization for OOTB use!
custom workflows with collection of example notebooks Customize • Supported build of the docling package • Available on the Red Hat AI Python Index • Utilize supported packages to create bespoke pipelines and workbenches Automate and scale workflows • Guided implementations to build and run Kubeflow Pipelines for batch data processing • Scheduled longer-term runs for larger data sets What’s New with Data Processing Prepare your data for model alignment and RAG with our supported version of docling ★ Star the Repo ★ Start Contributing :) https://github.com/docling-project
RHOAI NEXT Q4 2025 H1 2026 ▸ The docling package will be TP & available to users through the Red Hat Python Index ▸ Example notebooks for structured information extraction, RAG, synthetic data generation ▸ Example notebooks and pipelines to use Vision Models ▸ Example implementation for orchestrating data processing for larger datasets ▸ The docling package will be GA and available to users through the Red Hat Python Index ▸ Optimized GPU Performance with new pipelines ▸ Examples for multi-lingual data processing Prepare your data for model alignment and RAG and expect expanded use cases for multi-lingual data processing
LLM-powered & traditional blocks ▸ A maintained collection of high-quality, validated SDG pipelines that keeps growing ▸ Compose & orchestrate custom flows - from simple transforms to complex multi-stage pipelines ▸ Replace ad hoc scripts with a reusable framework - consistency and repeatability built in ▸ Scale synthetic data generation - async native and monitoring out of the box ▸ Easily extend & customize - create your own blocks and plug into existing flows with minimal code Introducing Synthetic Data Generation Hub What? Why? ★ Star the Repo ★ Start Contributing :) Python Library for data generation algorithms focused on customization & flexibility
data generation for enterprise model customization RHOAI 3.0 RHOAI NEXT Q4 2025 H1 2026 ▸ SDG Hub Python Index will be TP and will be available to users through the Red Hat Python Index as a supported build ▸ LiteLLM Integration with 100+ Model providers ▸ Dynamic Configuration allows runtime model switching ▸ Full Transition to Chat Completions API ▸ Developer Experience Improvements - Pydantic validation for flows ▸ Rich Logging for SDG Observability ▸ Multilingual Data Generation Flows (Japanese language) ▸ Enhanced Knowledge Pipeline with results on Quality Benchmark ▸ Support for data generation with GPT-OSS ▸ Recommended/Default Teacher Models for Validated flows ▸ Functional Support for any teacher model ▸ Synthetic Data Generation for Agentic Fine Tuning (Agent Trajectory Data Generation) ▸ Synthetic Data Generation for Domain Specific Tool Calling (based on customer need) ▸ SDG for production RAG evaluation ▸ Reusable Data Science Pipeline Components
for common training algorithms Introduction to Training Hub The Training Hub offers a consistent and reliable interface for accessing the latest training algorithms. It's a semantic layer with a stable API that remains consistent even after an algorithm is adopted by broader upstream communities. This allows you to leverage new research early while maintaining a consistent experience. With the upcoming 3.0 release, the Training Hub will support a range of powerful training methods, including: • Supervised Fine-Tuning (SFT) from instructlab.training • Orthogonal SFT , a new continual training algorithm • Full compatibility with LAB Multiphase Pipeline • Full Compatibility with the latest powerful open source model - GPT OSS (for SFT and continual learning) ★ Star the Repo ★ Start Contributing :) training hub
enable continuous learning with our breakthrough Orthogonal Subspace Fine-Tuning Enterprise environments are dynamic - data, policies and user needs evolve daily Large language Model Biggest Challenge with Fine tuning Catastrophic Forgetting- new knowledge overrides old, degrading performance 🔄 Replay Buffer 🔌 LORA Adapters + Model Merging Lack of replay buffers for all models Freeze most params -> limited gains, params pile over time Existing solutions fall short for enterprise scale continual learning Our Breakthrough: Orthogonal Subspace Fine-Tuning (OSFT) 🔄 A single continuously learning model Learns new tasks seamlessly Preserves prior knowledge
training algorithms for enterprise model customization RHOAI 3.0 RHOAI NEXT Q4 2025 H1 2026 ▸ Training Hub Python Index will be TP and will be available to users through the Red Hat Python Index ▸ Training Hub supports SFT, OSFT (continual learning) algorithms ▸ Training Hub contains example cookbooks with Llama, and GPT-OSS models ▸ Integration of Training Hub algorithms with Trainer v2 and Kubeflow SDK ▸ Guided Examples for for building KFtrainer packaged example from Training Hub notebook for LAB SFT and Continuous Learning. ▸ New Training Runtime Container Image ▸ Support for Training on NVIDIA ▸ Dynamic Hardware Profiles ▸ Hardware Discovery ▸ Support for new RL algorithms for Agentic Fine Tuning ▸ Integration of continual learning with Hugging Face PEFT ▸ Training support on other accelerators (AMD)
for AI application development and deployment RHOAI 3.0 RHOAI NEXT Q4 2025 H1 2026 ▸ RAGAS Framework : integration of RAGAS evaluation framework as a local and remote (KFP) evaluation provider in Llama Stack. ▸ Expanded list of validated Tier 1 evaluation tasks from LM Evaluation Harness in TrustyAI ▸ MLFlow 3 based distributed evaluation stack with native granular experiment tracking. ▸ Evals Collections: Industry-specific evaluation tasks to assess model suitability for a particular vertical and use cases ▸ Expand safety evaluation through integrated benchmarks and red teaming assessments ▸ BYO Eval Framework: Onboard custom evaluation tasks and frameworks to run on the platform's distributed evaluation capabilities.
and QLoRA Customize a base model for specific tasks or private data, using a range of approaches—from full fine-tuning to parameter-efficient methods—to balance performance and efficiency. Prompting Prompt tuning and engineering Design and engineer the prompts to enhance GenAI model responses and achieve more specific and accurate outcomes. Connecting models to data Build customized AI solutions that address domain specific business cases RAG Retrieval Augmented Generation Enhance Gen AI model generated text by retrieving relevant information from external sources, improving accuracy and depth of model's responses. Com ing soon Enhanced Enhanced
AI Playground Custom RAG Application Response Generation endpoint /v1/responses Embeddings Generation endpoint /v1/embeddings Vector Databases endpoint /v1/vector_stores Files object endpoint /v1/files Object storage Vector Database (local providers) Milvus, etc. Vector Database (remote providers) PGVector*, etc. Cloud LLM Provider Red Hat Inference Service Hybrid Search (semantic+keyword) embedding-models Validated Models Retrieval Augmented Generation (RAG) with Llama Stack Evaluations • LM Evaluation Harness • RAGAS Easily get started with RAG using our OpenAI-compatible APIs with self-hosted models or vecordDBs or through cloud based model providers
Q4 2025 H1 2026 ▸ Enhanced compatibility with common frameworks by using new OpenAI-compatible APIs for RAG capabilities (/v1/responses, /v1/files, /v1/vector_stores) ▸ Vector Databases: ∙ Llama Stack with a Milvus local provider ∙ Llama Stack with a Milvus and PGVector* remote provider ▸ Enhanced Llama Stack retrieval with server-side Hybrid Search (combining semantic and keywords search) support for vector databases. ▸ Default to granite-embedding-125m-english as the embedding model while supporting BYO embedding model. ▸ Integrated in the new AI Playground (details will be covered later in the presentation) ▸ Extended server-side reranking of retrieved results ▸ Expanded support for remote vector databases through partner ecosystem integrations. ▸ Extend Llama Stack remote provider support for open source vector databases such as Qdrant, SQLite, Chroma and others. ▸ Extended popular frameworks with native Llama Stack provider integration, including: ∙ LangChain & LangGraph ∙ LlamaIndex ∙ CrewAI ▸ Prompt template caching in Llama Stack ▸ Memory for RAG via OpenAI compatible /v1/conversations API for search and inference. * Dev Preview remote providers via custom ConfigMap https://github.com/opendatahub-io/llama-stack-k8s-operator/blob/odh/config/samples/example-with-configmap.yaml
v2: major technology upgrade we inherit from upstream, enabling better Kueue integration. Overall, KFTv2 will be deeply integrated with the Training Hub and Experiment Tracking experience. ▸ Checkpointing: manually or periodically, saving the state of a TrainingJob, so that it could be resumed later on ･ also enhances preemption experience, and ･ give disaster recovery to a degree ▸ MPI TrainingJob support with KFTv2 Highlights RHOAI NEXT H1 2026
between your models and data for consistency, velocity, and governance A centralized place for registering, managing and serving features used in machine learning models ▸ Feature catalog ▪ Enhance discoverability & collaboration between teams ▪ Govern access to data ▸ Simplify feature reuse across models ▸ Provide online store for fast, low-latency access to fresh features
3.0 RHOAI NEXT Q4 2025 H1 2026 GA Release ▸ RHOAI Dashboard Integration: Native Feast Feature Store UI embedded in OpenShift AI console ▸ Workbench Integration: Seamless notebook integration with pre-installed Feast modules and remote access ▸ Enterprise RBAC: Integrated governance and access control with OpenShift identity management Enhanced Integration throughout RHOAI ▸ MLOps Integration: Deep integration with Kubeflow Pipelines and model lifecycle ▸ GenAI Enhancements: Integration w/ LlamaStack, vector DBs and additional improvements to provide value and standardization for GenAI use-cases. ▸ Ray Integration: Distributed feature engineering with Ray workflows and transformations ▸ Extended Lineage: Cross-RHOAI component tracking (training jobs, model endpoints, notebooks) ▸ Advanced Monitoring: Real-time feature monitoring, drift detection, and quality analytics ▸ BYO Spark Support: Connect to external Spark clusters for feature transformations and materialization
OpenAI-Compatible Endpoints (optimized for vLLM). ▸ Boosts accuracy and reduces errors with SOTA methods, including planning, voting, and decoding strategies ▸ Integrates with reward-hub for feedback-driven improvement. ▸ Delivers better results from existing models — without retraining ▸ Available as Dev Preview in 3.0 through Red Hat PyPI ▸ Open source repo can be found at red.ht/its_hub What’s next ▸ Deeper integration with Red Hat AI products ▸ ITS powered data generation via sdg_hub ▸ Additional ITS algorithms ▸ Broader support for popular agent frameworks A Python library for inference-time scaling to improve model accuracy without retraining. Introduction to its-hub User: Hi! I recently placed an order and I'll be traveling when it arrives. I'd like to cancel it and get a full refund. Agent: I can help you with that. First, I need to authenticate your identity. Could you provide your name and zip code?" User: Sure! My name is Olivia Ito, and my zip code is 80218. Agent: Thank you, I've located your account. Could you provide the order ID you wish to cancel? Tool Calls: find_user_id_by_name_zip(name="Olivia Ito", zip="80218") get_user_details(user_id="olivia_ito_3591") User: The order ID is #W5442520. Agent: Great! Order #W5442520 is still pending and can be cancelled. Total: $663.85 with 3 items. Would you like to proceed? User: Yes, please proceed with the cancellation. The reason is I'll be traveling. Agent: Order #W5442520 has been successfully cancelled! Tool Call without ITS: cancel_pending_order(order_number="#W5442520", reason="no longer needed") Tool Call with ITS - cancel_pending_order(order_id="#W5442520", reason="no longer needed") ▸ Example from Tau-Bench, which tests agents with real users, APIs, and tasks ▸ Without ITS: the agent makes a wrong tool call (order_number) ▸ With ITS: the agent selects the correct tool (order_id) ▸ Agents are only useful if they’re correct — inference-time scaling ensures accurate, consistent results
accelerate the development and deployment of AI agentic workflows. ▸ Allows running and managing agents as microservices. ▸ Simplifies production deployment by managing LLM serving and scaling. ▸ Offers the foundation to build and manage agents with Llama Stack, and standardized communication protocols (MCP). ▸ Provides the flexibility to integrate preferred tools/frameworks (e.g., LangChain/ Langgraph). What’s next - Accelerate Agentic AI innovation
Highlights! RHOAI 3.0 RHOAI NEXT Q4 2025 H1 2026 ▸ Enable Air-Gapped/Disconnected for Llama Stack (Operator + Distro) ▸ Enable FIPS Compatibility for Llama Stack ▸ Documented Llama Stack Support Matrix and Provider/Component Version Compatibility ▸ Documentation of LLama Stack performance/capacity Limits ▸ Support of Llama Stack on Milti-arch (e.g., ARM) ▸ New Inference providers (IBM Watson.x , Hosted OpenAI, AWS Bedrock, Microsoft Azure OpenAI, VertexAI) ▸ Validating LlamaStack on Single Node OpenShift with RHOAI Support and Optimization ▸ Deployment Best Practices Guide for the Llama Stack Operator ▸ Day-2 Config and Deploy-Time Modularity for Llama Stack Providers ▸ Streamline Custom Llama Stack Deployments on OpenShift AI with BYOD ▸ E2E Observability for Llama Stack in the product ▸ Inclusion of more providers RAG/Agentic/Safety/Evals Disclaimer: This is a tentative roadmap and is subject to change.
3.0 RHOAI NEXT Q4 2025 H1 2026 ▸ Ready to use/deployed example MCP Servers (via Config-Map) in the RHOAI console (AI Available Assets) ▸ Use the AI Playground to compare, evaluate, and test w/ Available Models/MCP Servers ∙ Support existing AI applications using the OpenAI’s Agentic Responses API ∙ Support for connecting to remote MCP servers via SSE or HTTP Streaming with LLama Stack ▸ Enable approvals and HIL for tool use in Llama Stack ▸ Secure Onboarding & Discovery for MCP servers ▸ MCP user Sessions Authentication and Isolation ∙ Access Control & Lifecycle Management for MCP Servers with OpenShift AI ∙ Integration, Validation, and Cataloging of MCP Servers (including ISVs) in Red Hat AI Assets/Ecosystem Catalog ∙ MCP Gateway for federating/Securing tools & Introduction and Documentation for Deployment Best Practices ∙ Elicitation & Connectors Support in Responses API with Llama Stack ▸ Tracing/Observability for Agentic AI Applications ▸ Compatibility/testing against popular Agent frameworks: ∙ LangChain & LangGraph ∙ LlamaIndex ∙ CrewAI Disclaimer: This is a tentative roadmap and is subject to change.
Subject to change. Generative AI in Openshift AI 3.0 AI Hub A unified interface for Platform Engineers to oversee the control, lifecycle, and governance of AI assets, starting with RHAI validated models, and in the future MCPs and Agents. Gen AI Studio An integrated environment where AI Engineers can discover, consume, and experiment with GenAI assets. A unified, easy-to-use experience for Platform and AI Engineers
Assets: MCP, agents, prompts ▸ Components: ･ Catalog ･ Registry ･ Deployments AI Hub Future Feature - these are mock ups and early drafts. Subject to change. A single, streamlined space to operationalize AI models and other AI assets at scale
– Clear steps to design & configure MCP Servers • Try Before You Buy – Evaluate MCP Servers with loaded models directly in the Playground • Available AI Assets Listing – Quickly see and launch what’s ready to use • Llama Stack Tool Calling – Seamless tool invocation through MCP from your models Why it Matters • Instant Experimentation – Start building today • Model + Tool Synergy – Bring real-world capabilities to your AI workflows • Foundation for RHOAI 3.next – Governed today, certified assets tomorrow MCP in RHOAI 3.0 – Hands-On AI Tool Power Introducing Model Context Protocol (MCP) Connect your models to real-world tools and bring powerful capabilities to your AI Agents
Studio Highlights RHOAI 3.0 RHOAI NEXT Q4 2025 H1 2026 ▸ AI Asset Endpoints ∙ View available models allowing addition to playground ∙ MCP Surfacing view config map ingestion ▸ AI Playground ∙ Initial release of AI Playground for experimentation with Available AI Assets ∙ Test MCP Servers against available models ∙ Multi-document upload for vector knowledge source testing ∙ View the code for the current configuration of model and mcp servers chosen ▸ Playground Expansion ∙ Richer feature set with tool selection, model comparison, and advanced control ∙ Guardrail surfacing and prompt management for safer/standard experimentation ∙ Persistent sessions with chat history, tracing, and orchestration support ▸ MCP Enhancements ∙ Registry, Catalog, and Gateway to unify discovery and usage of MCP servers ∙ Enterprise-grade security with OAuth and RBAC hardening ∙ Validation and certification guidance to ensure trusted integrations ▸ Simplified document ingestion UI for knowledge source creation ▸ Integrated evaluations for continuous testing and improvement ▸ Secure handling of user secrets
ability to discover Red Hat models, Red Hat AI validated models, and popular community and custom 3rd party models to be easily compressed, tuned, and deployed from within OpenShift AI. Interactive Demo: https://interact.redhat.com/share/zjNrWF0UIsbQoAds9xZG Catalog will be GA in OpenShift AI in the 3.0 release
centralized repository within the OpenShift AI platform designed to manage the lifecycle of machine learning models via the Registry. Registry will be GA in OpenShift AI in the 3.0 release
Deployments page to facilitates the deployment of machine learning models by making them available as services accessible via API, enabling model inferencing. Deployments will be GA in OpenShift AI in the 3.0 release
Registry ▸ Model Registry GA in 2.25 ▸ Model Catalog GA in 2.25 ･ Red Hat shipped models ･ Red Hat validated models ･ Model performance insights ▸ Store models in OCI compliant storage ▸ Easily create model cars from registered models ▸ Sign models and model cars in model registry ▸ Populate Model Catalog with custom models ▸ Model Catalog connection to Hugging Face What’s New What’s Next
pipelines for GenAI What’s new? ▸ BYO Argo Workflows ▸ GitOps support ▸ Notebook components What’s next ? ▸ AI Pipelines for model customization. ･ Reusable components. ▸ Enhanced experiment tracking. ･ MLFLow integration ▸ AI Pipelines with better user experience ･ Better OOTB experience ･ PVC for object storage ･ Support for more DB. AI Researcher Data Scientist Notebook to Pipeline seamless experience OOTB Experiment tracking Local and remote execution MLFlow Integration Production ready pipelines Hardware profiles support GitOps Support Observability AI Engineer
integration What’s next? ▸ Experiment tracking for more than just pipelines ▸ Integration of MLFlow on RH AI for inner loop experiment tracking ▸ More GenAI visualizations and metrics ▸ AIP and MLFlow integration
have a true hybrid platform, we are enhancing our release process to support multiple architectures. Starting with IBM Z and ARM. IBM Power shortly after. 69 Access to cutting-edge open source innovations to keep up with a fast moving market. Increasing flexibility and choice with an open source approach Flexibility Choice Access to an open ecosystem of communities, technology providers, ISVs and customers. The value
RHOAI 3.0 RHOAI NEXT Q4 2025 H1 2026 ▸ Guardrails Orchestrator as a Llama Stack Safety, Guardrails provider [TP] ▸ TrustyAI service APIs moving to stable K8s API ▸ Updated Documentation for Safety/Evals Trusty by Function/Scope ▸ Guardrails Orchestrator as a Llama Stack Safety, Guardrails provider [GA] ▸ Exploring the Integration of NVIDIA Nemo Guardrails ▸ Provide blueprints for building guardrails in Agentic systems ▸ Deprecation of TrustyAI service alpha APIs
observability to all RHAI components throughout 2026! RHOAI 3.0 RHOAI NEXT Q4 2025 H1 2026 ▸ Platform Metrics Foundation: Centralized Observability infrastructure available through RHOAI Operator. ▸ Zero-Config GPU Monitoring: Auto-deployed NVIDIA DCGM on all GPU nodes with immediate metric availability ▸ Foundation Reference Dashboards: Reference Grafana dashboards for Data Scientists and Cluster Administrators running LLMs ▸ AI Performance Collection: KServe/vLLM metrics integration for TTFT, throughput tracking, token counts and more. ▸ Embedded Console Integration: Native dashboard experiences within RHOAI console. ▸ Workload Monitoring: Enhanced GPU utilization with Kueue integration and workload queuing dashboards ▸ Advanced AI Monitoring: Cross-component tracing for RAG, agents, and multi-LLM workflows ▸ llm-d Deep Observability: Production-ready dashboards with advanced LLM performance analytics
▸ Intense work in Kubeflow Notebooks community towards a re-architectured notebooks 2.0 solution. ▸ Admins will have centralized control through WorkspaceKind templates, which act as a single source of truth for configurations. ▸ The new architecture separates the individual user environments (Workspaces) from the admin-managed templates (WorkspaceKind), allowing for hundreds of workspace types and dynamic updates without recreating the environment. ▸ The new version will enable editing running workspace and provide pause/unpause capabilities similar to K8S-like managements. ▸ The new architecture is designed to handle thousands of use workloads and offers improved GPU management and resource tracking. ▸ It resolves issues such as limited environment flexibility and poor error visibility for AI/ML practitioners. Coming H1 2026
RHOAI 3.0 RHOAI NEXT Q4 2025 H1 2026 ▸ Jupyter Extensions: extensions to assist generation, debugging, and explanation directly in notebooks ▸ RStudio Image GA: Full production support for R-based data science workflows ▸ Enhanced Workbench Images: Improved container environments with optimized libraries and dependencies ▸ Intense work in Kubeflow Notebooks community towards a re-architectured notebooks 2.0 solution. ▸ Notebooks 2.0: Next-generation notebook experience with enhanced platform integration, admin management and template management. ▸ On-the-fly Reconfiguration: Modify GPU allocation and resources without restarting workbench sessions ▸ Modern Kubeflow Pipeline authoring based on Kubeflow Kale.
RHOAI 3.0 RHOAI NEXT Q4 2025 H1 2026 ▸ Ray and Codeflare updates: ∙ Architectural Simplification - Complete CodeFlare Operator removal for cleaner, more maintainable platform ∙ CodeFlare SDK providing a more aligned user experience by focusing on RayJob, while providing full access to RayClusters ▸ Accelerator Slicing: for all MIG-enabled devices, via the NVIDIA operator ▸ Kueue: supporting more workloads: ∙ Ray Training Jobs ∙ Training Operator-based Jobs ∙ Inferences Services and Workbenches ▸ Kubeflow Trainer v2 and Red Hat Build of Kueue 1.2: ∙ major technology updates for workload scheduling ∙ converged API for Training Jobs ▸ GPU and Workload specific Observability: based on RHOAI observability and accompanied by data scientists- and admin-centric views ▸ Model Checkpointing: to support better workload management, like Job-preemption and resume, to increase GPU utilization ▸ Dynamic Resource Allocation (DRA): focus on this technology for (not only) GPU partitioning, long lead-time, as a lot of OCP work is to be done!
to end experience in OpenShift AI Dashboard Evaluate Test model for performance and accuracy Deploy to stage Configure model serving Discover Explore models and assets Paula AI Platform Engineer Erica AI Expert Compare & promote Compare pipeline runs for variations on customization Customize Train the model for better performance / accuracy. Configure AI Playground Configure LLS, register models and associate mcp servers Deploy Model aaS Deploy and configure model-as-a-service Setup Enable the platform Paula AI Platform Engineer Deploy Package agent and deploy Experiment w/agent configs Prototype and evaluate models, tools, & sources Discover Explore models and assess Alex AI Engineer Paula AI Platform Engineer Adi Admin Setup Enable monitoring Monitor & Observe Continuously track utilization, performance and data drift Paula AI Platform Engineer Erica AI Expert Adi Admin Flow 1: Discover & Evaluate Flow 2: Customize Flow 3: Provide Models & MCP Servers Flow 4: Experiment with Agent Config and Deploy to Production Flow 5: Monitor & Observe Setup Enable the platform Flow 0: Set up the platform Flow 6: Integrate Features with a Workbench Explore Explore feature store for needed features Integrate & Utilize Integrate workbench to utilize features Erica AI Expert
∙ Catalog with Validated Models ∙ Registry ∙ Deployments ▸ Model Serving: ∙ New step by step wizard ∙ vLLM with distributed (llm-d) (GA) ▸ Hardware Profiles (GA) ▸ Kueue managed Hardware Profiles workloads (TP) ▸ Gen AI Studio: ∙ AI Asset Endpoints (TP) ∙ AI Playground (TP) ▸ Feature Store (GA) What’s New and Next in Openshift AI Dashboard Dashboard Capabilities and Experiences H1 2026 ▸ AI Hub: ∙ Model selection guidance with Validated model performance insights ▸ Model Serving ∙ Serve model as MaaS in wizard ▸ Gen AI Studio: ∙ MaaS in AI Asset Endpoints ∙ AI Playground enhancements (ie. model comparison, vllm metrics) ▸ Kueue: monitoring for all workloads (TP) ▸ MLFlow Integration (TP) ▸ AI Hub: additional AI Assets ∙ MCP Catalog & Registry ∙ Agent Catalog & Registry ▸ Prompt management ▸ RBAC: enhanced granularity on controls ▸ Observability: ∙ Platform metrics (ie. utilization) ∙ enhanced performance per deployment ▸ Evaluations (TrustyAI & lm-eval) ▸ Gen AI Studio: ∙ Knowledge source creation (Docling) for RAG ∙ MaaS quota visualization ▸ MaaS Admin Settings ▸ Enhance model customization experience ∙ Model Optimization UI (llm-compressor) ∙ Model Customization UI Red Hat Confidential. This is an uncommitted roadmap and Red Hat reserves the right to change it
RHAIIS 3.2.0 July 2025 RHOAI 3.0 RHAIIS 3.2.2 September 2025 RHOAI 2.25 Support for NVIDIA RTX PRO 6000 Blackwell Server Edition, GB200, and B200 across the Red Hat AI portfolio. Spyre AIU Model Serving on IBM Spyre AIU across x86, IBM Z, and Power architectures. What’s new - Red Hat AI Accelerator support RHAIIS 3.2.1 (DP) August 2025 TPU Preview support for Google Cloud TPU with Red Hat AI Inference Server. Product Manager: Erwan Gallen DP: Developer Preview - TP: Tech Preview - GA: General availability
Hat AI Accelerator support Product Manager: Erwan Gallen DP: Developer Preview - TP: Tech Preview - GA: General availability GB300 NVL72 NVIDIA Grace Blackwell Superchip Arm support. Large Scale AI platform. Red Hat AI Q1 2026 MI350X, MI355X Support for AMD Instinct MI350X and MI355X GPUs. Red Hat AI Q1 2026 Inferentia, Trainium Inferentia, Trainium AI accelerators ASICs support Red Hat AI H1 2026 ATOM ATOM NPU accelerator support for certified inference. Red Hat AI H1 2026
2025 1H 2026 Inference ▸ Distributed inference - llm-d GA ▸ Additional validated models ▸ LLM Compressor images GA ▸ MaaS (Dev preview) Model development & Alignment ▸ Modular & extensible solns for data ingestion, SDG, training & evaluation ▸ RAG - OpenAI APIs, local Milvus vector DB ▸ RAGAS evaluation framework ▸ Docling for document processing (Dev Preview) ▸ Feature Store GA Agentic ▸ Llama Stack APIs ▸ AI Hub UI & gen AI studio ▸ MCP support & MCP Server access in gen AI studio AI platform ▸ Model registry & catalog GA ▸ Platform metrics foundation - centralized observability infrastructure ▸ Expand platform support - ARM / Grace Hopper, IBM Z (GA), Power (TP) ▸ Hardware Profiles GA Inference ▸ Llm-d enhancements - eg. autoscaling, multi modal model support ▸ KEDA GA including UI ▸ OOTB ML Server runtime for expanded predictive model support ▸ MaaS GA - platform admin UI, external models mgmt Model development & alignment ▸ Enhanced experiment tracking (MLflow) ▸ Docling support (GA) ▸ Prompt design capabilities ▸ Kubeflow Trainer v2 and checkpointing capabilities ▸ Enhanced model customization experience Agentic ▸ MCP Server catalog & registry ▸ Agent catalog & registry ▸ Enhanced AI Playground - eg. model comparison, guardrails ▸ Knowledge source creation (Gen AI Studio UI) ▸ Inclusion of more Llama Stack providers across modules AI platform ▸ Enhanced native observability - model/agent monitoring, GPUaaS ▸ Integration of NVIDIA Nemo Guardrails ▸ Workbenches 2.0 - next-gen workbench experience ▸ RH Trusted Software Supply Chain integration ▸ GPU partitioning GA (based on DRA) Red Hat Confidential. This is an uncommitted roadmap and Red Hat reserves the right to change it
1 month life cycle EUS: 12 month cadence, 18 month life cycle Stable: 3 Month Cadence, 7 months life cycle 3.0 Lifecycle Updates Lifecycle Updates Red Hat AI products life cycle Fast Fast Fast EUS Support Full Support 3.2 3.3 3.4 3.5 3.9
Deep-dive’s from Red Hat and IBM experts on: ･ ⚡ Fast & efficient inference ･ 🎯 Model customization ･ 🤖 Agentic AI ･ 🌐 Scaling AI over hybrid cloud ▸ Hands-on insights to accelerate AI adoption ▸ Practical guidance to scale AI securely & efficiently 84 Your Path to Enterprise-Ready AI Join our Next [Virtual] Red Hat AI Event red.ht/ai-day-of-learning Sign up →
AI blog red.ht/ai-blog Follow us on X and YouTube x.com/RedHat_AI youtube.com/@RedHat Learn more about Red Hat AI Arcade demos • red.ht/InferenceBenchmark • red.ht/intro-inference • red.ht/model-optimization • red.ht/model-customization • red.ht/llm • red.ht/MaaS
Hat AI Inference Server red.ht/ai-inference-server-trial Use the 60-day trial of Red Hat Enterprise Linux AI red.ht/RHELAI-trial Use the 60-day trial of Red Hat OpenShift AI red.ht/RHOAI-trial Learn more about Red Hat AI
provider of enterprise open source software solutions. Award-winning support, training, and consulting services make Red Hat a trusted adviser to the Fortune 500. Thank you
chriscoyier
axbom
lcolladotor
charlesmeaden
greggifford
michaelherold
codingconduct
aleyda
seathinner
skipperchong
irinanazarova
