Title Speaker 18:00–18:20 Opening Remarks Red Hat Brian Stevens 18:20–18:45 Intro to vLLM Red Hat Michael Goin 18:45–19:15 Deploying PLaMo2 with vLLM: A Practical Guide Preferred Networks Shinichi Hemmi 19:15–19:35 LLM Compressor Red Hat Michael Goin 19:35–19:55 llm-d Red Hat Huamin Chen 19:55–20:00 Q&A 20:00–21:00 Networking & Lightning Talks (LT)
Hat we believe the future of AI is Open – source, models, infrastructure, serving. We are on a mission to bring the power of open-source LLMs and vLLM to every enterprise on the planet. Vision Mission
power of open There has been an explosion of capability in open models over the last 2 years Llama No OSS models Zephyr Llama 2 Mistral, Granite 2 DeepSeek-R1 Mixtral, Phi-2 Jan 2023 RedPajama, MPT, Falcon Mar 2023 May 2023 July 2023 Sept 2023 Nov 2023 Jan 2024 July 2024 Sept 2024 Nov 2024 Gemma2, Nemotron DBRX, Granite 3 Qwen2-VL Phi-3, Arctic DBRX, Phi-3 Llama 3, Qwen2 Mar 2024 May 2024 Jan 2025
power of open Open models are deployment targets today – and the trend is not slowing down • 650M downloads in 2024 • 85,000 Llama derivative models • 1B, 3B, 8B, 70B, 405B variants • Multilingual, Multimodal, Mobile • First reasoning model on par in quality with OpenAI o1 • 1-70B parameter distilled versions • Global market pandemonium? Llama R1 Headlines Models are commoditizing → many options for diverse enterprise needs
of open weight models and serving stack Open models play an important role in the enterprise AI landscape • Cost ◦ Self managed infrastructure ◦ 1B-405B size - match task difficulty to model • Customization ◦ Improve accuracy and costs with task specific tuning • Control ◦ Model lifecycle (no changes to the model in place) ◦ Resources (no rate limits / API downtime) • Security ◦ Complete data privacy (no 3rd party APIs)
Hat: Leaders in OSS GenAI inference Expertise across high performance inference and SOTA model optimizations • HPC engineering team dedicated to vLLM, 7 core vLLM committers on staff • Work on key subsystems, with a particular emphasis on fast model execution • ML engineering team builds vLLM’s optimization library llm-compressor • ML research team create pre-optimized models for deployment with vLLM Core Developers of vLLM Red Hat Community Contribution Commits By Organization
Hat: Leaders in OSS GenAI inference Expertise across high performance inference and SOTA model optimizations Optimized Model Hub LLM Compression Tools Llama Qwen Mistral DeepSeek Gemma Phi
TPU Gaudi Instinct GPU Llama Qwen DeepSeek Gemma Mistral Molmo Phi Nemotron Granite Spyre Edge Private Cloud Physical Virtual Public Cloud vLLM: The De Facto Open GenAI Inference Platform vLLM has emerged as the Linux of GenAI Inference
distributed inference at scale • Prefill/decode disaggregation • KV Cache distribution, offloading and storage hierarchy • AI-aware router • Operational telemetry for production • Kubernetes-based • NIXL inference transfer library Core Features: Open source release and announcement @May Red Hat Summit together with 10 founding members. 14 Why? Distributed architecture needed for maximum efficiency and meeting varying SLOs.
Size > 1 & Data Center Hardwares ･ Not the same workload as on-device inference for a single user ▸ How do you? ･ Efficiently schedule requests into the next forward pass? ･ Manage KV cache context and runtime memory footprint? ･ Make sure GPUs go brrrr 17 What Problem is vLLM Solving? Production Inference Serving
LLM is a function to predict the next token in a sequence ･ P(X_n | X_0 … X_n-1) ▸ To generate text, we “chain together” passes through the model ･ → A single request requires multiple passes through the model ･ → A single generation request can last multiple seconds ▸ Key Challenge: How to handle multiple concurrent requests 18 Why Is This A Hard Problem?
Original Innovation Paged Attention + Continuous Batching Alan Turing is a computer scientist and mathem atician renowned Logical KV blocks Request A Block Table computer scientist and mathe- matician Artificial Intelli- gence is the renowned future of tech- nology Alan Turing is a Physical KV blocks Artificial Intelli- gence is the future of tech- nology Logical KV blocks Request B Block Table
Year Journey Of vLLM vLLM has rapidly evolved from a research project to the open source default ▸ Pervasive → 100k daily installs in 2025; 50k GitHub stars ▸ Explosive Growth → 10x usage increase in 2024 ▸ Vibrant Community → 1000+ contributors
vLLM For Performance? vLLM implements the key optimizations for fast inference Inference Optimizations To make your models faster Distributed Inference To deploy large models efficiently
between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. User: Hello! Request A Request B 30 Automatic Prefix Caching Re-use KV cache blocks across requests! A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. User: How are you?
API (1): LLM class from vllm import LLM # Example prompts. prompts = ["Hello, my name is", "The capital of France is"] # Create an LLM with HF model name. llm = LLM(model="meta-llama/Meta-Llama-3.1-8B") # Generate texts from the prompts. outputs = llm.generate(prompts) # also llm.chat(messages)] A Python interface for offline batched inference
101 ▸ LLMs are a series of matrix multiplications of learned “parameters” ▸ Each “parameter” is a value represented by a series of some number of bits ▸ The more bits used to represent a parameter the more detail we can express: ･ Dynamic range (min-max value) ･ Precision (values close to zero)
101 Quantization aims to reduces the precision of a model’s weights (+ possibly activations) from high precision (e.g. BF16 training) to low precision formats (e.g. INT8 / FP8) without dropping model quality. Quantization Targets
Quantization (W8A16, W4A16, WNA16) ▸ Reduce GPU RAM requirements by squeezing down the parameters into lower bit precisions ▸ Reduce data movement at the expense of some extra compute to upconvert for each forward pass ▸ This is likely what you already know: GPTQ, AWQ, bitsandbytes, GGUF, etc
Quantization (W8A8, W4A8, W4A4) ▸ Weight-only quantization is not sufficient for speedups once under load (long prefills or batching). ▸ Quantizing weights and activations means we can finally use low precision tensor cores!
Compression with Fine-grained Quantization Not all quantization is the same, quality is important! Pass@1 score and standard deviation for quantized models on the popular reasoning benchmarks
Started with Quantization in vLLM Find pre-optimized models at hf.co/RedHatAI Pre-Optimized Model Hub LLM Compressor Llama Qwen Mistral DeepSeek Gemma Phi → red.ht/optimized-models → red.ht/llm-compressor
a Model with LLM Compressor Picking the right scheme and model Picking a Compression Scheme ▸ W4A16 (low batch size, memory bound) ▸ INT8 or FP8 (high batch size, compute bound) ▸ KV Cache Quantization (large context lengths) ▸ 2of4 Sparsity (high batch size, smaller model size)
LLM Compressor Ecosystem LLM Compressor and compressed tensors Compressed Tensors Integration with Transformers Research-backed Quantized Models Ready to Deploy Integrations with SFT Frameworks like Axolotl Adopted by foundation model labs!
end-to-end latency speedup for Llama 3.1 70b model ▸ 50% less GPUs ･ Dense (left) running on two A100 80GB ･ Compressed (right) running on one A100 80GB ▸ 99.14% accuracy recovery 58 Demo: Compressed Model Inference Acceleration in vLLM
Problem: LLM Inference is Different Traditional HTTP LLM Inference Short-lived, uniform requests Expensive requests with high variance Uniform latency requirements Diverse QoS needs (ms to hours) Simple round-robin works Cache locality matters Each replica equal Disaggregation opportunities
Scheduler: Available Scorers Scorer Purpose Benefit Session-aware Prefers pods from same user session Conversation continuity Prefix-aware Routes based on prompt prefix matching Cache hit optimization KV Cache-aware Optimizes for KV cache reuse Memory efficiency Load-aware Avoids overloaded pods Even distribution
and decode deployments ▸ Inference pool and model defined by Gateway API Inference Extension (GIE) ▸ Endpoint picker (EPP) deployment and service ▸ Relevant RBAC permissions 69 Component Deep Dive: Model Service
IGW and vLLM in llm-d collaborated on prefix-cache aware routing, building on IGW’s KV cache-aware load balancing. Evaluated on 2×8×H100 nodes using LMbenchmark with long-input/short-output workloads. Focus: stress KV cache reuse and test routing decision quality.
Contributors: ･ CoreWeave ･ Google Cloud ･ IBM Research ･ NVIDIA ･ Red Hat ▸ Communication: ･ 💬 Slack Workspace ･ 💭 GitHub Discussions ･ 📧 Google Group ▸ Open Development: ･ Apache 2.0 ･ Upstream-first ･ Component-based ownership 79 Community & Governance
key vLLM features Comment and review PRs that are interesting to you. Join the discussion on RFCs. Check out “good first issue” tags. Join vLLM Developer Slack Ask questions and engage with us via Slack. Join here. Engage with vLLM Office Hours Red Hat hosts bi-weekly vLLM Office Hours every other Thursday. We share project updates, dig into exciting topics, answer questions, and more. All sessions are recorded. You can engage with the slides and recording here. We are exploring ways to bring this to your region at appropriate times at the end of the year. Get involved with the vLLM Community 84
jpishikawa
recruitengineers
kuroneko13
yuyaumetsu
y_sera15
wolfgangwagner
kamadamakoto
cygames
cocacola917
sgrsn
shibuiwilliam
matayuuu
sergeychernyshev
geshan
reverentgeek
baasie
mclloyd
yeseniaperezcruz
rkendrick25
addyosmani
kneath
codingconduct
sabderemane
marketingsoph
