[Track 2] Furiosa-LLM: High-performance LLM Serving for RNGD

RNGD를 위해 설계한 고성능 엔터프라이즈 서빙 엔진 Furiosa-LLM을 소개합니다. 프로젝트의 동기, 최신 릴리즈의 상태와 기능, 기존 서빙 엔진과의 관계 및 차이점, 내부 구조와 프로젝트의 방향성에 대해 설명합니다.

Transcript

1. Furiosa LLM: High performance LLM Serving for RNGD Hyunsik Choi Head of Platform Software
2. 1 FuriosaAI 2 Introduction to Furiosa LLM 3 How We ve Evolved Furiosa LLM 4 Roadmap RNGD Latest Releases
3. Introduction to FuriosaAI RNGD
4. Introduction to FuriosaAI RNGD Furiosa NXT RNGD Server 넥스트 레니게이드 서버 8 RNGD 4 petaFLOPS 384 GB 12 TB/s Cards 2026년 1월 부터 양산 512 TFLOPS HBM3 용량 8 cards 메모리 대역폭 3 kW 소비 전력
5. Introduction to Furiosa LLM
6. Furiosa LLM at a Glance • vLLM Compatible Python API and OpenAI compatible API, Responses API • Support for text Generation, Multi modal VL , and Pooling models embedding, reranking, classification • PagedAttention Prefix Management with Radix Tree • Continuous and mixed batching with chunked prefill • Quantization: INT4, INT8, BF16, FP8, MXFP4, and NVFP4 • Tensor, pipeline, and data parallelism support • Tool calling, reasoning parser and structured output • Production metrics through Prometheus • Overlap scheduler Async scheduler • Experimental KV Connector • Experimental HiCache • Model support: Llama, Exaone4/4.5, Solar Open, Gemma4, Qwen3, .. Mooncake TransferEngine backend PD Disaggregation support FuriosaAI SW Stack Overview
7. Furiosa LLM at a Glance … Supporting modern LLM serving APIs including responses API, tool calling, structured output, ... vLLM compatible API and SOTA LLM optimizations Seamless integration into existing cloud infrastructures
8. Furiosa LLM at a Glance GPU A equivalent tier Power Efficiency of RNGD vs. GPUs, Benchmark Report of 2026.4.0 Release, FuriosaAI
9. Furiosa LLM: History and Motivation History •September 2023: Began research and development on LLM stack •May 2024: Began RNGD bring up June 2024: First GPT J demo •August 2024: HotChips demo OpenAI PagedAttention, GPT J, Llama •October 2024: Version 2024.1 released first release of Furiosa LLM •…. •Sep 2026: Version 2026.4.0 released Gemma4, Exaone4.5, Experimental HiCache, VL model optimization, .. •Oct 2026: Planned release of version 2026.5.0 Qwen3.6/3.8, Solar Open2, PD, HiCache, .. Motivation •Differences in architecture, compiler, and programmability •Challenges in leveraging open source ecosystems •No OSS community governance
10. Furiosa LLM: How it works
11. How We ve Evolved Furiosa LLM
12. Efficient Batching and KV Cache Management 3 New requests must wait What is 1 Wasted computation and IO Completed request Paged 2 Wasted Memory Batch Axis … PAD PAD Hi There Hi EOS EOS EOS What is LLM ? LLM stands for Large prompts Generated tokens … current sequence length Maximum context length
13. Efficient Batching and KV Cache Management PagedAttention maps logical KV blocks to physical memory blocks, allowing each request s KV cache to grow without requiring contiguous storage. Efficient Memory Management for Large Language Model Serving with PagedAttention, Woosuk Kwon et al., ACM SOSP 2023
14. Efficient Batching and KV Cache Management Continuous Batching with PagedAttention • • • • Update the batch every step. Completed requests l eave; waiting requests join as compute and KV memory capacity allow. Keep active requests running. Admit new requests while ongoing requests continue decoding. Grow KV storage on demand. Map logical KV blocks to physical blocks without requiring contiguous storage. Reduce memory waste. Support more concurrent r equests and improve resource utilization.
15. Handling Dynamic Shapes with Static Compilation Prefill Challenges • Prompt lengths and active batch sizes vary at runtime • Context lengths grow as decoding progresses • Our compiler requires fixed tensor shapes at compile time Static graphs, dynamic scheduling adapt dynamic requests to precompiled shapes at every execution step • Bucket aware scheduling Select compatible compiled kernels based on input length, KV length, and batch size. • Packing masking Pack tokens across requests; pad and mask unused positions to match fixed shapes. Decode Requests ? Extend
16. Balancing Prefill and Decode Timeline Prefill Prioritized Schedules Time between tokens without prefill interference Delay between tokens Stall free Schedule Taming Throughput Latency Tradeoff in LLM Inference with Sarathi Serve, Amey Agrawal et al., OSDI 2024
17. Balancing Prefill and Decode Mixed batching: decode extend prefill at each execution step
18. Accelerating Model Development Approach 1: Graph First What we tried • Reused Transformers models through graph rewritten and compiler optimizations • Preprocessed FX graphs for compiler input • Goal: Support any user provided model without dedicated implementations unlike TensorRT LLM/vLLM Challenges • Difficult graph pattern matching • Too many heuristics to maintain • Slow iteration • Hard to generalize pattern based optimizations PyTorch 2.0 PT2 Stack https://pytorch.org/get started/pytorch 2 x/
19. Accelerating Model Development Approach 2: Custom PyTorch Models What we tried • Reimplemented models in PyTorch, as vLLM and TensorRT LLM do • Split tokenwise computation and attention into separate kernels for mixed batching and paged attention • Structured model code to suit our compiler s FX/ATen path Challenges • Axis semantics became harder to track through our FX/ATen transformations. • Hard to express NPU optimization intent • Opaque compiler passes and slow iteration • More heuristics, higher maintenance costs • vLLM s model redesign and the SGLang IR proposal highlight similar challenges and community efforts to address them. PyTorch compilation process link
20. Accelerating Model Development Approach 3: TCL Tensor Contraction Language What TCL is • A Python DSL for tensor computations with explicit axis semantics • High level abstractions with explicit control over data reuse, parallelization, and fusion on TCP What Changed • Reuse optimized kernels across models • Build new models from existing kernels and implement only new components • Much faster kernel development, model integration, and optimization • All models, including legacy models, use TCL as of release 2026.4 • Multiple teams are exploring TCL beyond LLMs, including diffusion, vision, and custom models Public release planned by the end of 2026
21. Runtime Evolution: Lower Overhead, Simpler Execution Problem: Repetitive kernel launches and I/O operations cause significant runtime overhead. Example: Instruction memory double buffering Instruction memory size Target program size p0 p1 load p0; launch p0; load p1; launch p1; load p2; launch p2 I/O Comp load p0 load p1 launch p0 load p2 launch p1 launch p2 p2
22. Runtime Evolution: Lower Overhead, Simpler Execution HAL program is based on TCP s Programming Interface An Example of HAL program j0 y j1 Chapter 5 in TCP: A Tensor Contraction Processor for AI Workloads, Hanjoon Kim et al., ACM/IEEE ISCA 2024 x z j2
23. Runtime Evolution: Lower Overhead, Simpler Execution Approach 1 HAL Program Example: Instruction memory double buffering instruction memory size i0 target program size i1 p0 p1 p2 load p0 to i0; (launch p0 | load p1 to i1); (launch p1 | load p2 to i0); (launch p2 | load p3 to i1); (launch p3 | load p4 to i0); launch p4 I/O Comp load p0 load p1 load p2 load p3 load p4 launch p0 launch p1 launch p2 launch p3 launch p4 p3 p4
24. Runtime Evolution: Lower Overhead, Simpler Execution HAL Program Limitations • Explicit synchronization across programs • Complex composition and dynamic execution • task creation and synchronization overhead HAL program is based on TCP s Programming Interface An Example of HAL program j0 y j1 x z j2
25. Runtime Evolution: Lower Overhead, Simpler Execution Approach 2: Stream API A stream is an ordered execution queue for kernel launches and data transfers. • Submit work without waiting for every operation to finish • Preserve execution order within each stream • Use events to express dependencies between streams • Synchronize when the host needs completed results An example of event based synchronization across streams
26. Roadmap Latest Releases
27. Roadmap Latest Releases 2026.1 Feb 17, 2026 2026.3 June 30, 2026 Mixed batching by default Prefix caching Pooling model support llm d support Intelligence routing Production metrics K8s DRA, NPU operator support … 2026.2 Apr 30, 2026 Prefix aware DP router Prefix caching by default Responses API Per model bucket presets Qwen3/Exaone4 optimization TCL Kernel framework Vision language model support Solar Open, Qwen3 VL, K Exaone, .. Overlap scheduling support SWA support in Unified Radix Cache … 2026.5 Planned for Oct 24, 2026 Linear attention backend support Qwen 3.6/3.8, MedGemma, .. HiCache optimization NPU sampling by default Overlap scheduling by default Video modality support in VL models … 2026.4 Sep 30, 2026 Hierarchical KV cache offloading support Gemma4, Exaone 4.5, Harrier OSS v1, BGE, .. Vision language VL model optimizations KV cache and vision embedding cache in VL models Profile guided decode batching by default Experimental NPU sampling support KV cache connector mooncake backend support
28. Hierarchical KV Cache HiCache Three Tier Memory Hierarchy for KV Cache Management Multi Tier KV Cache Memory Management Using UnifiedRadixCache
29. Overlap Scheduling CPU overhead NPU Sampling CPU overhead
30. Furiosa LLM: Today and What s Next • Compatible APIs: vLLM compatible Python interfaces and OpenAI compatible Chat C ompletions and Responses APIs enable drop in replacement. • High Performance, High Efficiency: Vertically optimized for RNGD, delivering high per formance per watt. • SOTA Within Reach: We expect state of the art feature and model support for our target area by the end of 2026. • Building an Open Ecosystem: We plan to publicly release the TCL library, open source Furiosa LLM, and build the ecosystem together with the community.
31. Q A