High-performance LLM/VLM inference on AMD Strix Halo with mlxcel engine: Zero-Bloat Engine for Unified Memory Architecture

*This session was presented at AMD Advancing AI Seoul 2026 on November 4, 2026.

A 96 GiB unified-memory machine like AMD Strix Halo can hold a 30B model in GPU-addressable memory, but most inference engines were built for discrete GPUs and leave the zero-copy path unused. This talk presents mlxcel, a Rust inference engine built on Apple MLX. It serves LLM, VLM, embedding, speech and diffusion workloads with no Python in the request path, and its ROCm backend is now in v0.8 beta. We cover the 70 fixes we made to the community ROCm fork of MLX, the unified-memory problems we hit on RDNA 3.5, and how profiling set the order in which we ported kernels. Decoding block-float weights in software raised MXFP4 prefill on gpt-oss-20b by 70x, and capping in-flight GPU work cut peak memory 3.4x. Across 44 models, dense models reach 69-84% of the memory bandwidth limit: Gemma 4 31B decodes at 10.2 tok/s against a ceiling of about 10.8 tok/s, and a 30B-A3B MoE model reaches 76 tok/s. MoE models still leave a third or more of the bandwidth unused, and that is the next target.

Transcript

1. AI Enterprise AI Open Source AI MLOps AI Cloud AI PC High performance LLM/VLM inference on AMD Strix Halo with mlxcel engine Zero-Bloat Engine for Unified Memory Architecture Jeongkyu Shin Lablup Inc.
2. 2 Why Local Inference on Unified Memory Is Hard A 96 GiB unified memory box can hold a 30B model, but the engines people run on it were designed for other hardware. Built for discrete GPUs Hard to deploy and operate Leading engines assume VRAM behind PCIe or NVLink and leave the zero copy path of unified memory Apple Silicon, GB10, Strix Halo unused. Python/PyTorch stacks weigh about 13 GB, which hurts most on air gapped, on premise sites. C/C engines manage memory by hand, and few ship batching or multi tenant serving.
3. 3 mlxcel: High performance LLM/VLM inference engine Every workload No conversion step LLM, VLM, embedding, reranking, speech and diffusion in one engine mlx community checkpoints load as is, no Python in the request path Unified memory first Expanding reach Tuned for Apple M1 M5, NVIDIA GB10 DGX Spark and AMD Strix Halo; also runs on NVIDIA Hopper to Blackwell AMD ROCm in v0.8 beta; experimental native Windows; alpha TPU/NPU via OpenXLA mlxcel builds on Apple MLX and ports model coverage from mlx lm and mlx vlm MIT ; see NOTICE in the repo.
4. 4 UMA Advantage for MoE Architecture Discrete GPU: HBM behind PCIe CPU DRAM PCIe copy Unified memory: one shared pool GPU HBM CPU GPU Unified memory pool • Weights live in VRAM; overflow is copied in over the bus • CPU and GPU address the same tensors, with no PCIe copy in between • Data dependent MoE routing can stall each token on an expert transfer • Full sparse checkpoints stay resident in system memory • Host staging copies double the footprint and the bandwidth • Decode is bounded only by memory bandwidth Strix Halo uses the unified layout: one APU with a 96 GiB GPU addressable pool, so a 30B MoE checkpoint stays resident and each token reads only its active experts.
5. 5 Decoupled Architecture: Control meets Data ENTRY POINT CONTROL PLANE Rust DATA PLANE C Core HARDWARE BRIDGE EXECUTION C , CUDA C , ROCm C , IREE MLX Runtime Metal Model Loading Generation Loop mlxcel CLI generate / serve Apple Silicon Auto Detection KV Cache Tokenizer mlxcel server OpenAI HTTP Multimodal Pipeline Batch Scheduler Sampling Speculative Decoder FFI Bridge mlxcelverse CUDA NVIDIA mlxcelverse ROCm AMD mlxcelverse OpenXLA Google
6. mlxcel on AMD Strix Halo
7. 7 AMD Strix Halo: The Target Platform UNIFIED MEMORY GPU MATRIX MATH 96 GiB RDNA 3.5 bf16 GPU addressable LPDDR5X carve out on one APU; the OS sees a separate 31 GiB Radeon 8060S gfx1151 on Ryzen AI MAX 395; wave32, the same width as a CUDA warp hipBLASLt capability probe reports bf16 1, int8 1, fp8 0; there are no FP4 matrix instructions int8 Block float weights are decoded in the kernel Two memory budgets MXFP4, MXFP8 and NVFP4 weights are unpacked to fp16 or bf16 inside each kernel, then multiplied with FMA or WMMA mlxcel checks model size against the 96 GiB device pool. Builds and CPU reference runs must fit in 31 GiB of OS RAM. Test host: Ryzen AI MAX 395 with Radeon 8060S, Debian 13, ROCm 10.0 HIP 7.15 . 96 GiB is the configured GPU carve out.
8. 8 ROCm on RDNA 3.5 vs. Metal vs. CUDA Aspect ROCm on RDNA 3.5 gfx1151 Metal Apple Silicon CUDA NVIDIA SIMD width wave32 CDNA GPUs: wave64 simdgroup of 32 warp of 32 Matrix units WMMA 16x16x16, bf16/f16 simdgroup matrix Tensor Cores FP8 / FP4 math None: decoded in software None: decoded in software FP8 on Hopper , FP4 on Blackwell Unified, split pools: 96 GiB GPU, 31 GiB OS Unified, one coherent pool Discrete VRAM behind PCIe Kernel build Mostly ahead of time, some hipRTC JIT Generated at runtime, cached NVRTC JIT, cached After a GPU fault Every HIP call fails until exit Error reported through MLX events Sticky error until reset Memory Numerics: gfx11 WMMA returns 2 ²⁴ for 1 1 in bf16 and f16; errors stay within the standard f32 dot product bound. Metal and CUDA columns describe the platforms in general; they were not measured on the Strix Halo host.
9. 9 Bringing ROCm into mlxcelverse Community ROCm fork mlxcelverse ROCm overlay NripeshN/mlx MIT : 106 new backend files, 43.9K lines 111 backend files 16 core MLX files, copied whole at build time Overlays replace whole files. Core MLX files are copied whole at build time, so a pin bump never hits patch fuzz or a hunk applied twice. libmlx at mlxcel s MLX pin The same pin Metal and CUDA build on; ROCm files apply only to ROCm builds Drift check on every pin bump. It runs offline in 2 s inside make verify, so a bump made on a Mac fails if ROCm was not retargeted. PORTING TIMELINE, 2026 Sep 11 Sep 12 Sep 30 Spike: fork builds alone, 42 of 42 ops pass Overlay merged into mlxcel First throughput baseline on gfx1151 Oct 15 Oct 26 ROCm epic closed: 12,384 Native Windows build and tests pass native NVFP4 kernels
10. 10 70 Fixes to the ROCm Fork, by Category Most of these defects also exist in the community fork. Each fix ships with a test that fails when the fix is reverted. Grid limits Thread safety Quantized kernels 32 bit grid math failed to launch past 2³² threads, and one kernel left the last 257 elements unwritten; now 64 bit grid stride BLAS handles, the JIT cache and the event pool had no locks; 8 thread tests went from 20 of 20 failing to 0 of 20 Wrong scale strides and dtype dispatch gave NaN or relative errors up to 1e34 on MXFP4 and MXFP8 Missing primitives GPU fault handling Build system FFT, Hadamard and SearchSorted were stubs that called std::terminate; now native HIP kernels After a queue fault every HIP call fails. Waits used to block forever; they now return the error. HIP objects ignored header edits 0 of 41 rebuilt ; dependency files now track them
11. 11 Unified Memory Costs on Strix Halo SPLIT POOLS UNCACHED CPU READS SHARED CACHE LINES Budget from device memory CPU fallback stalls the GPU Silent wrong output Sizing from OS free memory would have skipped a 26 GB Mixtral. Budgets now use the 96 GiB pool. CPU side NVFP4 dequantize kept the GPU idle 1 s of a 1.5 s prefill. It now runs on the GPU. Multithreaded OpenBLAS and packed GPU scalars read stale lines. Fixed with 1 BLAS thread and 128 byte slots. 3.4x lower peak memory: capping in flight GPU work cut Llama 3.1 8B 4 bit from 20.6 GB to 6.1 GB, with decode speed unchanged. Before the cap, the host committed work every 2,000 ops without waiting on the GPU.
12. 12 Software Decode for Block Float Formats MXFP4 PREFILL MXFP4 DECODE NVFP4 PREFILL 70x 7.6x 3.1x gpt oss 20b, 512 tokens: 7.5 526 tok/s with expert batched kernels gpt oss 20b: 8.3 63.6 tok/s; the old per row kernel used 100 ms of each 115 ms step Nemotron Cascade 2 30B A3B, 512 tokens: 329 1,008 tok/s with GPU dequantize WMMA Branchless bit placement WMMA by measured row count Each E2M1 nibble is written into fp16 sign, exponent and mantissa bits; one final multiply removes the 2 ¹⁴ offset, so decode is exact RDNA 3.5 uses fused WMMA only below 128 rows, where it beat dequantize hipBLASLt; NVFP4 WMMA tiles start at 16 rows per expert
13. 13 Porting Order Set by Profiling Decode speedup per HIP kernel port, gfx1151 Dense decode is one kernel Paged attention, 32K 1.77x qmv wide takes 95 time, at 192 GB/s of Llama 3.1 8B decode GPU Hybrid models are launch bound 2,000 to 3,200 kernel launches per token and a 5 ms host gap per token SSM update, Granite 4.0 H Tiny 1.46x SSM update, Nemotron 3 Nano 1.45x The profile changed the order 1.40x GPU time overturned the call count plan: SSM update went first, fused add RMSNorm and RoPE went last Paged attention, 16K Samplers, Llama 3.1 8B 1.11x Fused MoE, Qwen3 30B A3B 1.02x Speedup decode tok/s after vs. before each port, batch 1. Paged attention measured on the server path Llama 3.1 8B 4 bit .
14. 14 Strix Halo: 44 Model Sweep Decode throughput, 20 47B models, Strix Halo MoE in blue Nemotron 3 Nano 30B A3B 75.9 tok/s gpt oss 20b MXFP4 64.2 tok/s Qwen3 30B A3B 63.2 tok/s • Experimental Setup mlxcel v0.8.1, ROCm 10.0 pp512/tg128, batch 1, one pass per model • Compatibility: 44 models tested, all run Qwen3.6 35B A3B 50.0 tok/s Dense, MoE and VLM; 4 bit, FP8, MXFP4 Gemma 4 26B A4B 48.1 tok/s From mlxcel s 167 supported architectures Mixtral 8x7B 26.4 tok/s Devstral Small 2 24B 15.9 tok/s Qwen3.6 27B 12.8 tok/s Gemma 4 31B 10.2 tok/s • Active parameters set the speed 3 4B active MoE: 3.8 5.9x dense 27B Mixtral 13B active : 26.4 tok/s Source: mlxcel ROCm sweep Nov 3, 2026 , one pass per model, 8 low outliers re measured once; Ryzen AI MAX 395, Radeon 8060S gfx1151 , 96 GiB carve out, HIP 7.15. Absolute tok/s, not a ratio to mlx lm.
15. 15 The Memory Wall: Gemma 4 31B Across Engines Gemma 4 31B, 4 bit, Strix Halo, single stream pp512/tg128 Engine Backend Prefill Decode mlxcel v0.8.1 ROCm 315 tok/s 10.2 tok/s llama.cpp ROCm n/a 10.5 tok/s llama.cpp Vulkan 260 tok/s 10.9 tok/s llama.cpp QAT Vulkan 276 tok/s¹ 10.3 tok/s 10.8 tok/s • Decode lands on the ceiling Four runs, 10.2 10.9 tok/s, within 6 • Memory sets the ceiling 212 GB/s measured 10.8 tok/s 19.6 GB of weights • Prefill depends on the engine Compute bound: mlxcel 315 vs. 260 tok/s bandwidth ceiling for a dense 31B 4 bit model on Strix Halo ¹5,471 token prompt, server run. All rows 4 bit weights; different hosts. Ceiling: measured 212 GB/s 19.6 GB of weights EVO X2 report . Sources: mlxcel ROCm sweep Nov 3, 2026 ; llama.cpp GitHub Discussion #23659 May 2026 ; EVO X2 Gemma 4 31B benchmark, frey bafang.patransformers.com Jun 2026 ; hogeheer499 commits/strix halo guide Jul 2026 .
16. 16 Where the Bandwidth Goes: Dense vs. MoE Est. decode bandwidth, of 256 GB/s blue: 3 4B active MoE Llama 3.1 8B 84 Devstral Small 2 24B 84 Qwen3.6 27B 76 Gemma 4 31B 69 Mixtral 8x7B 75 Nemotron 3 Nano 30B A3B 58 Qwen3 30B A3B Gemma 4 26B A4B Qwen3.6 35B A3B 46 40 • Dense models sit near the ceiling 69 84 of 256 GB/s; only fewer bytes per token helps • MoE leaves bandwidth unused 33 58 ; routing and kernel launches are the next target • MoE prefill is the slow step 300 vs. 1,061 tok/s for Llama 3.1 8B: 1.6 s to first token 33 Estimate: active parameters 4.5 bits per weight Peak: 256 bit LPDDR5X 8000. decode tok/s Nov 3, 2026 sweep , weights only; KV cache reads excluded, so true utilization is slightly higher.
17. 17 Batched Serving on Strix Halo Aggregate throughput at 4 requests vs. 1, Strix Halo • Experimental Setup mlxcel server, parallel 4, v0.8.1 512 token prompt, 128 tokens out Llama 3.1 8B 2.0x • 4 requests: 2x aggregate Llama 3.1 8B: 34 to 69 tok/s Qwen3 30B A3B: 37 to 68 tok/s • Cost: each request runs slower Qwen3 30B A3B 1.9x Llama 3.1 8B: 41 to 19 tok/s each Qwen3 30B A3B first token: 1.6 s to 3.1 s Source: mlxcel ROCm batch ladder Nov 3, 2026 , one pass per level; B 2 and B 4 start from a warm prompt cache. Aggregate includes prefill.
18. 18 Wrap up • What we built One Rust engine over MLX for Apple Silicon, NVIDIA and AMD ROCm, with no Python layer • What Strix Halo taught us Budget from the 96 GiB device pool, profile before porting, and expect MoE to gain most • Try it github.com/lablup/mlxcel; ROCm builds with features rocm in the v0.8 beta ON STRIX HALO 44 models 10.2 tok/s 76 tok/s tested, of 167 supported architectures dense 31B decode, at the memory ceiling 30B A3B MoE decode
19. 19 AI Enterprise AI Cloud AI Open Source AI MLOps Thank you! contact lablup.com https://www.facebook.com/lablupInc Lablup Inc. https://www.lablup.com Backend.AI https://www.backend.ai Backend.AI GitHub https://github.com/lablup/backend.ai Backend.AI Cloud https://cloud.backend.ai