Running gpt-oss 120b on AlmaLinux 10.1 and Ryzen AI 300 Series

Transcript

1. Cybertrust Japan Co., Ltd. Chief Open Source Officer Takanori Suzuki Running gpt-oss 120b on AlmaLinux 10.1 and Ryzen AI 300 Series
2. Intro • Hardware Used: Framework 13 (Model equipped with AMD Ryzen™ AI 300 series) ◦ Features integrated GPU (iGPU) adopting the "Strix Point" architecture • Target LLM Model: gpt-oss 120b (A large model requiring VRAM 60GB or more) Motivation: I bought a Framework 13 for private use, so I tried it out during the New Year holiday. (It's unrelated to this talk, but Framework is fun because various parts are interchangeable, so if you're interested, let's talk later.)
3. System Environment • OS: AlmaLinux 10.1 • GPU: Radeon 890M (Ryzen AI 9 HX 370 Integrated GPU) • Software Stack: ◦ AMD ROCm ◦ llama.cpp (HIP option enabled to utilize AMD iGPU)
4. Setup 1. ROCm Installation: Executed the official Quick Start procedure for RHEL 10.1 a. https://rocm.docs.amd.com/projects/install-on-linux/en/develop/install/quick-start.h tml 2. llama.cpp Build: Compiled with HIP enabled, specifying the Ryzen AI 300 series (Strix Point) GPU target "gfx1150". (Note: "Strix Halo" like AMD Ryzen AI Max+ 395 should be "gfx1151") $ mkdir build && cd build $ cmake .. -DGGML_HIP=ON \ -DAMDGPU_TARGETS=gfx1150 \ -DCMAKE_HIP_PLATFORM=amd \ -DHIP_PLATFORM=amd $ cmake --build . --config Release -j $(nproc)
5. Demo $ GGML_CUDA_ENABLE_UNIFIED_MEMORY=ON ./llama-server \ -m /path/to/gpt-oss-120b-GGUF/gpt-oss-120b-MXFP4-00001-of-00002.gguf --flash-attn on --ctx-size 0 --jinja -ngl 99 --no-mmap \ -ub 1024 -b 1024 gpt-oss 120b processing performance: around 16 tokens/s $ GGML_CUDA_ENABLE_UNIFIED_MEMORY=ON ./llama-server \ -m /path/to/gpt-oss-20b-GGUF/gpt-oss-20b-MXFP4.gguf --flash-attn on --ctx-size 0 --jinja -ngl 99 --no-mmap \ -ub 1024 -b 1024 gpt-oss 20b processing performance: around 23 tokens/s Note: There are also reports of 33.05 tokens/s for gpt-oss 120b on the Ryzen AI Max 395+
6. Next Step? By using AlmaLinux 10.1 and the latest AMD ROCm environment, it is now possible to run large LLMs even in a notebook PC environment. • Future Expectation: It would be great if AMD GPU Certification for AlmaLinux proceeds, following the NVIDIA GPU.