AI 챗봇에서 Agent로 사용처가 옮겨가며 토큰 사용량이 급격하게 증가하는 지금, 사용량과 함께 토큰 생성 속도에 대한 수요도 높아지고 있습니다. 최근의 LLM Inference 속도 가속 방법론들에 대해 살펴보고, Diffusion 방식을 사용하는 생성 속도 빠른 LLM 모델을 학습한 이야기를 다뤄봅니다.
Faster Token Generation 3 What We Are Training How LLMs Generate Tokens · Prefill and Decode, KV Cache · Latency, Throughput, TPS GPU/HBM bottlenecks · KV Cache Quantization · Speculative Decoding Speculative Decoding implementation · Training process and results key SD techniques
LLM ③ Sell tokens cost side your infra revenue side GPU LLM deployed 15 / 1M tokens GPU billed per token generated GPU t1 hourly billing / depreciation e.g., B200 1.99 / hr Cut cost t2 t3 serve requests generate tokens revenue scales with tokens sold use fewer GPUs Quantization · or produce more tokens per GPU Throughput · Speculative Decoding · Diffusion LLM
all at once Hello how are LLM forward 1 KV Cache all input tokens step 1 ③ Decode step 2 first token generated feed back repeat KV Cache reused Hello are LLM forward 1 LLM forward 1 you doing append parallel across tokens compute bound how one pass one new token memory bound you repeat per token N tokens N sequential passes cost scales with output length
K,V K,V The quick brown fox jumps over ① Prefill ② Decode prompt all input tokens in one parallel pass K,V written at once Each token s K,V is computed once and stored output one token per pass reads the cache, appends its own K,V decode reuses it instead of recomputing the whole prefix
all tokens Kᵀ 64 4 KV Cache Step N decode 1 new token q · 1 64 Out 4 64 step 1 writes K, V step N reads them back K and V stored once per token cached Kᵀ · 64 N 1 computed this step V 4 64 cached out · 1 64 V · N 1 64 reused from cache With the cache, step N multiplies only the new token s q K and V just grow by one entry per step
token per pass first token TTFT time to first token prefill TPOT time per output token elapsed time Chat / simple conversation little prefill TPOT dominates perceived latency Agentic workload Claude Code … heavy prefill each turn TTFT dominates 600k context 10 min to first token
1 e m GPU Compute Units BW y or m Verify /s B G peak compute Where the time goes 312 TFLOPs/s all GPU compute SM matmul Prefill Decode all memory reads weights append ak e p Decode prefill Prefill Prefill GPU compute SMs memory access HBM operational intensity FLOPs/byte dense matmul sits at the compute ceiling input length KV full weight matmul same GPU, opposite bottlenecks: prefill needs FLOPs · decode needs bandwidth raw FLOPs dominate, GPU compute units are the bottleneck
KV cache size Sliding Window ② CSA last 128 tokens · uncompressed local branch n win m Compressed Sparse Attn 4 tokens 128 ¼ 1 KV entry · Top k selection ¼ 4 tokens 1 entry via softmax gated pooling lightning indexer picks Top k entries per query full local fidelity before compression KV ¼ shared by every CSA and HCA layer compress only 128 tokens back causal Layer stack L0 HCA L1 HCA L2 CSA selective view V3.2 DSA applied on the compressed sequence indexer cost also drops to ¼ L3 HCA MLA V2·V3 compressed the per token latent dim L4 CSA L5 HCA … m Heavily Compressed Attn 128 tokens 128 tokens ¼ older tokens are not seen by this branch only recent tokens keep full resolution K/V Top k sparse ③ HCA CSA / HCA 1 KV entry · dense 1 entry 1/128 1M tokens 8K entries dense attention over all entries, no indexer KV 1/128 · cheap global view trades per token detail for full context reach coarse view complementing CSA s sharp one L0_1 are HCA, then CSA / HCA interleaved every layer also carries a SWA 128 branch V4 compresses the sequence axis itself · at 1M ctx: 10 KV cache, 27 FLOPs vs V3.2
per coord 0.42 1.18 9.73 0.07 0.63 codebook outlier 9.73 dominates the quantization scale ① rotate rotated 0.71 N 0, 1/d 0.88 0.79 ② Lloyd Max codebook quantized 3 bit Lloyd Max centroids for N 0,1 2.15 · 1.34 · 0.76 · 0.25 · 0.25 · 0.76 · 1.34 · 2.15 same for every model layer Π·x, random orthogonal precomputed once outliers spread out, coords 0.54 0.66 b bits per coord 2 4 bit ③ QJL sign bit sketch residual error 8 centroids per coord c₅ c₂ training free · calibration free c₄ keep sign c₆ codebook come from math, not data near optimal 16 JL projection 1 bit attention scores stay unbiased c₈ rotation precomputed 3 4 bit per coord proven per vector lower bound 4 5 smaller KV cache · negligible accuracy loss Google Research · ICLR 2026
but tiny Quick Draft model N candidate tokens drafted ahead ② Verify 4B brown cheap fox hopped over negligible cost next to the target model ONE forward pass of the target model checks every draft Target model 397B per token probability check fox hopped over 1 from target accept accept reject discarded free bonus token accept until the first reject, target supplies the token there 1 expensive pass MAL brown 2 accepted drafts Mean Accepted Length 1 target token vs 1 token/pass in plain AR average tokens accepted per draft round
what one forward pass of the target model yields AR vanilla decode MTP Medusa · DeepSeek DFlash block diffusion draft dLLM WeDLM · Block Diffusion t1 t2 1 pass 1 token t1 d1 1 pass 1 token t1 d1 t3 t4 no drafter t2 final d draft 1x N tokens need N sequential passes of the full model d2 d3 1.4 2x drafts made sequentially heads / small AR d2 d3 d4 1 draft pass fills the whole block in parallel t1 t t3 t4 t5 d5 verify next pass d6 2 4x target verifies all drafts at once t6 the model itself fills N positions per pass, committing by confidence 3 10x
hidden state LM head original final hidden state h in parallel t₀ next token Medusa head 1 draft 1 Transformer decoder blocks N target model Medusa head 2 draft 2 Embedding Medusa head k … draft k Prompt One backbone forward tokens so far 1 same h head every N tokens · heads are small MLPs on the frozen backbone · drafts verified next
MTP module 2 full trunk L blocks 1 extra block 1 extra block h h₁ input t₁ … tₙ Transformer blocks L h emb tₙ ₁ h₁ Transformer block 1 emb tₙ ₂ Transformer block 1 output head output head output head tₙ ₁ draft tₙ ₂ draft tₙ ₃ Medusa one h, N heads in parallel · DeepSeek MTP each draft conditions on the previous token chained 1 block modules causal chain higher acceptance
late decoder layers middle decoder layers early can fused feature concat l · m · h MLP FC fuse late features do tokens generated so far decoder layer 1 LM head d₁ feed back middle I g embedding early tiny, autoregressive richer signal than final h only MTP d₂ d₃ draft next token AR tiny 1 layer drafter, higher acceptance
Diffusion is good draft KV cache projected h ③ One parallel pass d₁ d₂ d₃ N 1 candidates, all at once t₀ Target model big mask mask mask target forward t₀ decoded bidirectional attention h multi layer BERT like, 2 layers hidden features are projected into the draft s KV cache mask slots take their values from the target s embeddings The drafter is not autoregressive decoded token MLP verify 1 new token accepted start the next block repeat until done the whole block appears in one bidirectional pass · AR happens only at target verify mask slot draft / projected features
one pass t₀ M M the Spark M low rank MLP / RNN c₁ c₂ c₃ sequential but 1 of the cost parallel block bidirectional logits 2 raw drafts logits 3 tail gets weak DSpark DFlash DFlash d₁ d₂ d₃ no full vocab logits computed just a correction on the draft a tiny AR corrector d₄ late positions often rejected DSpark d₁ d₂ d₃ quality holds to the block tail nudge each position s draft logits logits 1 ③ Effect on accept rate higher MAL more tokens accepted per target pass accepted parallel draft kept, tail quality fixed, 1 rejected overhead d₄
M M block diffusion drafter d₁ d₂ single candidate d₃ M 1 pass d₄ DFlash 2 top 16 per slot quick dark fox runs fast brown dog leaps swift red cat jumps ⋮ 16 ⋮ 16 ⋮ 16 ⋮ 16 selector scores adjacent pairs two tap dynamic conv tail slots often rejected draft quality decays toward the end of the block Same one pass draft 16 options per slot lossless a tiny selector path selector one coherent path no tail decay greedy output identical to the target 1 accepted token per pass up to 4.6
attention mask causal each token sees only the past decoding bidirectional inside the block decoding This is a pen t 0 t 1 t 2 t 3 left to right · one token per step Same transformer different mask t 1 This M a M t 2 This is a pen any order · several per step decode order · diffusion within a block, AR across blocks
token per pass and no each new token b A2D e.g., Diffusion Gemma makes Jack one full model pass 8 tokens 8 passes work and no block 1 a … denoise a 4 token canvas, commit, move on denoise loop All play until converged play makes all 4 commit at once AR: N passes for N tokens · A2D: a few denoise passes per block denoise loop Jack block 2 a until converged dull boy real model: 256 token canvas the whole block commits at once
data pipeline; the draft model learns to predict the target model s own outputs DFlash recipe Prompts Nemotron V2 Code Alpaca Target model generates the responses DFlash draft trained on target outputs DFlash authors data Nemotron Post Training Dataset V2 Alpaca Code The real world problem Single turn chat data Agent workloads Korean queries That dataset is chat data and single turn, so it only lifts narrow performance In environments that really need fast decode, like Claude Code, it doesn t help much… With Korean queries the model answers in Korean, and the speedup is small… So let s build a real dataset from agent environments
agents vLLM endpoint Claude Code · Codex · Open Code OpenAI / Claude compatible API Logger API request / response log What an agent workload needs 1 Codebase 2 User query 607 GitHub open source repos 17,615 issues 16,737 KO translated CC / CD / OC agents 103k 6,000 180k sessions · 50/50 EN·KO sessions actually executed trajectory turns Sessions repos issues GLM 5.2 DFlash 3 agents; every intermediate model call is logged as training data Final MAL 2.77 · TPS 1.92x vs. 1.4x, paper data https://huggingface.co/lablup/GLM 5.2 NVFP4 DFlash ko
10 more executed sessions training in progress Data factory run 607 17,615 16,737 issues · EN GitHub repos KO 103k 64,000 300k sessions · CC/CD/OC sessions executed trajectory turns GLM 5.2 run vs. Kimi K3 run GLM 5.2 Kimi K3 6,000 Sessions executed 64,000 180k Trajectory turns 300k MAL agentic usage 2.77 2.47 proxy, in progress Training in progress proxy MAL agentic usage 2.47 · same 103k session pool, 64,000 executed so far
Qwen 3 30B A3B model Qwen3 30B A3B AR Diffusion Block 4 diffusion Base · no SFT, no RL CPT · 50B Nemotron Korean data SFT · 6B Open Solar SFT data Token budget · 56B total CPT 50B continued pretrain From the base checkpoint: convert to block diffusion, then CPT peers benchmarks below . SFT 6B SFT, no RL. Quality holds vs. AR lablup/dllm a2d sft qwen3 30b a3b
09 18 11.9B dense params · 36 layers · hidden 4096 DeepSeek V4 style compressed attention mHC AR dLLM one checkpoint two modes block diffusion loss λ trained jointly 0.2, Data mix · code family 75 60 Stack v3 60 Specialized 10 Synthetic Code 10 Web 5 AR mode servable now: 1M prompt on vLLM native NVFP4 64 1M B200 8 nodes · InfiniBand context from step 0 Megatron Core patches · NVFP4 / FP8 QAT Progress · day 9 · step 26,112 / 238,419 10 CC Code 5 10 10 CC Math 10 11 val AR loss · 54.8B clean tokens of the 500B first tranche 1.4 PPL 4 / 27 35 s/step diffusion boost Diffusion mode mask recovery improving · free generation not yet 0/4 One 10B checkpoint runs both as an AR model and as a block diffusion model trained at 1M context from day one
Stack v3 Nemotron Synthetic Code 10 stack v3 code75 Nemotron CC Math 10 hash pinned with checkpoint Nemotron Specialized 10 recipe Nemotron 2.0T tokens tokenized raw 25 TB document level Megatron indexed synthetic code · CC Code · math · specialized · web Nemotron CC Code 5 Nemotron HQ web 5 code family 75 Why code heavy? target local coding agent · math / specialized for reasoning · 5 web keeps language 60 repo level code 60 Stack v3 repository code 3.5 TB · 8,192 shards · 3.6T tokens repository level repo id / commit / files every non empty file, licenses kept as metadata Filter dropped permissive only non vendor kept just 2.5 of bytes train on everything 40 Nemotron 75 source level ratio, not audited literal code Not yet cross corpus dedup · decontamination · agent / SFT data post pretraining code family mix, pinned by hash for the whole run
· PAD endoftext · chat template kept vocab special tokens · padded to 151,936 Qwen3 Reuse Qwen3 s vocab code multilingual incl. Korean already validated vLLM / SGLang support Qwen tokenizer template out of the box inherited as is one extra token for diffusion noise beats uniform corruption by 0.23 nat Proven coverage Serving compatibility tokenizer BPE MASK Why not train a new tokenizer? 152,320 151,936 padding trimmed on export HF / vLLM convention chat Zero re tokenization 2T Nemotron tokens already indexed with it chat template, add a single MASK token nothing else changes downstream
h Stack v3 on demand · deterministic .bin / .idx shards Megatron blend EOS per doc 16 shards · 5.5M tok/s each token weighted · val via blend per split Weighted sampler Tokenize on 1st request 1,048,577 token cache Parquet text HF fast tokenizer document column Repo text repository file … sorted by path · 4M char cap Parquet row group repo unit random offset Why not pre tokenize 3.6T tokens? 14 TB of int32 days of compute compute once when 64 CP ranks first ask, then cache sample ID RNG position · cross node lock hash verified · resumable from consumed samples Hold out 0.2 of repos by repo id hash · val batch out 50 legacy NSP set 50 Stack hold Lesson learned GPFS flock cross node exclusion PID temp names collided across pods 6 corrupted checkpoints, rolled back 200 steps Fix: POSIX record locks · unique temp files · fsync · atomic rename stress tested with 32 workers on 4 nodes
Dual Objective the same sequence laid down twice 2L 2,097,152 tokens What a masked block may attend to one rule joins V4 compressed attention with block diffusion clean x noised M M M block 256 M block 256 M HCA 128:1 CSA 4:1 far prefix indexer top 512 window active last 256 raw bidirectional committed prefix · read causally from the clean stream block … Inside the block full bidirectional attention AR loss Diffusion loss causal · next token CE CE of true token at each M M MASK · per block draw ratio p, mask that share Noised copy same tokens data counted in clean tokens only Megatron Core port 4 patches · argmax match vs HF 99.3_99.8 QAT: experts NVFP4 · attention FP8 · rest BF16 serving grid Before the block causal · last 256 tokens raw, older via CSA / HCA compressed keys Boundary safety only entries closed before block start tests exactly 0.0 leak committed prefix KV cache is reusable at inference vLLM / SGLang block diffusion path ; full attention dLLMs can t One forward pass trains both objectives on the same 1M tokens and the prefix rule keeps KV cache reuse intact
+ λ · ( L_diff + w_fix · L_fix ) λ λ 0.2 · w fix 0.25 · L diff uses 1/p weighting masked diffusion ELBO 0.2 AR is the deployment insurance; larger λ 1/p variance hurts AR B 256 fixed serving default · 5090 ridge point 1.7 an AR step MASK not uniform uniform corruption: same AR, diff 0.23 nat worse p dual val loss nat · 1.5B pilot · 1B tokens 4.00 3.50 every sample dual keeps supervision density at 1M 3.56 2.71 Diagnosis 2.92 2.56 2.00 λ 0.2 λ val AR loss Since 9/16 3.69 3.56 3.00 2.50 1.0 smaller λ keeps AR intact 0.5 λ 1.0 val diff loss closing the train_inference gap 15k 75 mask 77 recovered 100 mask 12 8 passes make it worse 12.4 8.8 Root cause Fix A · p full p U 0,1 never yields a fully masked block teacher forcing decoder s own commits exposure bias half of all blocks fully masked zero extra cost Small λ protects AR quality; fully masked blocks 0.5 Fix B · Decoder in the loop 2 no grad passes commit in decoder order 75 gold / 25 model train on that state · fix loss on self commits decoder in the loop rollouts teach the model what it actually sees at inference
a clean slice its noised twin Parallelism on 64 GPUs TP clean R0 R1 R2 R3 R3 R2 R1 R0 noised R0 R1 R2 R3 R3 R2 R1 R0 halo 256 token sliding window boundary exchanged P2P between neighbours only clean PP EP 1 V4 hybrid attention QAT assume TP 1 CP64 · gbs 2 · no offload 27.08 s CP32 DP2 · needs offload 30.13 s compressed keys CSA 4:1 · HCA 128:1 · differentiable all gather noised tokens never compressed · all indices in global positions same RoPE / block rule on any split Why not stock Megatron CP? ring / all gather CP assumes one causal stream. We have two streams that reference each other, causality expressed as global index sets, and block boundary rules. CP must be a power of two 1M healthy nodes required CP 0 exactly 8 1M · CP64 speed ladder 9.5e 7 0.0 CP2 vs CP1 loss gap BF16 future leak gradient probes Two collectives only 4 proc halo fwd/bwd reference 8.09 s AR only 9/6 · 130K tok/s 27 s dual stream QAT recompute production a 256 token halo and an all gather of compressed keys let 64 GPUs share one 1M sample
4 of 238,419 steps · 11 interval means 1M samples / 100 steps noisy · Stack hold out 50 NSP 50 5.00 4.90 54.8B 4.00 3.97 clean tokens 10.2B diffusion boost 3.00 9 days 0 NaN · 0 skipped · lr 2.93e 4 2.00 Lineage 1.00 2.03 1.55 0_5k from scratch · Stack v3 recipe diffusion boost 10_15k 15_20k into the first tranche: AR loss 8.2 pre switch val AR loss 26,112 ckpt every 100 steps 166 GB, torch dist , 5 kept · earlier 4K / 1M transition runs not counted 11 5_10k 1.89 1.41 PPL 4 val AR 1.4 nat · no regression after the switch part 2.1 1.9 partial mask diff keeps falling boost last 10 pts val diff loss partial mask 10 16 decoder first pass accuracy top 32 slots 25 33 1.4 nat, diffusion loss still falling, and the objective switch cost nothing on AR
Jul Aug Sep Phase 1 · AR diffusion conversion a2d Phase 2 · Drafter NSP Phase 3 · CADQ Phase 4 · newone Phase 1 · Mar Aug Phase 2 · May Aug Phase 3 · Aug Sep Phase 4 · Sep now Convert existing AR models to diffusion a2d , 8 families Separate drafter DFlash/DSpark plus NSP self speculation CADQ: Qwen3.8 27B converted to WeDLM style parallel decoding newone: 10B DSv4 from scratch, AR block diffusion 3.6 55 1.6 3.4 2.4 4.8 of target steps reached. Runs diverged, lost IB, plateaued or collapsed into repetition drafter speedup. NSP stalled at 2.5_2.8 tok/NFE vs a 10 goal in verify mode only same output as AR . Parallel mode ran at 0.92 of AR decoding speed vs AR in 1 pass 716 vs 150 tok/s . HumanEval still 3.0 vs 20.1
Cluster failures Weak starting point Output looped Qwen3.5 MoE, Gemma4 Nemotron, DMax, Fast dLLM DiffusionGemma SDAR on Qwen3 30B A3B Large MoE and 32k context; clipping pushed effective LR to 0 NCCL/IB hangs, silent socket fallback, checkpoint OOM LR dose too high on top of an A2D warm start Token loss missed free generation repetition 0.85_0.90 Completed AR base Qwen3 30B A3B SDAR, block 4 8 model families WORKAROUND WORKAROUND WORKAROUND WORKAROUND Hold context at 16k; switch to the SDAR block 4 recipe Fix IB first; verify checkpoints before resume Start from a fully pretrained AR base Judge on samples; let AR verify drafts Phase 3 Also stopped: T4 MegaDLMs no fused kernel for head dim 256 , T7 Fast dLLM v2 CPT suspected label leak , T10 dInfer decoding too slow . Most conversions broke on scale or infrastructure; the run that finished used a validated recipe
Korean DFlash drafter v3F, speedup by benchmark Kimi K3 drafter: MAL 3.82 vs no spec 118 tok/s · MAL in brackets 1.04_1.15 CAUSE HumanEval 4.87 3.4× Training features did not match the serving model tap index off by one, wrong AttnRes capture HOW WE WORKED AROUND IT MATH500 2.88 2.0× MBPP 2.65 1.9× GSM8K 2.50 1.8× Gate every drafter on logit parity with the serving forward pass NSP: 2.5_2.8 tok/NFE against a 10 goal CAUSE MT Bench 1.94 1.6× The model accepted only about 30 drafts α 0.3 of its own HOW WE WORKED AROUND IT Earlier DFlash drafters May Jun, Qwen3.6 / Gemma4 : MAL 1.97 2.67, but no no spec baseline, so no speedup figure Moved to CADQ, where AR verifies diffusion drafts Phase 3 Drafters sped up decoding 1.6_3.4 when aligned with the serving model; NSP reached about a quarter of its goal
Qwen3.8 27B Parallel mode: 0.92 of AR at matched quality vs plain AR decoding CAUSE 2.4× Matching AR quality meant committing fewer tokens per pass, which removed the parallel speed gain HOW WE WORKED AROUND IT Verify mode: AR checks each diffusion draft, giving AR identical output at 2.4 1.0× Repetition loops in 11 of 128 samples 0.92× CAUSE Unverified drafts drifted into loops on fully masked blocks; more full mask training did not help HOW WE WORKED AROUND IT Plain AR Parallel mode Verify mode Verify mode rejects drafts that diverge from AR, so loops never reach the output Only verify mode beat AR: 2.4 , against 0.92 for parallel decoding
per request, newone 10B DSv4 one B200, vLLM, 256 token block Fast, but HumanEval 3.0 speedup over AR vs 20.1 for AR CAUSE AR vLLM native Filling a fully masked block without AR guidance; outputs repeated 91.9 duplicate 4 grams 150 tok/s HOW WE WORKED AROUND IT Diffusion, 1 pass 4.8 716 tok/s 2 passes 3.8 Kept AR mode and infilling as the usable paths; the 16K run targets free generation directly 571 tok/s 4 passes 2.8 418 tok/s 1M context diffusion cost 18,000 GPU hours CAUSE 8 passes 1.8 16 passes 1.0 269 tok/s 156 tok/s Each long context diffusion experiment tied up the 64 B200 cluster, so iterations were slow HOW WE WORKED AROUND IT Moved diffusion training from 1M to 16K context to run more experiments Diffusion decoding runs up to 4.8 faster than AR 716 vs 150 tok/s ; quality, not speed, is the open problem
tokens/s, same target model 92 / 88 / 77 newone vLLM, 1 pass* 4.8× GLM 5.2 drafter, HumanEval CADQ verify, 27B CADQ threshold, 27B math 3.4× Infilling, newone 10B: accuracy with 25 / 50 / 75 a 256 token block masked 90 100 of Similar code in context: windows decoded in parallel after 8 passes 2.4× 60 1.56× vs 62 CADQ 2B pilot, GSM8K: diffusion vs AR at 1.46 tok/fwd * 716 vs 150 tok/s 1.8 at 8 passes ; quality below AR. Threshold mode: 0.92 on varied prompts. INFRASTRUCTURE AND TRAINING 130K tok/s 0.02 nat 10B AR at 1M context on 64 B200, new context parallel for clean noised streams 100 NVFP4 training loss vs BF16 over 1B tokens Raw diffusion is fastest 4.8 match Greedy output, train HF FP8/NVFP4 vLLM; 1M prompt in 56 s 2.8 2B training speed from the CADQ fused GDN kernel ; AR verified drafts gave the largest speedups at AR quality
verifies, or part of the answer is visible FAILS: filling an empty block with diffusion alone Diffusion drafts, AR verifies First pass on a fully masked block 3.4 / 2.4 7 9 GLM drafter · CADQ verify, token exact 56/56 Partial mask infilling 92 / 88 / 77 Free generation on HumanEval 25 / 50 / 75 of the block masked Similar code already in context 90 100 first pass accuracy, newone 10B parallel decoding after 8 passes 3.0 vs 20.1 diffusion vs AR decoding Training on more full mask blocks no gain 3.57 3.44 c/fwd; 7.4 8.6 Next: the 16K run targets the failing region; verify mode and infilling are usable today first pass
lablup
techseoconnect
inesmontani
marktimemedia
bluesmoon
ammeep
katarinadahlin
baasie
tonyrice
smashingmag
aleyda
minecr
