D-Cut: Adaptive Verification Depth Pruning for Speculative Decoding

LLM 推理的瓶颈在于自回归——每步只产一个 token，GPU 大部分时间在等显存。Speculative decoding^[1] 的思路是"先猜后验"：用小模型快速猜多个 token，再让大模型一次验证。The bottleneck of LLM inference is autoregressive decoding — one token per step, with GPU mostly waiting on memory. Speculative decoding^[1] breaks this by "draft then verify": a small model quickly drafts multiple tokens, then the large model verifies them in one pass.

加速效果取决于两件事：猜得准不准（接受率）和验证贵不贵（step time）。Speedup depends on two things: draft accuracy (acceptance rate) and verification cost (step time).

DFlash^[2] 用 diffusion model 做 draft——跳过逐 token 生成，一次并行出 15 个候选。DFlash^[2] uses a diffusion model as the drafter, producing 15 candidates in parallel in one shot.

每步：① Diffusion 并行生成 15 token + 1 bonus（$D=16$）；② Target 一次 forward 验证；③ 取最长接受前缀。低并发下很高效，但 batch size 上去后问题出现了。Each step: ① Diffusion drafts 15 tokens + 1 bonus ($D=16$) in parallel; ② Target verifies in one forward; ③ Accept the longest correct prefix. Works great at low concurrency, but breaks down at high batch sizes.

Target 每步要验证 $N = \text{bs} \times D$ 个 token。但平均只有 $\mathbb{E}[L] \approx 6.3$ 个会被接受——约 60% 的计算是浪费的。高 bs 时浪费更严重：Target verifies $N = \text{bs} \times D$ tokens per step. But only $\mathbb{E}[L] \approx 6.3$ get accepted — ~60% of computation is wasted. At high bs it gets worse:

Qwen3-8B, H20 GPU, TP1.Qwen3-8B, H20 GPU, TP1.

bs=64 时 full verify 比只保留 25% 慢了 2.42 倍。多出来的时间花在了"注定被拒绝"的位置上。At bs=64, full verify is 2.42× slower than keeping only 25%. The extra time is spent on positions that will be rejected anyway.

直接后果：投机解码在高 bs 下退化Consequence: Speculative Decoding Degrades at High bs

Qwen3-8B 上，DFlash-B16 从 bs=16 开始就比 AR 更慢。bs=64 时仅有 AR 吞吐的 42%。On Qwen3-8B, DFlash-B16 becomes slower than AR starting at bs=16. At bs=64 it only achieves 42% of AR throughput.

Draft model 给每个 token 一个 confidence 分数。已有工作^[3]表明，auto-regressive draft model 的 draft confidence 和最终的接受率呈明显正相关。我们发现，这个性质对于 diffusion-based draft model 也成立。定义 prefix product score：The draft model assigns a confidence score to each token. Prior work^[3] has shown that draft confidence of auto-regressive draft models is strongly correlated with final acceptance rate. We find this property also holds for diffusion-based draft models. Define prefix product score:

$$s_{i,k} = \prod_{t=1}^{k-1} c_{i,t}$$

$s_{i,k}$ 估计的是"前 $k$ 个位置全部被接受"的概率。以此为依据，在 batch 内跨请求做全局 top-$K$ 分配：$s_{i,k}$ estimates the probability that all first $k$ positions are accepted. Based on this, we allocate verify budget globally across requests via top-$K$:

不同模型的 verify cost 对 token 数量的敏感度差异巨大：Different models have vastly different sensitivity of verify cost to token count:

Dense 模型的 cost 接近 $O(N)$，裁剪收益大；MoE 的 cost 额外跟激活 expert 数量挂钩，相对变化没有那么激烈，更偏 memory-bound。所以 最优裁剪比例必须根据实际硬件 cost curve 来定。D-Cut 在 server 启动时实测 cost table，用数据而非假设做决策。Dense models have $O(N)$ cost — pruning helps a lot. MoE cost is additionally tied to the number of activated experts, making the relative change less dramatic and more memory-bound. So optimal pruning ratio must be determined by the actual hardware cost curve. D-Cut profiles the cost table at server startup, making decisions from measurements, not assumptions.

形式化Formulation

Batch 中 $bs$ 个请求，每个有 $D$ 个候选位置。请求 $i$ 的有效产出为 $A_i = \min(L_i, n_i)$（$L_i$ 为真实接受长度，$n_i$ 为保留深度）。展开期望：$bs$ requests in a batch, each with $D$ candidate positions. Effective output of request $i$: $A_i = \min(L_i, n_i)$ ($L_i$ = true accept length, $n_i$ = keep depth). Expanding the expectation:

$$\mathbb{E}[\min(L_i, n_i)] = \sum_{k=1}^{n_i} P(L_i \ge k)$$

保留第 $k$ 个位置的边际收益就是 $P(L_i \ge k)$。The marginal benefit of keeping position $k$ is $P(L_i \ge k)$.

Product score

用 draft confidence 的前缀乘积估计这个概率：Estimate this probability using the prefix product of draft confidences:

$$s_{i,k} = \prod_{t=1}^{k-1} c_{i,t}$$

性质：同一请求内单调递减，所以全局 top-K 自动满足前缀约束。Property: monotonically decreasing within each request, so global top-K automatically satisfies prefix constraints.

优化目标Objective

最大化吞吐 = 单位时间有效产出：Maximize throughput = effective output per unit time:

$$U_q = \frac{\sum_{r=1}^{K_q} s_{(r)}}{C_{\text{graph}}(bs, \rho_q)}$$

其中 $K_q = \lceil \rho_q \cdot bs \cdot (D-1) \rceil$。选 $U_q$ 最大的 bucket。where $K_q = \lceil \rho_q \cdot bs \cdot (D-1) \rceil$. Pick the bucket with highest $U_q$.

执行流程Execution Flow

离线（启动时 ~30s）：profiling cost table $C_{\text{graph}}(bs, \rho)$，$\rho \in \{0.25, 0.50, 0.75, 1.00\}$。Offline (~30s at startup): profile cost table $C_{\text{graph}}(bs, \rho)$, $\rho \in \{0.25, 0.50, 0.75, 1.00\}$.

在线（每步）：① Draft 生成 → ② 算 score → ③ 全局排序选 bucket → ④ 按 keep depth 送 target verify。Online (each step): ① Draft → ② Compute scores → ③ Global rank & select bucket → ④ Send kept positions to target verify.

CUDA Graph BucketCUDA Graph Bucket

verify 深度离散化为 4 个 ratio bucket，每个对应一个预 capture 的 CUDA graph shape，避免 graph miss。^*Verify depth is discretized into 4 ratio buckets, each corresponding to a pre-captured CUDA graph shape, avoiding graph misses.^*

^* 当前 D-Cut 实现采用 piecewise CUDA graph，实际上由于大部分 bs 的不同 ratio 存在 overlap，几乎不增加额外 capture 的 CUDA graph。The current D-Cut implementation uses piecewise CUDA graphs. In practice, since different ratios overlap for most batch sizes, this barely increases the number of additionally captured CUDA graphs.

Profiled on H20, vLLM=0.20.2, temperature=0, 仅 report 稳态吞吐 (active_bs ≥ 85%).Profiled on H20, vLLM=0.20.2, temperature=0, only reporting steady-state throughput (active_bs ≥ 85%).

吞吐对比Throughput Comparison

加速比 (vs AR)Speedup (vs AR)

Selector 行为Selector Behavior

vs EAGLE-3 / MTP

D-Cut 在 MoE 上全面超越 MTP，在 dense 8B 上全面超越 EAGLE-3。且无需修改 target model、无需额外训练。D-Cut outperforms MTP on MoE models and EAGLE-3 on dense 8B across all batch sizes. No target model modification needed, no extra training.

References

[1] Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast inference from transformers via speculative decoding. ICML 2023.

[2] Chen, J., Liang, Y., & Liu, Z. (2026). DFlash: Block Diffusion for Flash Speculative Decoding. arXiv:2602.06036.

[3] Li, Y., Wei, F., Zhang, C., & Zhang, H. (2024). EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. EMNLP 2024.