What this paper is really about

The core problem: speculative decoding is only as good as its drafter.

In speculative decoding, you run a low-cost draft model to propose tokens, then a larger target model verifies those tokens in parallel abstract. When you move from token-by-token draft models to parallel or semi-autoregressive (semi-AR) drafters that propose whole blocks in one forward pass, training gets tricky.

The difficulty is *global*: the draft distribution for a given position depends on where the decoding round starts, and where rounds start depends on how many tokens earlier rounds accepted abstract. Existing training objectives mostly use block-local surrogates that ignore this cross-round coupling, so they don’t actually optimize for “how many rounds does it take to finish the sequence?” abstract.

Yunxiao Zhao and Changxiao Cai propose a way to hit that target directly:

  • Represent speculative decoding as a Markov reward process (MRP) abstract.
  • From that, derive an Expected Decoding Rounds (EDR) objective that is exactly equal to the expected number of decoding rounds, with no extra hyperparameters abstract.
  • Derive an exact temporal-difference gradient that you can estimate from target-model rollouts, unbiased abstract.
  • Use the same framework to build an offline evaluator that estimates round counts from shared target rollouts, without actually running speculative decoding abstract.
  • Show that finetuning two state-of-the-art drafters, DSpark and DFly, with EDR improves mean accepted length and outperforms existing training objectives on nine math, code, and chat benchmarks abstract.

For anyone running agentic systems at scale, this is about training the drafting component so you actually get wall-clock speedups, not just nice-looking per-block metrics.

---

How speculative decoding with parallel drafters behaves

The paper assumes the standard speculative decoding pattern:

  • A draft model is low-cost and proposes tokens.
  • A target model is larger and more accurate, and it verifies those proposals in parallel abstract.

For parallel / semi-AR drafters, one forward pass proposes an entire block of tokens abstract. Verification then accepts some prefix of that block; anything after the first mismatch gets rejected.

The twist the paper focuses on:

  • The draft distribution at position i is not fixed.
  • It depends on where the current decoding round starts.
  • That starting point depends on how many tokens previous rounds got accepted abstract.

So each call to the drafter runs in a different “state of the conversation” defined by the current position in the sequence and how previous blocks were accepted or rejected. That coupling across rounds is the core modeling problem.

Most prior training approaches ignore this: they optimize block-local surrogates that assume a fixed starting point and don’t model the stochastic process over rounds abstract. This can give you higher “tokens per accepted block” in isolation but doesn’t necessarily minimize the *total* number of target-model calls.

---

The Markov reward view: turning decoding into an MRP

Zhao & Cai’s first move is to view speculative decoding as a Markov reward process abstract.

At a high level, a Markov reward process has:

  • States: here, these correspond to decoding situations—e.g., how many tokens are already committed and how far into the speculated block you are. The paper states that they “represent speculative decoding as a Markov reward process” but does not enumerate the exact state tuple abstract.
  • Transition dynamics: determined by the drafter’s proposal distribution and the target model’s verification; transitions encode “how many tokens get accepted this round and where the next round starts.”
  • Rewards: here, these are tied to rejection costs at each position abstract.

The payoff of this abstraction is that it lets them define a global objective:

The Expected Decoding Rounds (EDR) objective “weights local rejection costs by state occupancies and exactly equals the expected number of decoding rounds” abstract.

Concretely, “state occupancy” here is the probability of being in a given decoding state (a particular starting point for a round) under the joint drafter–target dynamics. Multiplying local costs by these occupancies and summing is standard Markov-reward math; the novelty is applying it to speculative decoding and choosing rejection costs so that the total expected reward equals the expected number of decoding rounds.

Two practical consequences documented in the abstract:

  • The EDR objective directly corresponds to the engineer’s real metric: “how many decoding rounds (i.e., target-model verifications) do I expect to need?” abstract.
  • The objective “introduces no auxiliary hyperparameters” abstract. That is, they don’t need to hand-tune block-local weights or tradeoffs between different kinds of errors; you just minimize expected rounds.

For a system builder, this is a critical property: you can try different draft architectures or scales and compare them on a single scalar that describes end-to-end verification cost, without having to guess how some surrogate loss will translate into speed.

---

Training with an exact temporal-difference gradient

Casting decoding as an MRP is only half the story. The other half is: how do you get gradients to train the drafter?

The paper claims:

“We then derive an exact temporal-difference gradient that supports unbiased stochastic optimization from target-model rollouts” abstract.

Key parts in that sentence:

  • Temporal-difference (TD) gradient: They’re borrowing techniques from temporal-difference learning, where you bootstrap value estimates from successor states. The abstract does not provide the explicit formula, but it states that this TD-based gradient is exact for their EDR objective abstract.
  • Unbiased stochastic optimization: When you estimate this gradient from samples—here, “target-model rollouts”—the estimate is unbiased with respect to the true gradient of EDR abstract.
  • From target-model rollouts: You collect trajectories of what happens when the target model verifies drafted tokens and then use those trajectories to compute gradient updates abstract.

For implementation, what matters is that you don’t need extra simulators or approximations: you run the target model, record what it accepts and rejects across rounds, and those logs are sufficient to construct unbiased gradient estimates for the drafter under the EDR objective abstract.

This is importantly different from many heuristic drafter-training pipelines that:

  • Use only local cross-entropy or KL losses on block predictions.
  • Don’t account for how mispredictions change the distribution over future starting positions.

Here, the gradient is defined on the process, not just per-block snapshots.

---

Offline evaluation without running spec decode

One of the most practical pieces in the abstract is the offline evaluator:

“The same framework also yields an exact offline evaluator for round counts, enabling paired drafter comparisons on shared target rollouts without running speculative decoding” abstract.

Two benefits:

  1. Paired comparisons: You can evaluate two different drafters using the *same* set of target-model rollouts. That reduces variance and makes A/B decisions easier abstract.
  2. No speculative decoding required: You don’t have to actually run the full speculative decoding loop for every candidate. You can reuse existing target-model logs and apply the evaluator to estimate expected decoding rounds abstract.

Operationally, this means you could:

  • Run your target model in a baseline mode, log token-level behavior.
  • For each candidate drafter, apply the offline evaluator to those logs to estimate how many rounds it would have needed under speculative decoding.

The abstract calls this evaluator exact for round counts abstract, so within the assumptions of their MRP model, you’re not approximating the metric.

For an engineering team, this opens the door to things like:

  • Rapid iteration on drafter architectures without wiring each into the live speculative decoding pipeline.
  • Regression tests where you ensure new drafter versions don’t regress in expected rounds against a fixed set of logs.

The abstract, however, does not specify the exact statistics required from the rollouts or the computational overhead of this evaluator.

---

What they actually measured: DSpark and DFly

The authors don’t stop at theory. They evaluate on two state-of-the-art speculative drafters:

  • DSpark
  • DFly

They report that finetuning both with EDR:

  • “Consistently improves mean accepted length.”
  • “Outperforms existing training objectives across nine benchmarks spanning math reasoning, code generation, and chat” abstract.

From this we can extract a couple of practical signals:

  • Mean accepted length is another way of quantifying speculative decoding efficiency: how many tokens from the draft model tend to survive target verification. The abstract states that EDR training improves this for both DSpark and DFly abstract.
  • The benchmarks cover three important workload classes—“math reasoning, code generation, and chat” abstract—which aligns well with typical agentic tasks.
  • EDR doesn’t just match other objectives; it “outperforms” them on these benchmarks abstract. The abstract does not give numerical margins or per-task breakdowns.

What’s missing from the abstract is how this translates to wall-clock speedups or GPU utilization: they focus on expected decoding rounds and accepted length, rather than specific latency or throughput numbers.

---

How it fits into an agentic stack

Even though the paper is framed around speculative decoding, the implications are squarely about end-to-end system performance:

  • Any agent harness that uses a large target model can front it with a speculative drafter to cut the number of target forward passes. The abstract defines speculative decoding exactly in these terms: “a low-cost draft model to propose tokens that the full-size target model verifies in parallel” abstract.
  • Parallel / semi-AR drafters are especially valuable because they “improve drafting efficiency by proposing an entire block in a single forward pass” abstract. EDR gives you a principled way to train those block drafters so they’re optimized for *global* efficiency rather than local metrics.

In an agentic system that chains multiple calls to a large model (plans, tool calls, reflection, etc.), shaving target decode rounds on each turn compounds across the whole workflow.

The offline evaluator matters for production as well:

  • You can maintain a library of candidate drafters and continuously evaluate them against production target logs using their exact offline evaluator for round counts abstract.
  • This creates a feedback loop where every real-world interaction becomes training/evaluation data for better drafters, without disturbing the serving path.

---

Why this matters now

The timing is about where inference bottlenecks are shifting:

  • As models get larger and multi-step agents become the norm, inference costs are dominated by expensive target-model decoding.
  • Speculative decoding is one of the few algorithmic levers that can improve per-token cost without changing the target model.

But large gains only come if the drafter is well-trained. The abstract points out that prior work uses “block-local surrogates that ignore this cross-round coupling, and therefore do not directly optimize the global decoding efficiency” abstract.

By:

  • Modeling the full process as an MRP,
  • Defining EDR that exactly matches the target metric,
  • Providing an exact TD gradient from target rollouts, and
  • Giving an exact offline round-count evaluator,

this work closes the loop between what you care about in production (rounds, accepted tokens) and what you train on.

The empirical result that EDR finetuning improves mean accepted length and outperforms previous objectives on nine realistic benchmarks with DSpark and DFly abstract makes it more than just a theoretical curiosity.

---

What is not documented

Based on the abstract and metadata alone, the following are not established in the source:

  • The exact mathematical form of the EDR objective, including how local rejection costs are defined.
  • The explicit definition of the Markov states, transition probabilities, and reward function used in their Markov reward process.
  • The derivation details or closed-form expression of the “exact temporal-difference gradient.”
  • The precise statistics required from target-model rollouts to compute the gradient and offline evaluator, and their computational cost.
  • Any quantitative speedups in wall-clock latency, throughput, or GPU FLOP utilization resulting from EDR training.
  • The architectures, sizes, or training details of DSpark and DFly beyond being “state-of-the-art drafters.”
  • Numerical benchmark results, per-task breakdowns, or comparisons against specific baseline objectives.
  • Integration recipes or implementation details for deploying EDR-trained drafters in specific serving frameworks or libraries.