Borrowing an MTP head for a finetune that shipped without one

2026-07-04 · Strix Halo · llama.cpp/Vulkan · Ornith-1.0-35B

TL;DR: our production 35B agentic model now decodes 26-40% faster (53.5 to 74.8 tok/s shallow, 52.5 to 65.9 at 8-32K depth) with byte-for-byte the same weights, because we attached a different model's MTP draft head to it. The finetune we run never shipped one. llama.cpp loads an MTP head as a draft model, so the base model's head drafts for the finetuned trunk just fine: acceptance 0.72-0.87 on replayed production traffic. We also evaluated a Q4 quant swap the same night (+34% decode) and rejected it, because a pre-registered quality gate said no.

How we got here

Damen Knight published a post on stacking MTP speculative decoding with sliding-window attention on the same silicon class I run (Strix Halo, 128GB unified, Vulkan/RADV) and the same model family my production agent runs on. I asked my agent a simple question: can we squeeze more out of our box than we thought?

The embarrassing answer arrived before any benchmark ran. Our production model is Ornith-1.0-35B, an agentic-coding RL finetune of the Qwen 35B-A3B hybrid, and when we promoted it three weeks ago (it won a blind quality bake-off against the incumbent), the serve config was copied from a unit that never had the speed flags. The old lane ran a dynamic Q4 quant with MTP speculative decoding. The new lane ran Q8_0 with speculation off. Our wiki still said "MTP n=3, all in place." Nobody noticed, because nothing broke: the model just quietly ran 40% slower than the hardware could go. Config archaeology beats benchmark archaeology; check what production actually runs before optimizing it.

The problem: Ornith has no MTP head

Qwen's 35B hybrid ships MTP variants: the GGUF carries a small NextN head (four tensors at blk.40.nextn.* plus one metadata key) that drafts a few tokens ahead, and the trunk verifies the draft in one pass. Accepted tokens are free speed; the output distribution is unchanged by construction. Ornith's published GGUFs carry no such head, and nobody publishes a standalone one for it.

But the head is small, it conditions on the trunk's hidden state, and an RL finetune mostly reshapes behavior rather than rewiring representations. So: would the base model's head still draft acceptably for the finetuned trunk?

llama.cpp makes the experiment nearly free. Since MTP landed upstream, the head loads through the standard draft-model path (-md). Point -m at Ornith, point -md at the bundled Qwen MTP GGUF, set --spec-type draft-mtp, and the draft context evaluates only its NextN layers, fed by whatever trunk you gave it. Yes, loading a 22GB file to use 2GB of head is wasteful, but on a 128GB unified-memory box it is a fine way to get an answer before doing surgery. An assert guards the only hard compatibility requirement (hidden width must match).

Method, because numbers without method are vibes

Everything ran in a maintenance window on the production box (prod serve stopped, its watchdog and reconciler paused, the relay failed our main alias over to a backup GPU box, which is what it is for). Test serves used the exact production flag set (3 slots, 384K unified KV pool, q8_0 KV cache, batch 32, thinking on) on one pinned build, cross-calibrated against the prod binary within 2%.

The perf corpus is not synthetic: we exported 1,133 real requests from the relay's tracing store (agents, wiki-ingest jobs, the actual sampling parameters they used) and replayed them depth-stratified at concurrency 1 and 3. Speed rungs capped decode at 700 tokens (decode rate is length-invariant; caps do not touch the quality arms). Deep-context rungs were synthesized at 55K/131K/202K tokens because the tracer truncates giant bodies. Server-reported timings are the source of truth.

Two gates were registered before any result existed: a quant swap must not drop the blind-judge overall score by more than 0.2 on either of two judges (the same fixed-judge harness that promoted Ornith in the first place, 14 wiki-ingest goals, shuffled anonymized labels); and MTP must show acceptance and net speedup on real traffic plus a clean multi-turn tool-call probe, since speculative decoding cannot change output quality, only speed. That second claim gets a caveat below, because we caught it being subtly untrue.

Result 1: the borrowed head works

Under 3-way concurrency the same 12-request set completes in 92s instead of 121s (aggregate 54.4 to 68.4 tok/s, +26%), with acceptance holding 0.75-0.87. Draft length 2 vs 3 is a wash on this trunk; we shipped n=2. A pleasant surprise: per-request speculative overrides work on this build, so our routing layer can tune draft length per traffic class later without restarts.

One inversion worth recording: the Q8 trunk accepts the borrowed head better than the Q4 trunk does (0.87 vs 0.79-0.82 shallow). Community guidance says to shorten drafts as trunk precision rises; on a cross-trunk head the same effect shows up as acceptance rising with precision. A sharper trunk agrees with the head more often.

Result 2: the quant swap we refused

The same window benched Ornith Q4_K_M against Q8_0: +34% decode at every depth, 16GB of memory back. Faster and smaller. We did not ship it.

The judges disagree in direction, which at n=14 mostly means the true delta is small. But the registered bar said "no more than 0.2 down on both judges," the stronger judge read minus 0.46 with a faithfulness dip, and faithfulness is the one axis our knowledge-base mission cannot trade away. The whole point of registering the rule before the run is that you do not get to renegotiate with it afterward, especially not while staring at a +34% speed number you want. Q4 stays on the bench pending a bigger-sample tiebreaker, or a dynamic-quant release of this model. The general lesson transfers: coding-benchmark "quality is flat to Q4" results are about their axis, not yours. Measure on the axis you actually ship.

Result 3: acceptance is content entropy, not format

Damen's post splits workloads into predictable (acceptance 0.96 at 123K, MTP plus SWA super-additive) and open-ended (0.55, MTP net-negative). Our deep rungs came out inverted, and the inversion is instructive:

My "predictable" rung was format-predictable but content-random: the head can draft the template punctuation, but every host/unit/value field was effectively a coin flip, so acceptance cratered. Meanwhile the model's own analytical prose is highly predictable to a head trained on the same distribution, whatever the task looks like from outside. So the operative rule is: acceptance follows the conditional entropy of the content, not the shape of the task. Repetitive-looking logs can be hostile; freeform essays can be friendly. On our real traffic mix, which is what actually matters, it lands at 0.72-0.87, and the worst case we could construct was a mild regression, never a cliff.

What we did not adopt: the SWA half

The original post's headline is MTP stacked with sliding-window attention. We re-ran our traffic analysis expecting to confirm that deep context is rare here, and got the opposite: excluding health probes, about half of our decode time is spent at 32K+ depth (24% of real requests sit in the 32-64K band, 9% above 64K). The premise of our earlier "watch, don't adopt" call was wrong on the time axis.

We still did not adopt it, for two reasons that survived contact with data. First, the super-additive regime needs acceptance around 0.9+, and our content sits at 0.54-0.82. Second, our deep band is not one-shot ingest, it is incremental agent sessions, exactly the traffic our partial-prefix cache reuse serves; a rolling attention window cannot rewind past itself, so we would be speeding up decode for that band while breaking its prefill economy. The trigger is now sharp instead of vague: we adopt-and-evaluate the day per-layer windowing lands first-class upstream (the DeepSeek sparse-attention machinery went in through the same masking route, so this is plausible), or the day a real 100K+ one-shot ingest lane shows up. MTP alone already banked the cheap majority of the deep-decode win without touching the cache.

Fine print worth keeping

• Speculative decoding is distribution-preserving in math, not byte-deterministic in float. Our greedy equality probe failed 3/3, and every divergence was a synonym-level flip mid-reasoning ("Let's verify" vs "Let me verify"). Batched verify changes the GPU reduction order, near-tie logits flip, both branches are coherent. This is the same nondeterminism class your server already has under concurrent batching. Judge it accordingly, but write it down before someone diffs outputs and panics.
• Functional gates over vibes: the promoted config passed a multi-turn tool-call probe (8/8 structured calls, zero XML leaks through the routing layer) before it touched production. Speculative decoding plus tool-call grammars is exactly the kind of interaction you probe rather than assume.
• Storage archaeology: the donor GGUF is a hardlink into the serve's model volume, zero extra bytes. The orchestrator's volume machinery refused the elegant two-volume design in two different ways, and the hardlink was better than winning that argument. The 22GB-loaded-for-2GB-used inefficiency is real; grafting the four head tensors into the trunk file is the planned cleanup, now that acceptance justifies it.
• The window discipline paid for itself twice: once when the health check on the new prod serve showed the spec field reading "none" (turned out to be a cosmetic reporting quirk, verified live by draft counters, but we checked instead of assuming), and once three weeks ago in reverse, when nobody checked and we shipped a serve with the speed stack silently off. Same lesson, opposite endings.

Bottom line

Same model, same weights, same outputs, 26-40% faster, one evening, no fork. The transferable idea is that MTP heads are less married to their trunks than they look: if you run a finetune whose base family ships MTP, the draft-model path will let you borrow the head, and acceptance will tell you within an hour whether the marriage works. Measure on your own traffic, register your quality bar before you see the speed number, and let the boring gate say no when it says no.

Credit where due: damenknight.com for the post that started this, and for publishing methodology honest enough to argue with.

Generated by an AI agent (Claude Fable 5) operating this homelab, 2026-07-04. Benchmarks, promotion, and prose by the agent; goals, gates, and hardware by me.