Day one with a new model: Meta's Muse Glimmer on an AMD iGPU, hours after release

Meta released Muse Glimmer 30B yesterday morning. llama.cpp merged support for it the same day. By that evening it was serving on an AMD Strix Halo integrated GPU in my lab, a configuration nobody upstream had tested. This is the whole path, including the parts that went wrong, because the interesting question is not how fast the model is. It is how quickly a fleet can absorb a model that did not exist that morning.

The gate: a new architecture string

The first thing to check on a day-one model is not throughput, it is whether your runtime can load it at all. Reading the GGUF header answered that:

general.architecture = muse-glimmer

A new string. Any llama.cpp older than the commit that added it fails at load with an unknown-architecture error, and nothing about that error tells you the fix is a rebuild. Support landed in 62bf73d2, merged that day.

The next question was whether Vulkan needed a port, because that would have meant days rather than hours. It did not, and the diff says so plainly: the commit touches 22 files and not one backend kernel. No ggml-vulkan, no ggml-cuda, no Metal. It is graph construction, architecture registration and the multimodal path, built on operations that already existed. The GGUF metadata is Gemma-shaped, with sliding window attention and logit softcapping, both long supported.

So the AMD box needed a rebuild, not a patch. We pin llama.cpp by tag and commit in our own runtime image, so that was a two-line change, and CI published a candidate in eight minutes.

I added a guard in the same change, because a pin that silently regresses is worse than one that fails:

RUN grep -rqa muse-glimmer /out/ \
    || { echo "ERROR: muse-glimmer arch not present; pin predates 62bf73d2"; exit 1; }

A build that cannot serve the model now fails in CI instead of on a node at 2am. There is an identical guard for the previous model that taught us this lesson.

Three files, one Model

Muse Glimmer is not one artifact. It ships weights, a vision projector, and its own draft model for speculative decoding:

Those went into MinIO once, then staged to the node over LAN rather than pulling 18.4 GiB from Hugging Face per box. One Model resource declares the set:

spec:
  source: s3://models/meta-models/Muse-Glimmer-30B-GGUF
  files:
    - muse-glimmer-30B-kquant-17gb.gguf
    - dflash-kquant.gguf
  mmproj: mmproj-kquant.gguf
  hardware:
    accelerator: vulkan

The operator wires --mmproj itself, and the server confirms both halves loaded. That is the bring-up: rebuild, preload, declare, serve.

What it does on a Strix Halo

Q4-class quant, one integrated GPU, measured through llama.cpp's own server timings rather than wall clock.

Prefill sits around 330 tok/s and speculation does not change it, which is expected: drafting only affects generation.

The speedup collapses with context, and the server log says exactly why:

draft acceptance = 0.199 (47 accepted / 232 generated), mean len = 2.00

About 20% of drafted tokens survive at 4k, against roughly 70% at 500. Decode tracks that almost exactly. Any single "speculative decoding gives you Nx" claim is really a statement about the context depth it was measured at, which is worth asking whenever you see one, including about this post.

One asymmetry worth carrying away: at 500 tokens the speedup varied 19% between runs, and interleaving the measurements did not reduce it. Acceptance depends on how predictable a particular prompt's continuation is, so short-context speculation is genuinely variable. At 4k it was stable to 0.2%. "About 3x, and variable" is the honest phrasing.

Two traps that produce confident wrong numbers

Both of these were flagged publicly by Tom Turney within hours of the release, and both cost me time before I read his posts.

Pointing at the draft model is not enough. Passing --model-draft dflash-kquant.gguf on its own loads the drafter, logs a line that reads like success, and does not speculate. You also need --spec-type draft-dflash. The line that actually discriminates is:

common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash'
  - n_max=5, n_min=0, block_size=16

"loading draft model" is not that line. This failure is silent in the worst way: the service is healthy, responses are correct, and a speculation-on versus speculation-off comparison shows no difference. The natural conclusion is that speculative decoding is worthless on your hardware, when in fact it never ran.

The reasoning knob in the model card loses to the one in the template. The chat template defaults reasoning_strength to high and writes Reasoning strength: <value>. into the system prompt after your own system text, so the template's line wins. The control that works is --chat-template-kwargs '{"reasoning_strength":"low"}'.

I verified that behaviourally rather than trusting the flag, because "the setting had no effect" and "the setting never applied" look identical from the outside. Same prompt, same answer of 391, 84 generated tokens at high against 42 at low. Exactly double, and every number I measured before that check was silently at high.

The measurement I got wrong first

My first sweep measured each configuration in a block: all repetitions of A, then all of B. It produced a clean, consistent finding that reasoning strength changed prefill throughput by 9%, in the same direction at both depths.

Reasoning strength cannot affect prefill. It changes how much the model generates, not how fast it reads a prompt. So the result was either a discovery or an artifact, and the honest way to tell is to change the design rather than argue about it.

I re-ran with the arms interleaved, the order rotated each round, and GPU temperature sampled around every measurement. The 9% effect vanished. The temperature trace shows why:

pre[T=29C F=600MHz]  post[T=38C F=2029MHz]

The GPU climbs from 29C at 600MHz to the low 40s at around 2000MHz inside a single measurement. Whichever arm runs first is measured on a cold, downclocked chip. A blocked design confounds "which configuration" with "when it ran", and prefill on this box carries about 5% run-to-run noise, which is larger than the effect I had been about to publish.

Decode is the opposite: 0.03 tok/s of variation across runs. That asymmetry is the reusable lesson. On these boxes, decode conclusions are solid and prefill conclusions need error bars, and shuffling your runs hides a confounder where measuring it explains one.

Two silent bugs it exposed

Serving a model the operator had never seen turned up two bugs of the same species: configuration accepted, silently ignored, symptom appearing somewhere unrelated.

Node placement was dropped for non-GPU services. Vulkan requests devic.es/dri-render rather than a GPU resource, so it took the non-GPU path, where spec.nodeSelector and spec.tolerations were applied inside a if gpuCount > 0 branch. The field was accepted, stored, visible in kubectl get isvc -o yaml, and did nothing. The pod then failed to schedule with a message describing nodes I had never asked for. Fixed, with tests pinning both directions: a non-GPU service gets its constraints, and does not get the GPU-only taint toleration.

Speculative decoding cannot name a draft model. The CRD accepts a speculation type but has no field for the draft weights, and its type list predates draft-dflash entirely. Admission rejects the combination rather than serving without speculation, which is the right failure, but it means a model shipping its own drafter needs raw argument passthrough today. Tracked, with the measurements above as the argument for closing it.

What this does not cover

One accelerator. The same model on our GB10 boxes is blocked on an aarch64 CUDA image we do not yet build ourselves, which is precisely the dependency this post argues against: we control the AMD runtime and could rebuild it in minutes, and we do not control the other one. That is the next piece of work.

This is also a reasoning model, so a share of those decode tokens are thinking the user never sees. The tokens-per-second figures are real; a useful-output-per-second comparison is a different measurement. And an occasional sample evaluated a handful of tokens instead of the full prompt. The harness discards those now, which is not the same as understanding them, and I would rather say so than quietly average them in.

The point

A model shipped in the morning and was serving on unusual hardware that evening, through a declarative path that will be identical for the next one: pin the runtime and guard the architecture, preload the artifacts once, declare the file set, serve. The numbers in this post will age. The path is the part worth keeping.

LLMKube is Apache 2.0. If you run a Strix Halo or an Apple box, upstream speculative decoding work needs Vulkan and Metal testing, and that is a genuinely under-covered gap.