Switching to a smaller model

In my last post I set up my homelab with a big 122B model. It ran at about 27 words per second, and I was happy with it.

Then Qwen3.8-27B came out. It’s a quarter of the size, and it can read images. I almost didn’t try it, because I was sure it would be slower.

Why I expected it to be slower

The 122B model I run is a “mixture of experts”. It’s split into many parts, and each word only uses a small slice of it. The new 27B is a normal model: it uses all of itself for every word.

On my machine, speed mostly depends on how much of the model has to be read for each word. The big model reads about 7 GB per word. The small one reads about 20 GB. So the small model should be about three times slower.

I tested it first, so I’d have something to compare against:

Just as expected. A smaller model, and much slower.

The setting that changed everything

Qwen3.8 comes with a tiny extra model, about 1.4 GB, called a draft head. It uses a technique called multi-token prediction.

It works like this. The tiny model quickly guesses the next three words. The big model then checks all three guesses at once, and checking three words costs it about the same as writing one. Every correct guess is a free word.

Think of a junior developer writing a first draft and a senior reviewing it. If the draft is mostly right, the senior approves several lines in one pass instead of writing each one.

In llama.cpp, turning this on is one setting. Same machine, same prompts:

Code got 2.7 times faster, prose 2 times faster. The small model was now as fast as the big one on code.

Why code got faster than prose

Without the draft head, the speed was the same for every prompt. With it, the speed depends on what I ask for.

The reason is how easy the next words are to guess. Code is very predictable. After def, self. or an opening bracket, there are only a few likely next words, so the tiny model guesses right most of the time. English has many more reasonable next words, so more guesses miss.

llama.cpp reports this for every request:

• Code: about 78% of guesses were right
• Prose: about 51% of guesses were right

That gap is the whole speed difference.

Was it worth switching?

On speed alone, it’s about even. Faster for code, slower for prose.

I switched for the last two rows. 56 GB of memory came back. Before, the chat model used so much memory that my PDF tools were fighting it for space. Now they have room. And the new model can look at an image or a page screenshot directly.

What I learned

1. Check if your model has a draft head. Without that one setting, I would have been running a model 2.5 times slower than it should be, and it would have looked like it was working fine. It starts, it answers, it’s just quietly slow.
2. My rule of thumb was right, but incomplete. “Speed depends on how much is read per word” assumed one read gives one word. The draft head breaks that assumption.
3. Pin your versions. While doing this, I found that a llama.cpp setting had been renamed between versions. Now I pin the exact version, so an update can’t quietly break my setup.

Still open

• Is a higher-quality version of the model file (Q6 or Q8) noticeably better for my work? I have the memory for it now, but I haven’t tested it.
• If the model can read a page image directly, do I still need my PDF extraction tools? My guess is yes for long PDFs and tables. I haven’t measured it yet.

If you run local models, check whether yours has a draft head. For me it was one setting, and it doubled the speed.

References

• Better & Faster Large Language Models via Multi-token Prediction, the paper behind the draft head
• Qwen3.8-27B and Qwen3.5-122B-A10B model cards
• llama.cpp