Setting up my homelab

With most AI tools, every question I type goes to someone else’s server. I wanted my own private AI instead: one that runs at home, reads my own books and notes, and keeps all of it in the house.

I also wanted to understand how it works under the hood. I learn best by building things, usually badly the first few times. So I bought a small machine, put it under my desk, and started.

I named it Outis Mneme, “nobody’s memory”. In the Odyssey, Odysseus tells the Cyclops his name is Nobody, and that is how he gets away. A box that remembers what I read and tells nobody felt like a good fit.

The machine

An ASUS Ascent GX10. It’s a small desktop built on NVIDIA’s GB10 chip, the same one inside the DGX Spark.

What matters for AI is the memory: 128 GB, shared by the CPU and the GPU. On a normal PC, the graphics card has its own small memory, and a big model doesn’t fit there. Here the GPU can use almost all 128 GB, so I can run models that would normally need a server.

It sits under my desk, and I reach it from my Mac over Tailscale, a private network between my own devices.

How I set it up

The stack is simple, and it all runs in Docker:

Open WebUI talks to llama.cpp the same way apps talk to OpenAI’s API. So from the chat page, it feels like any other AI chat. The difference is where it runs.

What I tried

Most of the fun was in the things that didn’t work the first time.

Ollama → llama.cpp. Most people start with Ollama. I almost did too. Then I read Stop Using Ollama and went straight to llama.cpp, the engine underneath it. It makes you choose every setting yourself. Annoying if you just want a chatbot tonight. Perfect if you want to learn.

A 27B model → a 122B model. My first model was Qwen3.6-27B. For chat and summaries it was great. But when I asked it to reason through a problem, it was sometimes wrong and still sounded completely sure. I could only catch it when I already knew the answer. So I moved to a much bigger model.

Will a 122B model even run? I assumed a 122B model would crawl. It doesn’t, because of how this one is built. It’s a “mixture of experts” model: it’s split into many smaller parts, and each word only uses a few of them (about 10B of the 122B). So it only has to read a small part of itself for every word, and it runs at around 27 words per second. That’s faster than I can read.

Default search → better search. Open WebUI’s default search model cut every book page into small fragments. Answers lost context. I switched to nomic-embed-text, which keeps a whole page together.

Four PDF readers. This one ate weeks. The built-in PDF reader broke words apart: “example” became “e xample”. It looks like a typo, but search treats it as a different word. I tried Tika, then Docling, then MinerU, and ended up writing a small router: research papers go to MinerU, everything else to Docling. After the switch, about 28% of what I had indexed turned out to be junk.

How local models felt to use

Honestly, better than I expected.

• Speed is fine. About 27 words per second on the 122B model. It feels like a normal chat app.
• Privacy is real. My questions, my books and my notes never leave the machine.
• Long chats slow down. The model rereads the whole conversation for every new word. At around 30,000 words of conversation, it was about 16% slower.
• Smaller models sound sure even when they’re wrong. The 27B was great for everyday chat, but for reasoning I trust the 122B more.
• Book search works, citations don’t yet. It finds the right book, but it can’t point me to the page. A citation I can’t check isn’t worth much. I’ve parked that for now. I can find the page myself.
• The box is nearly full. About 120 of 128 GB at peak. Big models eat memory fast.

One small thing made me smile: nvidia-smi shows [N/A] for GPU memory. On this machine there is no separate GPU memory to report.

What I want to try next

• Qwen3.8-27B, a new smaller model with vision built in. My guess is that it will be slower than the 122B, even though it is a quarter of the size.
• Free speed that I’m not using. My model file includes extra parts for a speed-up technique (multi-token prediction), and my current setup ignores them.
• Monitoring with Prometheus and Grafana, so I can see speed and memory over time instead of checking by hand.
• Agents: letting the AI do small jobs for me on a schedule, not only answer questions.

I wrote about the Qwen3.8 test in the next post.

If you’re thinking about a homelab like this, or already run one, I’d like to hear what you chose.

References

• Stop Using Ollama, why I went straight to llama.cpp
• Qwen 3.6 is awesome, why I picked my first model
• llama.cpp, Open WebUI
• ASUS Ascent GX10 on ServeTheHome, a good overview of the machine