← Back

Building Nugget: a local-first AI assistant on a Raspberry Pi

Published: 2026-09-04

As local LLMs get smaller and better, I've wanted to see if I could make an assistant that runs in my house. I have a fairly powerful computer with a bunch of threads and two 24GB GPUs. It's not server hardware, but it acts like a server. I also have a handful of Raspberry Pis. The one I used for this was running Home Assistant, but I never really used it.

I called the Pi Nugget. The larger computer is called Sombra. The idea is simple: every request starts on Nugget. Small requests stay there. Clear complex requests can get handed to Sombra automatically.

When IFM released K2 Horizon 0.9B, I immediately wanted to try it on a Pi. It ran. Slowly. It managed around six tokens per second across the prompt and answer, with generation closer to five. Fast enough to prove the idea was possible, but too slow for something I would actually talk to.

I also tried adding Uno and using DiffuGPT as a drafter. Neither got close to the improvement I needed. For a while I thought about building a mesh where one Pi handled tokens, another handled drafting, and another generated the answer. The numbers killed that idea pretty quickly. Generating an answer is one long chain of dependent steps, so passing it between Pis adds network delays in the middle of the slowest part.

Extra Pis may still be useful later for speech, search, monitoring, or handling separate requests. For now, one Pi owns the whole local response.

The small model that made it work

I swapped K2 for LFM2.5-350M-QAD. It is much smaller, so it runs a lot faster. It also knows less. That was when it clicked: Nugget does not need to know everything if it can recognize what it should handle and what should go somewhere else.

Nugget should answer small things quickly, use a few tools, and send everything else to Sombra.

That 20.976 number is generous because it counts the prompt too. Actual output was closer to 13.8 tokens per second in a later greeting, and five requests is nowhere near a stress test. Still, it was fast enough that the idea stopped feeling silly.

What I actually wanted from tool calling

At first I thought Nugget needed to be great at tool calling. The tests made me realize I wanted something more specific. It should be able to search the web for a quick answer, fetch one of my documents, check something like the weather, and send anything complicated to Sombra.

It is similar to the newer Siri flow where a small request stays with Siri and a larger one can move to ChatGPT. I wanted the same idea without having to approve the handoff every time.

I ran model-only tool tests on Sombra using fake web and note tools. LFM could format those calls correctly. The problem was knowing when to give up. In four new complex requests, it sent zero to Sombra unless I named Sombra myself.

Sometimes it refused the request. Sometimes it searched only the first part. If I asked “what is that?”, it could invent a search instead of asking what I meant. The model knew how to call a tool, but it was not good at deciding what it should do.

Putting a harness around the model

So I stopped letting the 350M model run the whole show. I put a small Python app in front of it. A few obvious rules run first, a tiny classifier handles the fuzzy middle, and the app decides what actually gets called. The classifier can make around 1,400 decisions per second on the Pi, so this adds basically no noticeable delay.

The router has five choices:

• Reply: let the local model answer.
• Web: look up one public fact.
• Notes: fetch one of my documents.
• Clarify: ask what I meant instead of guessing.
• Sombra: send the full request to the larger model.

Local replies, clarification, and Sombra work now. The web and notes routes exist, but the actual integrations are still next on the list.

The app also keeps my original words. When web and notes are connected, the app will pass my request instead of letting the small model rewrite it. LFM had already dropped places, dates, and the subject of follow-up questions, so I stopped giving it the chance. If Nugget is not sure what “what about tomorrow?” refers to, it asks instead of guessing.

Making Sombra automatic

Forcing a request to Sombra worked early on, but that was not the assistant I wanted. I did not want to say “use Sombra” every time a question got difficult. Nugget now catches clear planning, coding, comparison, and multi-step requests before LFM gets a vote. It checks that Sombra is online, then forwards my original request.

There is still an Always use Sombra switch for testing. With it turned off, the automatic route runs. The page shows whether Nugget or Sombra actually answered, along with the route and response time. That made it much easier to spot bad decisions instead of guessing which model I was talking to.

Testing the router

A few hand-picked examples would have made this look better than it was, so I tested it on 500 requests it had not seen. The first version scored 329 out of 500. That test exposed a bug in how follow-up context was handled. I fixed the bug, froze the new version, and ran another set of 500 new requests once.

Going from 65.8% to 76.6% was a decent improvement. It also sounds better than it is. Eleven requests stayed local when they should have gone to Sombra, and too many easy requests went the other direction. The router is useful for testing, but I do not completely trust it yet.

Where it is now

Nugget is actually running on the Raspberry Pi now. It serves a small chat page over my private Tailscale network, so I can open it from my computer without an SSH tunnel. A normal greeting takes around 1.5 to 1.7 seconds on the Pi. A harder request can move to Sombra automatically.

The page always says what answered: the Nugget model, a quick reply from the router, or Sombra. That came from a funny bug. I told it “Holy cow, that was fast” and got “Thanks” back in 0.00 seconds. It was not impossibly fast inference. It was a canned reply. Now the page says that.

The current code passes 124 tests both on my computer and on Nugget. Web search is next because I want it to answer quick Google-style questions, not just open a results page. After that I want short-term memory in the browser, read-only Obsidian access, and voice input and output.

The part I like most is that the tiny model does not have to pretend it can do everything. Nugget can be fast for the small stuff, then get out of the way when Sombra is a better fit. If the rest works, I can finally add the ridiculous “powered by the sun AI” part and sort of mean it.