A Personal Assistant on a 9B Model: Results and Failures
I run YBM, an assistant for my own computer, on a 9B model and an 8 GB laptop GPU. It passed 30 of 32 everyday requests, and 15 of 20 on an exam I sealed before the work, short of my target of 18. This note gives the results, two failures I found, and what I have not measured.
I built YBM so that I can message my own computer from my phone and have it work on the files, web pages, and programs there. It uses a small model, Qwen3.5 9B [1], so the work costs nothing and my files stay on my machine. llama.cpp runs the model on the laptop's RTX 3080 GPU, which has 8 GB of memory.
A small model sometimes misreads a request, and sometimes says it did something it did not do. Most of the code around the model exists to catch that. This note is about how I measure whether it works, and what I found when it did not.
The test
A suite is a fixed list of requests in plain words. Each one is sent to the running assistant through its real Telegram channel, and code checks the outcome: the reply, the files that changed, and the approvals that were asked. Benchmarks for tool-using agents, such as τ-bench [2], also judge the end state of the work and not the wording of the answer.
One suite is a sealed exam. I wrote its 20 requests and froze them (their hash is in the repository) before I did the work they test, and I ran them once, so I could not tune the assistant to their wording. I wrote both the assistant and the suites, so these are my own results on one machine.
Results
Table 1. Requests passed in the last full run of each suite on November 19, 2026. Each suite changed between runs, so a row is a snapshot of one build, not a trend. Data: [7].
| Suite | Requests | Passed | What it asks |
|---|---|---|---|
| Everyday | 32 | 30 | Find files, count the Steam games, read five websites, build a small app |
| Longer requests | 14 | 12 | Plan a pool, retry after a failure, find a device on the network |
| Memory | 8 | 8 | A fact comes back; a fact that changes keeps its history (7 on the run before) |
| Sealed exam | 20 | 15 | Revise a plan twice, retry differently, projects that come back. My target was 18. |
| Browser and desktop | 10 | 5 to 9 | Five runs; the desktop cases need an unlocked screen |
The sealed exam is the result I trust most, because I could not tune the assistant to it, and it fell short. Table 2 lists its five misses. Three have a traced cause and a fix, and one has a cause I have not proven. The fifth was a correct answer in words the sealed list did not accept.
Table 2. The five misses of the sealed exam. The exam is not rerun, because a sealed exam is run once. Data: [7].
| Request | What happened | Status |
|---|---|---|
| Revise a plan twice | The check on the answer's claims took “will be generated” for a finished write, and the task stalled | Fixed |
| “I live in Dallas.” | 3.5 minutes to become a task (see below) | Fixed |
| Find a paper, tell me when it ends | A question about a date started a folder watch (see below) | Fixed |
| Rename files with a script | A code-writing step ran 12 minutes with no model call logged | Cause not proven |
| Retry when notes are missing | The reply said no quote was found, in words the sealed list of accepted phrases did not hold | Counted as a miss, as sealed |
A field the model should have left empty
To make the model's answer parse, llama.cpp can restrict decoding to text that fits a grammar, here a JSON schema [3] [4]. My model server has one slot, so it decodes one request at a time, and a slow request holds up every other call.
The short message “I live in Dallas.” took 3.5 minutes to become a task. The call that keeps facts about me returns a list, and each item has an id field that means the id of a fact already on the list. With an empty list the model had no id to give. It wrote about 1,400 tokens of random letters and digits instead.
Under the grammar the server decoded them at 7 tokens per second. The same model decodes about 45 without one. Every other call waited behind it, and giving up on the call did not help, because the server kept writing. The same call without a grammar took 1.8 s.
The fix was to give every field the model writes under a grammar a length. On the same four messages, the call now takes 2.9 s and 25 tokens, where it took 198 s and 1,366 tokens.
A yes or no that no threshold could separate
A decision model reads a text and a question and returns a probability for each answer in one pass, without writing any text [5]. YBM runs one on the CPU to route messages, which I wrote about in [6].
One skill starts a watch on a folder when the decision model says yes to “is the user asking to be told later when something changes?” The exam's request only asked for a date: “find the paper about the dishwasher's cover and tell me when it ends.” The model said yes at 0.89. It scored real watch requests from 0.73 to 0.92, so no threshold could separate the two.
I reworded the question to say what a watch is, a message each time something new happens, and that a date which can be looked up now is not one. I also added seven near misses to the set I calibrate with. Afterwards the lowest yes was 0.57 and the highest no was 0.41. On a separate probe of 24 requests it caught 10 of 10 watches and took 0 of 14 look-ups for one. Before, it took 1 of 14. The margin is still thin.
What I have not measured
- Other GPUs and other models. The 9B is the only model I measured end to end.
- macOS and Linux against a real model. Continuous integration starts YBM there, but no task has run on a real model.
- Runs longer than a day. A 600-page research run produced a thin report.
- A comparison with any other assistant. I have not run the same requests on one, so I make no claim about how YBM compares.
The point
A 9B model on an 8 GB laptop GPU runs an assistant that passed 30 of 32 everyday requests, and 15 of 20 on a sealed exam. The three failures I traced were not the model misunderstanding a request. They came from what surrounds it: a check that misread a promise as a claim, a field that cost minutes, and a yes or no whose scores overlapped. All three fixes are in the code around the model.
Next: the full results page, with every miss and what I have not measured.
References
- Qwen. 2026. “Qwen3.5-9B” model card, Hugging Face (Apache-2.0).
- S. Yao et al. 2024. “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains,” arXiv:2406.12045.
- llama.cpp. “GBNF grammars,” repository documentation.
- B. T. Willard and R. Louf. 2023. “Efficient Guided Generation for Large Language Models,” arXiv:2307.09702.
- vLLM Semantic Router. 2026. “d3-lite” model card, Hugging Face.
- O. Erge. 2026. “Routing Agent Messages with a Local Decision Model” and LocalDeploy
benchmarks/decision-routing. - O. Erge. 2026. YBM, results page
docs/MEASURED.md, with the suites, the result folders they were read from, and the changelog entries for the fixes.