Routing Agent Messages with a Local Decision Model
My agent waited about 15 s for an 8B LLM to classify every message. I tested local decision models for the job. The best fit, d3-lite, does it in half a second on the CPU, just as accurately.
I built YBM, an agent that runs on my laptop and does things on my computer when I message it on Telegram. Before it acts, it has to classify each message: is it chat, a question about its own progress, or a task, and which of its tools does the task need? There are 15 such routes.
An 8B LLM made that call, and it was slow: about 15 s per message before YBM could even start the work, on a GPU it shares with the model doing that work.
Why a decision model
A decision model is built for this kind of question. It reads a text and a list of options and returns a probability for each option in one pass, without writing any text, so it is fast [1].
I wanted one that runs on my own machine rather than in the cloud, so I served each candidate with LocalDeploy, my local model server, through llama.cpp [2]. I tested 11 open models, from 0.14B to 4.5B parameters, against the LLM YBM was using. I am sharing the results in case they help someone with the same problem.
The test
Each test is one message and one question. For example:
- Message
- "remind me tomorrow at 9am to send the budget draft"
- Question
- Which route should handle this message? 15 options, each with a one-line description, such as "chat", "files and folders", "open a site, web search" and "reminders and recurring jobs".
- d3-lite's answer
- "reminders and recurring jobs", 98% sure, in 0.6 s
I wrote 98 messages like this, covering all 15 routes, and counted how many each model routed correctly. Everything ran on one laptop: a Ryzen 9 5900HS with 8 CPU threads in use, and an RTX 3080 Laptop GPU with 8 GB. The models were quantized to 4 or 8 bits per weight (Q4_K_M or Q8_0), which makes them smaller and faster.
Results
Table 1. Median time to route one message and the share of the 98 messages routed correctly. d3-lite uses the one-line route descriptions YBM ships; with longer ones it was as accurate but twice as slow on the CPU. Data: [5].
| Model | Size | Quantization | CPU time | GPU time | Correct |
|---|---|---|---|---|---|
| Qwen3-VL 8B, the LLM [4] | 8B | Q4_K_M | not run | 15 s* | 92% |
| d3-mini [1] | 4.5B | Q4_K_M | 7.8 s | 0.17 s | 95% |
| Kev-4B | 4B | Q4_K_M | 7.9 s | 4.6 s** | 94% |
| d3-lite [1], my choice | 0.85B | Q4_K_M | 0.53 s | 0.04 s | 93% |
| d3-nano [1] | 2.2B | Q4_K_M | 2.0 s | 0.08 s | 93% |
| lev | 4B | Q4_K_M | 16 s | 0.36 s | 91% |
| Kev-0.8B | 0.8B | Q8_0 | 2.2 s | 0.11 s | 86% |
| d1-3B [3] | 3.1B | Q4_K_M | 2.6 s | 0.09 s | 83% |
| d3-edge [1] | 0.59B | Q8_0 | 0.92 s | 0.03 s | 78% |
| d1-omni-600M [3] | 0.59B | Q8_0 | 1.1 s | 0.03 s | 56% |
| Laya | 0.42B | Q8_0 | 0.59 s | 0.02 s | 28% |
| Julia-1 | 0.14B | Q8_0 | 0.14 s | 0.02 s | 8% |
* The LLM does not fit in 8 GB, so about a third of it ran on the CPU. ** Kev-4B also filled the GPU. The d1 models use LiquidAI's LFM Open License; the others are Apache-2.0.
The point
d3-lite from vLLM Semantic Router routes a message in half a second on the CPU, as accurately as the 8B LLM, and leaves the GPU to the LLM. YBM now asks it first and passes a message to the LLM only when d3-lite is less than 90% sure. In my tests that handled about half the messages without the LLM and lost no accuracy. With a spare GPU, d3-mini is the most accurate at 0.17 s.
Next: the 98 messages, every result, and how to rerun the test.
References
- vLLM Semantic Router, Decision 3.0 models: “d3-lite” model card, Hugging Face, 2026; project repository vllm-project/semantic-router.
- llama.cpp. 2026. “model: support vllm d3 models,” pull request 30383; tested with release b11976.
- Liquid AI. 2026. “d1-3B” model card, Hugging Face (LFM Open License v1.0).
- S. Bai et al. 2025. “Qwen3-VL Technical Report,” arXiv:2511.21631.
- O. Erge, LocalDeploy
benchmarks/decision-routing: the 98 messages, every run's raw results, model-file hashes and scripts.