Skip to article
Oney Erge

Routing Agent Messages with a Local Decision Model

My agent waited about 15 s for an 8B LLM to classify every message. I tested local decision models for the job. The best fit, d3-lite, does it in half a second on the CPU, just as accurately.

I built YBM, an agent that runs on my laptop and does things on my computer when I message it on Telegram. Before it acts, it has to classify each message: is it chat, a question about its own progress, or a task, and which of its tools does the task need? There are 15 such routes.

An 8B LLM made that call, and it was slow: about 15 s per message before YBM could even start the work, on a GPU it shares with the model doing that work.

Why a decision model

A decision model is built for this kind of question. It reads a text and a list of options and returns a probability for each option in one pass, without writing any text, so it is fast [1].

I wanted one that runs on my own machine rather than in the cloud, so I served each candidate with LocalDeploy, my local model server, through llama.cpp [2]. I tested 11 open models, from 0.14B to 4.5B parameters, against the LLM YBM was using. I am sharing the results in case they help someone with the same problem.

The test

Each test is one message and one question. For example:

Message
"remind me tomorrow at 9am to send the budget draft"
Question
Which route should handle this message? 15 options, each with a one-line description, such as "chat", "files and folders", "open a site, web search" and "reminders and recurring jobs".
d3-lite's answer
"reminders and recurring jobs", 98% sure, in 0.6 s

I wrote 98 messages like this, covering all 15 routes, and counted how many each model routed correctly. Everything ran on one laptop: a Ryzen 9 5900HS with 8 CPU threads in use, and an RTX 3080 Laptop GPU with 8 GB. The models were quantized to 4 or 8 bits per weight (Q4_K_M or Q8_0), which makes them smaller and faster.

Results

Table 1. Median time to route one message and the share of the 98 messages routed correctly. d3-lite uses the one-line route descriptions YBM ships; with longer ones it was as accurate but twice as slow on the CPU. Data: [5].

ModelSizeQuantizationCPU timeGPU timeCorrect
Qwen3-VL 8B, the LLM [4]8BQ4_K_Mnot run15 s*92%
d3-mini [1]4.5BQ4_K_M7.8 s0.17 s95%
Kev-4B4BQ4_K_M7.9 s4.6 s**94%
d3-lite [1], my choice0.85BQ4_K_M0.53 s0.04 s93%
d3-nano [1]2.2BQ4_K_M2.0 s0.08 s93%
lev4BQ4_K_M16 s0.36 s91%
Kev-0.8B0.8BQ8_02.2 s0.11 s86%
d1-3B [3]3.1BQ4_K_M2.6 s0.09 s83%
d3-edge [1]0.59BQ8_00.92 s0.03 s78%
d1-omni-600M [3]0.59BQ8_01.1 s0.03 s56%
Laya0.42BQ8_00.59 s0.02 s28%
Julia-10.14BQ8_00.14 s0.02 s8%

* The LLM does not fit in 8 GB, so about a third of it ran on the CPU. ** Kev-4B also filled the GPU. The d1 models use LiquidAI's LFM Open License; the others are Apache-2.0.

Qwen3-VL 8B LLM15 s, 92% correct
d3-mini 4.5B7.8 s, 95%
d1-3B2.6 s, 83%
d3-nano 2.2B2.0 s, 93%
d3-lite 0.85B0.53 s, 93%
Figure 1. Time to route one message: the decision models on 8 CPU threads, the LLM on the GPU (with a third of it on the CPU). d3-lite is as accurate as the LLM and about 30 times faster.

The point

d3-lite from vLLM Semantic Router routes a message in half a second on the CPU, as accurately as the 8B LLM, and leaves the GPU to the LLM. YBM now asks it first and passes a message to the LLM only when d3-lite is less than 90% sure. In my tests that handled about half the messages without the LLM and lost no accuracy. With a spare GPU, d3-mini is the most accurate at 0.17 s.

Next: the 98 messages, every result, and how to rerun the test.

References

  1. vLLM Semantic Router, Decision 3.0 models: “d3-lite” model card, Hugging Face, 2026; project repository vllm-project/semantic-router.
  2. llama.cpp. 2026. “model: support vllm d3 models,” pull request 30383; tested with release b11976.
  3. Liquid AI. 2026. “d1-3B” model card, Hugging Face (LFM Open License v1.0).
  4. S. Bai et al. 2025. “Qwen3-VL Technical Report,” arXiv:2511.21631.
  5. O. Erge, LocalDeploy benchmarks/decision-routing: the 98 messages, every run's raw results, model-file hashes and scripts.