How to Run a 14B Model on an 8 GB GPU
Qwen3-14B has 29.5 GB of released BF16 weights. This article maps the GPU-memory budget and explains why loading the model one part at a time makes it runnable on an 8 GB GPU.
Qwen3-14B is a 14B-parameter model in the Qwen3 family [1]. Its released BF16 weights need 29.5 GB, while my NVIDIA RTX 3080 Laptop GPU has 8 GB. Loading the whole model at once is therefore impossible.
One option is smaller, approximate weights. The other, examined here, is to keep the released weights outside the GPU and load each part only when the model needs it.
GPU memory during inference
Everything using the GPU competes for the same memory. For this discussion, I divide that memory into five parts:
- Msys
- System and display. Memory already in use before the model starts. My laptop used 2.2 GB with Windows, a browser, and a code editor open.
- Mrt
- GPU software. Memory used by the driver and libraries such as PyTorch.
- MW
- Weights. Fixed by the model and its number format. Calculated below.
- MKV
- Stored keys and values. Memory kept for every token of the input prompt and of the generated answer. It grows as the conversation grows. Explained below.
- Mact
- Activations. Temporary results inside the layer that is being computed, released when the layer finishes.
MGPU is the memory on the graphics card. GPU makers label memory in binary units, so my "8 GB" card holds 8.6 GB in the decimal units this post uses throughout. I still call it an "8 GB GPU" in prose, but every chart and calculation below uses 8.6 GB.
That leaves about 6.4 GB before model weights, the KV cache, runtime allocations, and activations. Only weights and the KV cache can be calculated from the model; the other terms must be measured on the machine.
Number formats for weights
The weights are by far the largest fixed term in Eq. (1), so the first question is how much memory one weight requires. The memory needed for the weights follows directly from how many there are and how many bytes each one takes:
where N is the number of weights and b is the number of bytes each weight uses.
Quantization uses fewer bits per weight, lowering b in Eq. (2). It also changes the released values and can reduce accuracy.
Table 1. Common number formats for language model weights. The size under each format is the storage for one weight; 1 byte is 8 bits.
| Format | Storage per weight | Typical use |
|---|---|---|
| FP32 | 4 bytes | Traditional training and reference calculations |
| FP16 | 2 bytes | Inference on GPUs, mixed-precision training |
| BF16 | 2 bytes | Training and release of recent open models |
| 8-bit integer | About 1 byte | Quantized inference |
| 4-bit integer | About 0.5 byte | Local inference on consumer GPUs |
BF16 is the released format for Qwen3-14B [2]. With 14.8 billion weights at 2 bytes each, it needs 29.5 GB. Eight-bit and 4-bit weights would need about 14.8 GB and 7.4 GB before format metadata.
This series keeps BF16. Here, exact weights means inference ultimately uses the released BF16 values, not an 8-bit or 4-bit approximation. Matching output tokens is a separate measured result.
Parts of a language model
A token enters through the embedding table, passes through 40 decoder layers in order, and leaves through the output head. Each decoder layer contains about 330M weights; the output head contains about 778M.
This sequence is the key: the GPU needs the current part, not all 29.5 GB at the same instant.
Table 2 applies Eq. (2) to the main parts.
Table 2. Weights of Qwen3-14B by part, in BF16. Rows may not sum exactly because of rounding.
| Part | What it does | Weights | Memory |
|---|---|---|---|
| Embedding table | Looks up the numerical representation for each token | 778M | 1.6 GB |
| One decoder layer | Mixes information between tokens and transforms each token | 330M | 0.66 GB |
| All 40 decoder layers | 13.2B | 26.4 GB | |
| Output head | Scores every token as the possible next token | 778M | 1.6 GB |
| Total | 14.8B | 29.5 GB |
Each individual model component fits on the GPU. The problem is keeping all 40 decoder layers resident together: they require about 26.4 GB.
- Weights
- 29.5 GB
- GPU memory
- 8.6 GB
- Layers that fit
- 10 of 40
The model is larger than the GPU, but it uses its major weight groups one after another.
Running one part at a time
Streaming keeps weights outside GPU memory until they are needed. During decoding, the engine loads the embedding table, then each decoder layer in order, then the output head. It releases a part after use and repeats the sequence for the next output token.
This trades memory for data movement. The GPU holds the current weights, the KV cache, and temporary work rather than the whole model. The next articles quantify the context-memory cost and the latency of fetching weights from storage.
References
- Qwen Team. 2025. “Qwen3: Think Deeper, Act Faster.” April 29, 2025. Qwen3 Technical Report. Model card and configuration: Qwen/Qwen3-14B.
- Number formats and quantization: D. Kalamkar et al. 2019, “A Study of BFLOAT16 for Deep Learning Training”; T. Dettmers et al. 2022, “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale,” NeurIPS 2022; T. Dettmers and L. Zettlemoyer. 2023, “The Case for 4-bit Precision: k-bit Inference Scaling Laws,” ICML 2023.
- Hardware specifications: GeForce RTX 3080 Mobile, 8 GB GDDR6, 448 GB/s memory bandwidth; PCIe 4.0 x16, about 32 GB/s per direction.
- O. Erge, Afterimage; measured tools and benchmark details in
docs/HOW_IT_WORKS.md.