Skip to article
Oney Erge

How to Run a 14B Model on an 8 GB GPU

Qwen3-14B has 29.5 GB of released BF16 weights. This article maps the GPU-memory budget and explains why loading the model one part at a time makes it runnable on an 8 GB GPU.

Qwen3-14B is a 14B-parameter model in the Qwen3 family [1]. Its released BF16 weights need 29.5 GB, while my NVIDIA RTX 3080 Laptop GPU has 8 GB. Loading the whole model at once is therefore impossible.

One option is smaller, approximate weights. The other, examined here, is to keep the released weights outside the GPU and load each part only when the model needs it.

GPU memory during inference

Everything using the GPU competes for the same memory. For this discussion, I divide that memory into five parts:

Msys+Mrt+MW+MKV+Mact≤MGPU (1)
Msys
System and display. Memory already in use before the model starts. My laptop used 2.2 GB with Windows, a browser, and a code editor open.
Mrt
GPU software. Memory used by the driver and libraries such as PyTorch.
MW
Weights. Fixed by the model and its number format. Calculated below.
MKV
Stored keys and values. Memory kept for every token of the input prompt and of the generated answer. It grows as the conversation grows. Explained below.
Mact
Activations. Temporary results inside the layer that is being computed, released when the layer finishes.

MGPU is the memory on the graphics card. GPU makers label memory in binary units, so my "8 GB" card holds 8.6 GB in the decimal units this post uses throughout. I still call it an "8 GB GPU" in prose, but every chart and calculation below uses 8.6 GB.

That leaves about 6.4 GB before model weights, the KV cache, runtime allocations, and activations. Only weights and the KV cache can be calculated from the model; the other terms must be measured on the machine.

Number formats for weights

The weights are by far the largest fixed term in Eq. (1), so the first question is how much memory one weight requires. The memory needed for the weights follows directly from how many there are and how many bytes each one takes:

MW=N×b (2)

where N is the number of weights and b is the number of bytes each weight uses.

Quantization uses fewer bits per weight, lowering b in Eq. (2). It also changes the released values and can reduce accuracy.

Table 1. Common number formats for language model weights. The size under each format is the storage for one weight; 1 byte is 8 bits.

FormatStorage per weightTypical use
FP324 bytesTraditional training and reference calculations
FP162 bytesInference on GPUs, mixed-precision training
BF162 bytesTraining and release of recent open models
8-bit integerAbout 1 byteQuantized inference
4-bit integerAbout 0.5 byteLocal inference on consumer GPUs

BF16 is the released format for Qwen3-14B [2]. With 14.8 billion weights at 2 bytes each, it needs 29.5 GB. Eight-bit and 4-bit weights would need about 14.8 GB and 7.4 GB before format metadata.

This series keeps BF16. Here, exact weights means inference ultimately uses the released BF16 values, not an 8-bit or 4-bit approximation. Matching output tokens is a separate measured result.

Parts of a language model

A token enters through the embedding table, passes through 40 decoder layers in order, and leaves through the output head. Each decoder layer contains about 330M weights; the output head contains about 778M.

This sequence is the key: the GPU needs the current part, not all 29.5 GB at the same instant.

Table 2 applies Eq. (2) to the main parts.

Table 2. Weights of Qwen3-14B by part, in BF16. Rows may not sum exactly because of rounding.

PartWhat it doesWeightsMemory
Embedding tableLooks up the numerical representation for each token778M1.6 GB
One decoder layerMixes information between tokens and transforms each token330M0.66 GB
All 40 decoder layers13.2B26.4 GB
Output headScores every token as the possible next token778M1.6 GB
Total14.8B29.5 GB

Each individual model component fits on the GPU. The problem is keeping all 40 decoder layers resident together: they require about 26.4 GB.

Embedding table and output head, 1.6 GB each Decoder layer, 0.66 GB each
Weights
29.5 GB
GPU memory
8.6 GB
Layers that fit
10 of 40
Figure 1. Qwen3-14B weights drawn to scale, in the order a token passes through them. The vertical line marks the memory of my 8.6 GB GPU. The embedding table and 10 of the 40 decoder layers fit if only weights are considered. With the 2.2 GB already used by the system on my laptop, only about 7 decoder layers would fit.

The model is larger than the GPU, but it uses its major weight groups one after another.

Running one part at a time

Streaming keeps weights outside GPU memory until they are needed. During decoding, the engine loads the embedding table, then each decoder layer in order, then the output head. It releases a part after use and repeats the sequence for the next output token.

This trades memory for data movement. The GPU holds the current weights, the KV cache, and temporary work rather than the whole model. The next articles quantify the context-memory cost and the latency of fetching weights from storage.

References

  1. Qwen Team. 2025. “Qwen3: Think Deeper, Act Faster.” April 29, 2025. Qwen3 Technical Report. Model card and configuration: Qwen/Qwen3-14B.
  2. Number formats and quantization: D. Kalamkar et al. 2019, “A Study of BFLOAT16 for Deep Learning Training”; T. Dettmers et al. 2022, “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale,” NeurIPS 2022; T. Dettmers and L. Zettlemoyer. 2023, “The Case for 4-bit Precision: k-bit Inference Scaling Laws,” ICML 2023.
  3. Hardware specifications: GeForce RTX 3080 Mobile, 8 GB GDDR6, 448 GB/s memory bandwidth; PCIe 4.0 x16, about 32 GB/s per direction.
  4. O. Erge, Afterimage; measured tools and benchmark details in docs/HOW_IT_WORKS.md.