How Context Length Affects GPU Memory

The KV cache stores information from every prompt and generated token. For Qwen3-14B, it grows from 1.3 GB at 8K tokens to 5.4 GB at 32K tokens.

Oney ErgeOctober 14, 20265 min read

The KV cache

LLMs generate one token at a time. Attention uses information from the earlier tokens, so inference software stores the keys and values it has already computed instead of recomputing them at every step.

A closer look at attention Optional: a worked example of queries, keys, and values

Queries, keys, values, and attention are fundamental pieces of the transformer architecture [1].

In each layer, attention computes three short lists of numbers for every token: a query, a key, and a value. The query of the newest token is compared with the keys of the tokens available to it. A stronger match gives a higher score, which causes more of that token's value to contribute to the output.

Table 1 works through this for the text “the dog ran.” It scores each available token against the newest one, “ran,” converts those scores into weights, and blends the values into one output for “ran.”

For this simplified example, I omit the usual score scaling and the learned projections that produce queries, keys, and values, and show only the part needed to explain the KV cache.

Table 1. A simple attention step for the newest token, “ran,” whose query is [0, 2]. The numbers are illustrative.

Token	Key	Value	Score, query × key	Weight	Contribution, weight × value
the	[1, 0]	[0.2, 0.1]	0	0.06	[0.01, 0.01]
dog	[0, 1]	[0.9, 0.4]	2	0.47	[0.42, 0.19]
ran	[1, 1]	[0.5, 0.7]	2	0.47	[0.24, 0.33]
Blended output for “ran”					[0.67, 0.52]

Each score comes from multiplying the query by that token's key, number by number, and adding the results. Softmax then converts the scores into weights that add up to 1. The output is the sum of every token's value multiplied by its weight:

w_{i} = \frac{e^{s_{i}}}{\sum_{j} e^{s_{j}}}, output = \sum_{i} w_{i} v_{i}

(1)

where s_i is the score of token i, w_i is its weight, and v_i is its value.

Storing keys and values instead of recomputing them

When the model adds a new token, the keys and values of earlier tokens do not change. The KV cache keeps them in GPU memory, while each new token adds one key and value in every decoder layer.

Prompt token Answer token Computed in this step

Figure 1. KV cache of Qwen3-14B while it answers the input prompt “the dog ran.” Each cell holds one key and one value for every set of keys and values in that layer. Only the newest column is computed; the others are reused. Rows for layers 3 to 39 are not drawn.

Size of the KV cache

Qwen3-14B uses grouped-query attention: 40 query heads share 8 sets of keys and values [2]. The cache size is

M_{KV} = 2 \times L \times n_{kv} \times d_{h} \times b \times T

(2)

The factor 2 counts keys and values; L is the number of layers; n_kv is the number of key/value sets; d_h is the size of each one; b is bytes per value; and T is prompt plus generated tokens.

For Qwen3-14B in BF16, this is

2 \times 40 \times 8 \times 128 \times 2 = 163,840 bytes

or about 0.16 MB per token: 1.3 GB at 8K tokens, 5.4 GB at 32K, and 21 GB at 128K.

Figure 2. KV cache of Qwen3-14B as the prompt and answer grow, calculated with Eq. (2) for BF16. The dashed line marks the memory of my 8.6 GB GPU.

Total memory for Qwen3-14B on an 8 GB laptop GPU

Qwen3-14B needs 29.5 GB for BF16 weights. A 3K-token prompt and answer add about 0.5 GB of KV cache, and my laptop already uses 2.2 GB. That exceeds 32 GB before GPU software and temporary activations.

Even 4-bit weights do not by themselves make the workload fit: the 7.4 GB estimate plus system use leaves no room for the cache, runtime allocations, and format metadata. The calculator below makes those trade-offs visible.

Memory check for Qwen3-14B

8.6 GB laptop GPU

Weight format BF16 8-bit 4-bit

Include system use (2.2 GB)

Input prompt tokens 2K

Answer tokens 1K

System
Weights
KV cache (input prompt + answer)
Total

Weights come from M_W = N × b, and the KV cache from Eq. (2). GPU-software memory and activations are not included and must be measured separately. Quantized files also store scale factors and other metadata.

What the cache changes

The KV cache is not a one-time model cost: it grows with every prompt and output token. On a small GPU, it competes directly with the memory available for weights and temporary work.

Streaming lets the model's weights fit by loading one part at a time. Next: the latency cost of fetching those weights.

References

A. Vaswani et al. 2017. “Attention Is All You Need.” NeurIPS 2017.
J. Ainslie et al. 2023. “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.” EMNLP 2023.