Skip to article
Oney Erge

Why Weight Offloading Slows Inference

Offloading lets a 29.5 GB model run below 2 GB of GPU memory, but fetching weights from storage took 27 to 33 s per generated token in this measured workload.

Oney Erge5 min read

Every generated token passes through every decoder layer. Any weight not already close to the GPU must be read, prepared, and copied again for that pass.

The path from storage to the GPU

A stored weight is read from the SSD into RAM, unpacked if compressed, and copied over PCIe into GPU memory. The GPU computes from that final copy.

  1. SSDModel files
  2. Read2.3 GB/smeasured, cold
  3. RAMMain memory
  4. Unpack, copy over PCIe≤ 32 GB/sPCIe specification
  5. GPU memory448 GB/s to the GPU's processor
Figure 1. The path of a streamed weight on my laptop. The SSD speed was measured by reading a stored model file after clearing the operating system's page cache [7]. The PCIe and GPU-memory speeds are specifications [1].

The operating system can keep recently read files in unused RAM, its page cache. A repeated read reached 16.6 GB/s on my laptop, about 7 times the cold read [7]. The measurements here clear that cache before each run.

Three steps for each weight

Before the GPU can use a weight, the weight is read from the SSD, unpacked if it is stored compressed, and copied to GPU memory. The time until it is ready is roughly

tprep≈Sdiskbdisk+tunpack+SbPCIe (1)

Sdisk is stored size, bdisk is SSD read speed, tunpack is unpacking time, S is unpacked size, and bPCIe is copy speed.

On this laptop, reading a 0.66 GB decoder layer cold takes about 0.29 s; copying it over PCIe takes about 0.02 s. Reading the whole 29.5 GB model takes about 13 s, versus under 1 s for copying. Storage reads dominate this configuration.

Lossless compression: fewer bytes to read, more work to unpack

Afterimage stores the weights compressed without loss [7], using shorter codes for common BF16 exponent values [2]. It restores the released bits before computation.

For Qwen3-14B, storage falls from 29.5 GB to 20.3 GB, saving about 4 s of reading at 2.3 GB/s.

The saving is not free. Every compressed weight must be unpacked before use, which adds tunpack to Eq. (1). Compression shortens the time to a ready weight only when

S−Sdiskbdisk>tunpack (2)

that is, when the reading time it saves is longer than the unpacking time it adds.

At the minimum-memory point, it was not enough: Afterimage took 33 s per token against AirLLM's 27 s because unpacking offset its read-time saving.

Loading during computation

A streaming engine can prefetch: while the GPU computes one layer, the next layer is already loading [4]. Loading is free only when it finishes before the GPU needs the weight; otherwise the GPU waits.

GPU Loading layer 1 layer 2 layer 3 GPU waits layer 4 load 2 load 3 load 4 hiddenhiddennot ready in time time
GPUlayer 1layer 2layer 3layer 4GPU waitsLoadingload 2load 3load 4hiddenhiddennot ready in timetime
Figure 2. Loading while computing, simplified. Each layer is loaded while the GPU computes the layer before it. Loading that finishes in time is hidden; loading that does not makes the GPU wait. The durations are illustrative.

Two equally large weights can therefore have very different latency costs. What matters is whether their loading lies on the request's critical path.

Keeping some weights close

GPU memory can keep some weights resident; RAM can keep others unpacked. Both avoid some of the work above. Table 1 shows the measured trade-off on my laptop.

Table 1. Measured operating points for Qwen3-14B BF16 on the RTX 3080 Laptop GPU: 4 prompts, 4 generated tokens each, greedy decoding, cold page cache. Every method produced the same output tokens. Source: Afterimage results [7].

MethodMeasured PyTorch peakRequest time per generated token
AirLLM 3.2.0, one layer at a time [6]1.6 GB27 s
Afterimage minimum-memory1.7 GB33 s
Afterimage 4 GB plan, weights kept on the GPU3.9 GB17 s
Hugging Face Accelerate, weights on GPU, RAM, and SSD [5]3.8 GB14 s

With about 4 GB, the measured requests take 14 to 17 s per token rather than 27 to 33 s. The added memory holds weights that no longer make the trip.

The next question is which weights deserve that scarce space. A rule based only on bytes moved can still keep the wrong ones.

What to optimize

Storage reads dominated this cold-cache run: about 13 s for the model versus under 1 s for copying. The loading that arrives late, rather than the most bytes, determines the stall.

Next: how H6.5 chooses which weights should stay close to the GPU.

References

  1. Hardware specifications: GeForce RTX 3080 Mobile, 8 GB GDDR6, 448 GB/s memory bandwidth; PCIe 4.0 x16, about 32 GB/s per direction.
  2. D. A. Huffman. 1952. “A Method for the Construction of Minimum-Redundancy Codes.” Proceedings of the IRE 40 (9).
  3. T. Zhang et al. 2025. “70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11).” NeurIPS 2025.
  4. Y. Sheng et al. 2023. “FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU.” ICML 2023.
  5. Hugging Face. “Big Model Inference.” Accelerate documentation.
  6. AirLLM 3.2.0.
  7. O. Erge, Afterimage. Measured operating points in the README and docs/HOW_IT_WORKS.md; the cold and warm read measurements in docs/REPRODUCE.md.