Skip to article
Oney Erge

How H6.5 Optimizes Weight Placement

H6.5 chooses where to keep model weights so the GPU spends less time waiting for them. The weights themselves remain unchanged.

Oney Erge5 min read1 min video
Video overview. I used Claude Code to create this video from the paper. Longer version (3 min).
Transcript

Llama-3.3-70B needs 141 GB of weights. An RTX 5090 has 32 GB.

So the weights wait on the SSD and in RAM, and travel to the GPU, layer by layer.

That trip is slow: read, unpack, copy. When a layer arrives late, the GPU sits idle.

Spare GPU and RAM space can keep some weights close. H6.5 decides which of the 723 tensors get it.

Size is the wrong guide. Take two layers of the same size. Keeping the blue one on the GPU saves nothing. Keeping the orange one saves 2 seconds, because its trip made the GPU wait.

So H6.5 records one real request, replays it for about 1,800 placements, and keeps the fastest.

It placed 198 of 723 tensors differently: 23% less peak GPU memory, and about 13% faster.

Spend scarce GPU memory on the weights that make the GPU wait.

Llama-3.3-70B needs about 141 GB for its BF16 weights. My paper tests how to place them across storage, RAM, and GPU memory on a 32 GB RTX 5090 [1]. The implementation and results are in Afterimage [2].

What H6.5 optimizes

Each weight tensor, such as an attention matrix, can stay compressed or uncompressed on the SSD, unpacked in RAM, or unpacked on the GPU. H6.5 searches for a placement that reduces request time within fixed memory budgets. The model still computes its layers in order.

The main experiment allows 8 GB of GPU memory, including 5.2 GB for persistent weights, and 16 GB of RAM. With 723 tensors, testing every possible placement live is impractical.

Why timing matters

A data-movement rule favors tensors that save the most transfer bytes per byte of memory. Offloading systems such as FlexGen and Accelerate also consider memory capacity and data movement [3] [4]. H6.5 considers when those transfers occur: loading that overlaps computation may add no delay, while a late load makes the GPU wait (Figure 1).

As placed Blue kept on the GPU Orange kept on the GPU 1 2 blue 4 GPU waits 2 s orange 6 8 s 1 2 blue 4 orange 6 8 s, no change 1 2 blue 4 orange 6 6 s, 2 s sooner 02 s4 s6 s8 s
blue: in RAMorange: on the SSDGPU waitsAs placed8 s1246Blue kept on the GPU8 s, no change1246Orange kept on the GPU6 s, 2 s sooner124602 s4 s6 s8 s
Figure 1. An illustrative example with equal-size layers. Keeping the blue layer on the GPU saves no time because its loading overlaps computation. Keeping the orange layer removes a 2 s wait.

In a recorded Llama request, the data-movement placement left the 2.1 GB output head compressed on the SSD. Reading and unpacking it after the last decoder layer made the GPU wait 1.56 s (Figure 2).

GPU, data-movement rule Output head GPU, head kept (replay) last layer done GPU idle 1.56 s done read 1.25 s unpack 0.31 s done, 1.56 s earlier −2 s−1 s0+1 s+2 s
GPU, data-movement ruleidle 1.56 sOutput headread 1.25 sunpack 0.31 slast layer doneGPU, head kept (replay)done, 1.56 s earlier−2 s−1 s0+1 s+2 s
Figure 2. End of a recorded Llama-3.3-70B request on the RTX 5090 [1]. Dark marks show GPU computation. The lower timeline predicts the effect of keeping the head on the GPU; it is a replay, not a separate measurement.

H6.5 evaluates whole placements rather than always pinning the output head. In laptop tests, it kept the head in RAM and rearranged attention and feed-forward tensors instead.

How the search works

H6.5 records reads, unpacking, copies, and computation in real requests, including which events must finish before others can start. It then estimates alternative placements by substituting measured preparation times and replaying those dependencies:

cj=dj+maxk∈pred(j)ck,T^=maxjcj (1)

Each event finishes at cj: its duration dj plus the latest finish among its predecessors, pred(j). The last event determines the predicted request time T̂.

The search improves several feasible starting placements step by step, using a replay score that penalizes unstable predictions. The selected placement is frozen before live tests on new requests.

Preparing this Llama placement took about 9 minutes, including 3.4 s to score 1,817 placements. This is a setup cost, not an extra cost per request. Preparation is repeated when changes to the model, hardware, workload, or budget require a new placement.

What the placement changed

H6.5 changed the placement of 198 of 723 tensors, freeing GPU space for the output head (Figure 3). Both plans are predicted to read about 78 GB from storage per pass; their timing differs.

Data-movement rule
334 tensors, 5.10 GB
H6.5
output head, 2.10 GB143 tensors, 3.02 GB
Placed differently
198 of 723
Moved from GPU to SSD
191
Of those, uncompressed
82
Figure 3. GPU-resident Llama weights under the same budgets. Counts come from the frozen placement files [2].

Measured results

Table 1 compares H6.5 with the data-movement rule (traffic control) and a rule ranking each tensor by preparation time saved per byte of memory (greedy control).

Table 1. H6.5 against two placement rules, using one-token requests with the same engine and budgets [1].

Compared withModel, GPUPeak extra GPU memoryRequest time
Data-movement rule, 20 roundsLlama 70B, RTX 509023% lower, in all 2013% lower
Time-saving rule, 6 roundsLlama 70B, RTX 509020% lower16% lower
Data-movement rule, 8 roundsQwen3-14B, laptop13% lower4% lower
Data-movement rule, 8 roundsGemma 2 27B, laptop12% lower11% lower

Peak extra GPU memory is measured above pre-run use. Request time includes prompt processing. Rounds use matched requests in random method order, preserving the released weights. Qwen's latency difference was within run-to-run variation.

Keeping the output head ready avoids its temporary GPU allocation, helping explain the lower peak memory. Across three repeat studies on new prompts (35 rounds), pooled request time was about 18% lower than traffic and 15% lower than greedy. Memory was lower in every round; latency gains varied by session.

Limits

The evaluation covers two machines, mainly with one-token requests. Replay estimates guide the search; Table 1 reports live measurements.

In an exploratory 32-token Qwen3-14B test, the one-token placement was slower than streaming all weights from the SSD. A follow-up placement with no host-resident weights was faster on new prompts, consistent with preserving more operating-system file cache. Replay does not model this cache.

Next: how speculative decoding reduces target-model passes.

References

  1. O. Erge. 2026. “Schedule-Aware Exact Weight Placement for Large Language Models with Limited GPU Memory.” Submitted to arXiv (cs.PF), October 2026. Forthcoming preprint; the link will be added when it is announced.
  2. O. Erge, Afterimage. Paper evidence, frozen placements, and the verifier in evidence/h65-paper1.
  3. Y. Sheng et al. 2023. “FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU.” ICML 2023.
  4. Hugging Face. “Big Model Inference.” Accelerate documentation.