Skip to all writing
Oney Erge

Writing

Notes on running large models with limited GPU memory.

LLM inference

4 articles

  1. 1

    How to Run a 14B Model on an 8 GB GPU

    6 min read

  2. 2

    How Context Length Affects GPU Memory

    5 min read

  3. 3

    Why Weight Offloading Slows Inference

    5 min read

  4. 4

    How H6.5 Optimizes Weight Placement

    5 min read

Speculative decoding

2 articles

  1. 1

    How Speculative Decoding Speeds Up Inference

    6 min read

  2. 2

    Speculative Decoding: Draft Models, Draft Length, and Trees

    4 min read

Decision models

1 article

  1. 1

    Routing Agent Messages with a Local Decision Model

    3 min read