How to Calculate Local LLM VRAM: Formulas for Weights, KV Cache, and Quantization

Running large language models locally often presents a frustrating challenge: developers spend hours downloading tens of gigabytes of model weights, only to face an immediate Out-Of-Memory (OOM) error upon execution. This typically happens because many users calculate only the static size of the model file on disk. However, true hardware planning requires accounting for dynamic memory allocations like Key-Value (KV) cache scaling, activation overhead, and CUDA context allocation.

To determine exact GPU VRAM requirements and avoid these crashes, one must evaluate three distinct memory pools:

$$\text{Total VRAM (GB)} = \text{Model Weights Memory} + \text{KV Cache Memory} + \text{Activation & Context Overhead}$$

---

1. Calculating Model Weight Memory

The baseline VRAM requirement is dictated by the parameter count ($P$) and the quantization precision ($b$, measured in bits per weight). Unquantized models run at 16-bit floating point ($\text{FP16}$), requiring roughly two bytes per parameter. To run on consumer hardware, quantized formats like $\text{GGUF}$, $\text{EXL2}$, and $\text{AWQ}$ compress these weights down to 8, 4, or even 2 bits per weight.

The baseline formula for a model’s weight memory footprint is:

$$\text{Memory}_{\text{weights}} (\text{GB}) = \frac{P \times b}{8 \times 10^9}$$

Where:

• $P$ = Total parameter count (e.g., $8 \times 10^9$ for an 8B model).
• $b$ = Average bits per weight (e.g., $16$ for $\text{FP16}$, $4.8$ for $\text{Q4_K_M}$, $8.5$ for $\text{Q8_0}$).

Quantization Overhead Factor

It is important to remember that quantized files carry metadata overhead for block quantization scales and tensor headers. Because of this, the effective bits per weight usually sit slightly above the nominal quantization tier:

• $\text{FP16}$ (16-bit): $2.00 \text{ bytes/param}$
• $\text{Q8_0}$ (8-bit): $\approx 1.08 \text{ bytes/param}$
• $\text{Q4_K_M}$ (4-bit Medium): $\approx 0.58 \text{ bytes/param}$
• $\text{EXL2}$ (4.5 bit): $\approx 0.56 \text{ bytes/param}$

---

2. Context Window VRAM Usage & K/V Cache Overhead

The primary cause of unexpected CUDA OOM crashes is Context Window VRAM usage. As the context length ($L_{\text{ctx}}$) expands, the Key-Value (KV) cache must store past token representations in GPU memory. This prevents the system from having to recompute attention at every single inference step, but it eats up VRAM quickly.

Grouped Query Attention (GQA) Formula

Modern architectures (such as Llama 3 and Mistral) utilize a highly efficient technique called Grouped Query Attention (GQA). GQA reduces KV cache memory usage by grouping key and value heads, saving significant memory during long conversations:

$$\text{KV Cache Memory (GQA)} = 2 \times n_{\text{layers}} \times \left( \frac{n_{\text{kv_heads}}}{n_{\text{heads}}} \times d_{\text{model}} \right) \times L_{\text{ctx}} \times S_{\text{bytes}}$$

Where:

• $n_{\text{layers}}$ = Number of transformer layers.
• $d_{\text{model}}$ = Hidden dimension size.
• $L_{\text{ctx}}$ = Active context window length in tokens.
• $S_{\text{bytes}}$ = Precision bytes ($2 \text{ bytes}$ for $\text{FP16}$, $1 \text{ byte}$ for $\text{Q8}$, $0.5 \text{ bytes}$ for $\text{Q4}$).

---

3. Quantization Memory Comparison

Understanding how these formulas translate to real hardware is much easier with a reference table. Below is a breakdown of common VRAM targets for 8B and 70B models across different quantizations:

Model Size	Quantization Type	Nominal Bits	Weights VRAM	8k Context Cache	32k Context Cache	Total Recommended VRAM
8B Parameters	$\text{FP16}$	16.0	$16.0 \text{ GB}$	$1.0 \text{ GB}$	$4.0 \text{ GB}$	24 GB (RTX 3090 / 4090)
8B Parameters	$\text{Q8_0}$	8.5	$8.5 \text{ GB}$	$1.0 \text{ GB}$	$4.0 \text{ GB}$	16 GB (RTX 4060 Ti 16GB)
8B Parameters	$\text{Q4_K_M}$	4.8	$4.8 \text{ GB}$	$1.0 \text{ GB}$	$4.0 \text{ GB}$	12 GB (RTX 3060 / 4070)
70B Parameters	$\text{Q8_0}$	8.5	$74.0 \text{ GB}$	$5.0 \text{ GB}$	$20.0 \text{ GB}$	96 GB (4x RTX 3090 / Mac Studio)
70B Parameters	$\text{Q4_K_M}$	4.8	$42.0 \text{ GB}$	$5.0 \text{ GB}$	$20.0 \text{ GB}$	64 GB (2x RTX 3090 / 4090)

---

4. GPU VRAM vs System RAM: CPU Offload Bottlenecks

When total memory requirements exceed physical GPU VRAM, local runners like llama.cpp allow users to offload layers ($n_{\text{gpu_layers}}$) to their System RAM via the CPU.

While CPU offloading is a great fallback to prevent CUDA OOM crashes, it introduces severe memory bandwidth bottlenecks that users should expect:

• NVIDIA RTX 4090 Memory Bandwidth: $\approx 1008 \text{ GB/s}$
• Dual-Channel DDR5 System RAM Bandwidth: $\approx 64 \text{ to } 80 \text{ GB/s}$

Because of this gap, offloading even 20% of transformer layers to system RAM drops generation speeds drastically—from a fluid 60+ tokens per second down to 8–12 tokens per second—due to PCIe bus transfers and system memory latency.

---

5. Automated Hardware Sizing

Instead of calculating layer math manually for every single deployment, developers can evaluate their hardware configurations directly using interactive tools:

• The Local LLM VRAM Calculator allows users to model exact KV cache scaling, quantization precision overhead, and hardware limits before downloading any model weights.
• The LLM Hardware Benchmark Tool helps analyze PCIe bandwidth bottlenecks, CPU layer offload performance, and predicted tokens-per-second throughput across standard consumer GPUs.