Hardware benchmarks
Tokens-per-second, memory bandwidth, and quantization quality loss, measured on real hardware.
VRAM calculator
Estimate total VRAM for a given model size, quantization, and context length. Estimates only — see the note at the bottom.
- Model weights
- —
- KV cache
- —
- Runtime overhead (10%)
- —
- Total VRAM
- —
Fits on:
This won't fit on local hardware
Based on the estimate above, this configuration exceeds every local GPU profile listed. Renting cloud GPU capacity is the practical option for a workload this size.
RunPod
On-demand GPU pods, billed by the minute.
Lambda Labs
Reserved and on-demand cloud GPU instances.
Disclosure: We may earn a commission from cloud providers if you spin up an instance through these links, at no extra cost to you. This never affects the VRAM math above — see /methodology.
Live changes
- No changes yet — adjust a control above.
Generic estimate: weight size
is bits-per-weight × parameter count; KV cache uses a reference
architecture for the selected size bucket (7B/13B/70B: real published
models; 120B: extrapolated, no real model exists at that exact
size). Model-specific (HF-synced):
weight size is the exact .gguf file size fetched from
Hugging Face — no estimation. KV cache is still estimated:
this schema doesn't carry layer count, head count, or head dimension,
so the calculator parses an approximate parameter count from the
model's title (e.g. "8B") and reuses the nearest size bucket's
reference architecture for the KV math, same caveats as the generic
mode. If no parameter count can be parsed from the title, it falls
back to the 7B architecture and says so next to the quantization
dropdown. Batch size is fixed at 1 in both modes. Treat all figures
here as a starting estimate, not a guarantee. Full formula and known
limitations: /methodology.
Calculator FAQ
How accurate is the VRAM calculator?
It depends on the mode. In "Generic estimate" mode, both the model weight size and the KV cache figure are estimates based on a reference architecture for the selected size bucket. In "model-specific" mode, the weight size is exact — pulled directly from the real .gguf file size synced from Hugging Face — but the KV cache figure is still an estimate, for the reasons below.
Why is KV cache always an estimate, even for synced models?
KV cache size depends on a model’s layer count, attention head count, and head dimension. The Hugging Face data this calculator syncs (exact .gguf file sizes) does not include those architecture details, so the calculator matches each model to the nearest reference architecture bucket by parsing an approximate parameter count from its name. See /methodology for the exact formula and its limitations.
Does more VRAM always mean I can run a bigger model?
No. VRAM is consumed by three separate things: model weights, KV cache, and runtime overhead. A long context window can consume more VRAM than the model weights themselves, especially on older MHA-style architectures. Sizing for weights alone is the most common mistake people make.
What does "Fits on" actually check?
It compares the calculator’s total estimated VRAM figure against the stated VRAM of a few common consumer GPU configurations. It does not account for VRAM already in use by your OS, other applications, or a second model already loaded.
RTX 4090 24GB
24GB VRAM handles 70B models at Q4 quantization with room for an 8K context window.
View benchmarkRTX 4060 Ti 16GB
Best price-to-VRAM ratio for 8B-13B models at full precision quantization.
View benchmarkRyzen 9 7950X
Usable CPU-only fallback for 7B models when GPU offload isn't available.
View benchmarkDDR5-6000 Dual Channel
Minimum viable bandwidth for hybrid CPU/GPU offload setups without heavy bottlenecking.
View benchmarkGGUF Q4_K_M
The sweet spot for most consumer GPUs — minimal quality loss, roughly 4x size reduction.
View benchmarkAWQ 4-bit
Faster inference than GGUF at similar bit-width, at a small extra quality cost.
View benchmark