How much VRAM do you actually need to run a local LLM?
Heads up: some links in this article are affiliate links. If you buy through them I may earn a small commission at no extra cost to you. As an Amazon Associate I earn from qualifying purchases. I only recommend gear I'd run myself.
The single question that decides whether a local LLM setup will be a joy or a constant fight is how much VRAM do I have. Get it right and models load instantly and generate fast. Get it wrong and the model spills into system RAM, where it runs at a crawl that no amount of tweaking will fix. Here’s how to size it before you spend money — the arithmetic is simpler than the forums make it look.
The one formula
A model’s weights take, very roughly:
VRAM for weights ≈ (parameters in billions) × (bytes per parameter)
Bytes per parameter comes from the quantization — how many bits each weight is squeezed into:
| Quantization | Bytes/param | A 7B model | A 13B model | A 70B model |
|---|---|---|---|---|
| FP16 (full) | 2 | ~14 GB | ~26 GB | ~140 GB |
| 8-bit (Q8) | 1 | ~7 GB | ~13 GB | ~70 GB |
| 4-bit (Q4) | 0.5 | ~3.5 GB | ~6.5 GB | ~35 GB |
Most people run 4-bit quantization, because it cuts memory in half again with a quality loss that’s hard to notice for chat and coding. That’s why a 7B model fits comfortably on an 8 GB card and a 13B model is happy on 12 GB.
Don’t forget the context window
The weights are only part of it. The KV cache — the model’s working memory for the current conversation — grows with how much context you give it, and it’s the thing that quietly pushes you over the edge. A model that loads fine at a 2K context can spill to system RAM at 32K, and the symptom is the same “why did it get slow” that people blame on everything else.
Rule of thumb: leave 1–3 GB of headroom on top of the weights for a normal context, more if you run long documents. If your tool lets you set the context length explicitly, set it to what you actually use — an oversized context window reserves memory you’re paying for and not using. (This is the exact trap that halved my generation speed the first time; I dug into it in running a local LLM on a home GPU.)
The tiers that matter when buying
You don’t need to match a number exactly — you need to land in the right tier:
- 8 GB — 7B models at 4-bit, short-to-medium context. A great, cheap entry point and genuinely useful for coding and chat.
- 12 GB — 13B models comfortably, or 7B with a big context. The sweet spot for most people: noticeably more capable, still affordable used.
- 16 GB — headroom for 13B at longer context, or squeezing a quantized larger model. Where “occasionally frustrated” becomes “rarely.”
- 24 GB — 32–34B models, or a 70B at aggressive quantization. The point where a single consumer card runs the models that feel genuinely smart.
- Two 24 GB cards / 48 GB — 70B models properly. Diminishing returns for a home setup unless the bigger models are the whole point.
The most cost-effective move for a home lab is almost always a used card with more VRAM over a newer card with less — VRAM is the wall you hit, and it doesn’t get cheaper by being recent.
For local LLMs, prioritize VRAM capacity over raw speed — the memory is what decides which models you can run at all.
Browse 16GB+ GPUs ↗The short version
Take the parameter count, multiply by 0.5 for 4-bit, add a couple of gigabytes for context, and buy the tier above that. VRAM capacity decides which models you can run; everything else only decides how fast. Start there and you’ll skip the single most common mistake in local AI.