How much VRAM do you actually need to run a local LLM?

Heads up: some links in this article are affiliate links. If you buy through them I may earn a small commission at no extra cost to you. As an Amazon Associate I earn from qualifying purchases. I only recommend gear I'd run myself.

The single question that decides whether a local LLM setup will be a joy or a constant fight is how much VRAM do I have. Get it right and models load instantly and generate fast. Get it wrong and the model spills into system RAM, where it runs at a crawl that no amount of tweaking will fix. Here’s how to size it before you spend money — the arithmetic is simpler than the forums make it look.

The one formula

A model’s weights take, very roughly:

VRAM for weights ≈ (parameters in billions) × (bytes per parameter)

Bytes per parameter comes from the quantization — how many bits each weight is squeezed into:

Quantization Bytes/param A 7B model A 13B model A 70B model
FP16 (full) 2 ~14 GB ~26 GB ~140 GB
8-bit (Q8) 1 ~7 GB ~13 GB ~70 GB
4-bit (Q4) 0.5 ~3.5 GB ~6.5 GB ~35 GB

Most people run 4-bit quantization, because it cuts memory in half again with a quality loss that’s hard to notice for chat and coding. That’s why a 7B model fits comfortably on an 8 GB card and a 13B model is happy on 12 GB.

Don’t forget the context window

The weights are only part of it. The KV cache — the model’s working memory for the current conversation — grows with how much context you give it, and it’s the thing that quietly pushes you over the edge. A model that loads fine at a 2K context can spill to system RAM at 32K, and the symptom is the same “why did it get slow” that people blame on everything else.

Rule of thumb: leave 1–3 GB of headroom on top of the weights for a normal context, more if you run long documents. If your tool lets you set the context length explicitly, set it to what you actually use — an oversized context window reserves memory you’re paying for and not using. (This is the exact trap that halved my generation speed the first time; I dug into it in running a local LLM on a home GPU.)

The tiers that matter when buying

You don’t need to match a number exactly — you need to land in the right tier:

  • 8 GB — 7B models at 4-bit, short-to-medium context. A great, cheap entry point and genuinely useful for coding and chat.
  • 12 GB — 13B models comfortably, or 7B with a big context. The sweet spot for most people: noticeably more capable, still affordable used.
  • 16 GB — headroom for 13B at longer context, or squeezing a quantized larger model. Where “occasionally frustrated” becomes “rarely.”
  • 24 GB — 32–34B models, or a 70B at aggressive quantization. The point where a single consumer card runs the models that feel genuinely smart.
  • Two 24 GB cards / 48 GB — 70B models properly. Diminishing returns for a home setup unless the bigger models are the whole point.

The most cost-effective move for a home lab is almost always a used card with more VRAM over a newer card with less — VRAM is the wall you hit, and it doesn’t get cheaper by being recent.

For local LLMs, prioritize VRAM capacity over raw speed — the memory is what decides which models you can run at all.

Browse 16GB+ GPUs ↗

The short version

Take the parameter count, multiply by 0.5 for 4-bit, add a couple of gigabytes for context, and buy the tier above that. VRAM capacity decides which models you can run; everything else only decides how fast. Start there and you’ll skip the single most common mistake in local AI.