The best used GPUs for a home LLM server (2026): buy VRAM, not hype

Heads up: some links in this article are affiliate links. If you buy through them I may earn a small commission at no extra cost to you. As an Amazon Associate I earn from qualifying purchases. I only recommend gear I'd run myself.

If you’ve read how much VRAM you actually need, you already know the punchline: for local LLMs, VRAM capacity decides which models you can run at all, and everything else only decides how fast. That single fact is why the used GPU market is where home AI servers get built — you’re buying memory, and memory doesn’t get cheaper just because a card is newer. Here’s how to spend a budget well.

Why used, and the one risk

A used card with more VRAM will out-serve a newer card with less, every time, for LLM work. The risk isn’t performance — it’s condition: ex-mining cards, dried-out thermal paste, and dead fans are the things to screen for (see the checklist at the end). Buy from a seller with returns, and the math strongly favors used.

By budget

Entry (~8GB): run 7B models comfortably. The sweet spot for a first LLM box. 8GB holds a 7B model at 4-bit with room for a normal context, which is genuinely useful for coding and chat. Look for older-gen cards with 8GB — they’re cheap and plentiful on the used market.

A fine first LLM card — runs a 7B model at 4-bit with headroom.

Browse 8GB GPUs ↗

Mid (~12GB): 13B models, or 7B with a big context. The best value tier for most people. The jump from 8→12GB is the difference between “often works” and “rarely frustrated.” This is where I’d point anyone building their first serious home LLM server.

Upper-mid (~16GB): headroom, longer context, quantized bigger models. 16GB lets you run 13B comfortably at long context or squeeze a quantized larger model. If your workloads involve big documents, this is the tier to target.

More VRAM is the upgrade that changes which models you can run — prioritize it.

Browse 16GB GPUs ↗

High (~24GB): the models that feel genuinely smart. 24GB is where a single consumer card runs 32–34B models, or a 70B at aggressive quantization. This is the tier people mean when they say a local model finally “feels like the real thing.” Used 24GB cards hold their value for exactly this reason — but they still beat buying new.

Enthusiast (dual 24GB / 48GB): 70B done properly. Two 24GB cards split a 70B model across them. Diminishing returns for a home setup unless the largest models are the whole point — but if they are, two used 24GB cards cost far less than one new card with equivalent memory.

What to avoid

  • Chasing clock speed or newness over VRAM. A newer 8GB card loses to an older 16GB one for LLMs. Ignore the benchmark charts built for gaming.
  • Cards with under 8GB. Below 8GB you’re fighting the tool constantly; it’s a false economy.
  • No-return “as-is” listings on a card you can’t inspect.

Checking a used GPU before you pay

  1. Ask why it’s being sold and for photos of the actual card (not a stock image).
  2. Run it under load if you can — a stress test for 10–15 minutes, watching temps. Thermal throttling early points at old paste or a failing fan.
  3. Listen to the fans — grinding or rattling means a bearing on the way out (fans are replaceable, but factor the cost).
  4. Check the ports and bracket for damage, and that the model matches the listing.
  5. Buy where you can return it. The single best protection against a bad card.

A meat thermometer for your CPU it is not — but a cheap USB power meter and a stress tool tell you most of what you need before the card ever goes in your server.

The short version

Set your budget, buy the most VRAM it will get you on the used market, screen the card’s condition, and buy somewhere with returns. Do that and you’ll have a home LLM server that runs the models you want for a fraction of new-card money — which is the entire reason to self-host in the first place.