Running a local LLM on a home GPU — and reaching it from anywhere with Tailscale

If you have a desktop with a half-decent GPU sitting idle most of the day, it can be a private LLM server — no per-token bill, no data leaving your house, and it answers from your laptop, your phone, or a script on another box. I’ve been running this setup for months. Here’s the version that actually works, including the one setting that silently cut my generation speed in half before I understood what it did.

The shape of the setup

Three pieces:

1. Ollama on the machine with the GPU. It downloads models, loads them into VRAM, and exposes an HTTP API.
2. Tailscale on both machines, so the GPU box and your laptop share a private network no matter where either one is.
3. Any OpenAI-compatible client on your laptop, pointed at the GPU box’s Tailscale address.

That’s it. No reverse proxy, no port forwarding, no dynamic DNS, nothing exposed to the public internet.

Step 1: Ollama on the GPU box

Install it, then pull a model. For a 8–12GB GPU, a 7B model quantised to 4 bits is the sweet spot — fast and genuinely useful for coding and chat:

ollama pull qwen2.5:7b
ollama run qwen2.5:7b "say hi"

By default Ollama only listens on localhost. To let another machine reach it, bind it to all interfaces. On Linux with systemd:

sudo systemctl edit ollama

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Then sudo systemctl restart ollama. Do not do this on a machine with a public IP and no firewall — the whole point of the next step is that Tailscale, not the open internet, is what reaches this port.

Step 2: the context-length trap

This is the part nobody warns you about. Ollama defaults to a context window of 2048 tokens. Feed it a longer prompt and it silently truncates — your carefully assembled context just falls off the front, and the model answers as if it never saw it.

The instinct is to crank the context way up. Resist it. Here’s the trade-off I learned the hard way: context length costs VRAM, and the moment your model plus its context no longer fits in the GPU, Ollama spills the overflow to system RAM and runs part of the model on the CPU. When that happens, generation doesn’t fail — it just gets slow. On my box, pushing a 7B model to a 16k context spilled about a quarter of it to CPU, and a request that took 1.5 seconds with a short prompt took over two minutes with a long one.

The fix is to set context deliberately, not maximally, and to bake it into the model rather than hope a per-request flag sticks:

cat > Modelfile <<'EOF'
FROM qwen2.5:7b
PARAMETER num_ctx 8192
EOF
ollama create qwen2.5-8k -f Modelfile

Now qwen2.5-8k always loads with an 8k window. Check what’s actually loaded and how much of it is on the GPU versus the CPU:

ollama ps

If the PROCESSOR column shows anything other than 100% GPU, either lower num_ctx or use a smaller quantisation. Keep it fully on the GPU — a model that fits is worth far more than one that’s technically bigger but crawls.

Step 3: reach it from your laptop with Tailscale

Tailscale puts both machines on a private mesh network. Install it on the GPU box and your laptop, log in to the same account on both, and each gets a stable 100.x.y.z address that works from anywhere — coffee shop, phone tether, another country.

# on the GPU box
tailscale ip -4      # note the 100.x address it prints

From your laptop, the Ollama API is now reachable at http://<that-address>:11434. Test it:

curl http://100.x.y.z:11434/v1/models

Because the port only listens on the Tailscale interface’s reachable network and nothing is forwarded from your router, there’s no public attack surface. If you want to lock it down further, Tailscale ACLs can restrict which of your devices may reach port 11434 at all.

Step 4: point a client at it

Ollama speaks the OpenAI API shape at /v1, so nearly every tool works with just a base URL:

curl http://100.x.y.z:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen2.5-8k","messages":[{"role":"user","content":"Explain a mutex in one paragraph"}]}'

Most coding assistants and chat front-ends have a field for a custom OpenAI base URL — put http://100.x.y.z:11434/v1 there, leave the API key blank or set anything, and you’re running against your own hardware.

The hardware that makes this pleasant

You don’t need a data-centre card. The two things that matter are VRAM (it decides which models fit fully on the GPU — see the context trap above) and having enough system RAM so the rest of your machine isn’t starved while a model is loaded.

• For running 7B–14B models comfortably, a used 16GB card is the single best value in this whole setup. 16GB lets a 14B model run fully on the GPU with room for a real context window.
• If you’re building the box around it, 32GB of system RAM is the floor and 64GB is comfortable.

What you end up with

A private, free-to-run model endpoint that:

• costs nothing per token and keeps every prompt on your own hardware,
• is reachable from all your devices without exposing a single port,
• loads instantly and answers fast, because you sized the context to fit the GPU instead of maxing it out.

The context-length lesson is the one to take away: bigger isn’t faster. A model that fits entirely in VRAM will beat a “bigger” configuration that spills to the CPU every time. Set num_ctx to what your card can hold, confirm it with ollama ps, and enjoy your own LLM.