What does that GPU actually buy you?
The one number that matters for running AI locally is VRAM — your graphics card's memory. It decides the biggest model you can run. Everything below is organised by it. Prices are mid-2026 US street prices and move fast; token/sec figures are ballpark.
A 70-billion-parameter model at usable quality needs about 40 GB of VRAM — so no single consumer GPU runs one fully, not even the 32 GB RTX 5090. The real paths to 70B are two GPUs, a workstation card, or an Apple machine with 64 GB+ of unified memory.
What each VRAM tier runs
| VRAM | Biggest LLM | Example models | Speed | Also enables |
|---|---|---|---|---|
| 8 GB | 7–9B @ Q4 | Llama 3.1 8B, Qwen3 8B, Ministral 8B, Phi-4-mini | ~30–40 t/s | SDXL image gen; embeddings & classifiers; SD1.5 LoRA training |
| 12 GB | 12–14B @ Q4 | Gemma 3 12B, Qwen3 14B, Phi-4 14B, R1-Distill-14B | ~30–45 t/s | SDXL comfortable; FLUX (quantized); SD1.5 LoRA comfortable |
| 16 GB | ~24–32B (MoE) @ Q4 | Mistral Small 24B, Gemma 3 27B, Qwen3 30B-A3B (MoE) | ~32–46 t/s | FLUX (FP8/GGUF); SDXL LoRA training; longer contexts |
| 24 GB | 32B dense @ Q4 comfortable | Qwen3 32B, Gemma 3 27B, R1-Distill-32B | ~73–87 t/s (32B) | Local video gen (quantized); full FLUX + QLoRA; SDXL LoRA with room |
| 32 GB | 32B @ Q6/Q8 or big context | 32B near-lossless; partial large-MoE (GLM, Qwen3-235B-A22B) | ~2× a 4090 | 1080p local video; fastest single-card prompt processing; LoRA with headroom |
"Q4" = 4-bit quantization, the standard space-saving trade-off — see the glossary. MoE models (like Qwen3 30B-A3B) punch above their memory footprint by activating only part of the network per token.
Representative GPUs & prices
| Tier | Card | VRAM | MSRP | Street (Jul 2026) | Notes |
|---|---|---|---|---|---|
| 8 GB | RTX 5060 | 8 GB | $299 | ~$300 new | Entry; fine for 7–8B + SDXL |
| 8 GB | used RTX 3060 Ti | 8 GB | — | ~$180–250 used | Budget option |
| 12 GB | RTX 5070 | 12 GB | $549 | ~$550–650 new | Good all-rounder |
| 12 GB | used RTX 3060 12GB | 12 GB | — | ~$180–240 used | Best cheap 12 GB for AI |
| 16 GB | RTX 5060 Ti 16GB | 16 GB | $429 | ~$500–575 new | Best entry AI value |
| 16 GB | RTX 5070 Ti | 16 GB | $749 | ~$880–1,070 new | Faster 16 GB, inflated |
| 16 GB | RTX 5080 | 16 GB | $999 | ~$1,400+ new | 16 GB not 24 — poor $/VRAM for AI |
| 24 GB | used RTX 3090 | 24 GB | — | ~$600–900 used | The $/VRAM value king for local AI |
| 24 GB | used RTX 4090 | 24 GB | — | ~$2,000+ used | Faster, but 2–3× the 3090 price |
| 32 GB | RTX 5090 | 32 GB | $1,999 | ~$2,900–4,300 new | Only 32 GB consumer card; ~575W |
Two corrections worth internalising: the RTX 5080 is 16 GB, not 24 — the only new card above 16 GB is the 32 GB 5090. And the used RTX 3090 remains the value pick for local AI: 24 GB for well under a thousand dollars.
The Apple Silicon alternative
Macs share one pool of memory between CPU and GPU, so a big-RAM Mac can load models no consumer NVIDIA card fits. The catch: prompt processing is slower and there's no CUDA, so training tools and image/video generation are weaker.
| Mac memory | Usable for models | Runs comfortably | Speed |
|---|---|---|---|
| 32 GB (M-series Pro) | ~24 GB | up to ~24–32B Q4; 30B MoE | 8B ~45–55, 32B ~15–20 t/s |
| 64 GB (M-series Max) | ~48 GB | 70B Q4/Q5 feasible; big-context 32B | 70B ~12–18 t/s |
| 128 GB (Max / Ultra) | ~96 GB | 70B comfortably; 100B+ MoE | 70B ~12–18 t/s |
Does local actually save money?
Usually not — at least not on raw token cost. Against today's cheap and free hosted open models (DeepSeek, Llama, Gemma via OpenRouter), a home rig can take years of heavy daily use to pay for itself, and against free tiers it essentially never does. Local wins on the things that aren't the per-token price: privacy, no limits, offline, and control.
Why run local at all — honestly
The upsides
- Privacy — prompts and data never leave your machine. The number-one real reason.
- No rate limits — hammer it 24/7, no per-request throttling.
- Offline — works with no internet at all.
- Cost at high, sustained volume — past break-even, extra tokens are basically free power.
- Uncensored & custom — run any open weight, your own fine-tunes, niche models.
- Learning — you understand the whole stack instead of renting a black box.
The honest downsides
- Upfront cost — a capable rig runs $600 (used 3090) to $3,000+ (5090 or big Mac) before a single token.
- Slower & weaker than frontier cloud — a local 32B is not GPT-5/Claude-class.
- Maintenance tax — drivers, quant formats, CUDA/ROCm, thermals, VRAM Tetris.
- Power, heat & noise — 350–575W under load.
- VRAM ceiling — 70B+ and long context need multi-GPU or a big-memory Mac.