Can My GPU Run This LLM?

Ollama uses Flash Attention 2 and Q4_K_M quantization by default, significantly reducing VRAM vs raw FP16. Use the framework selector and set quantization to Q4KM for a close match. Also, Ollama can spill to CPU RAM, so it may "run" a model that technically doesn't fit fully in VRAM.

Mixture-of-Experts models like DeepSeek-V3 or Mixtral have many "expert" sub-networks. Only a fraction activate per token, but ALL must reside in VRAM for fast switching. A 671B MoE model still loads all 671B parameters—you can't selectively load only active experts.

The KV (Key-Value) cache stores attention states for all previous tokens. It grows linearly with sequence length: a 70B model at 128K context adds 12+ GB for KV cache alone on top of model weights. This is why "context length" is as important as model size for VRAM planning.

Speed estimates are based on memory bandwidth (for small-batch inference) and calibrated against community benchmarks. Expect ±20% accuracy. Actual performance depends on driver version, PCIe bandwidth, thermal throttling, and concurrent system load.