Guide

Open-weight models on consumer GPUs: what runs on what

Which open-weight models run on consumer GPUs, and what hardware do you need?

Updated · 5 min read · kept current from our daily coverage

Most open-weight models under 13B parameters run on modern consumer GPUs with at least 8GB of VRAM. Models up to 70B are feasible on 24GB cards like the RTX 4090 using 4-bit quantization, but with significant performance compromises. Anything larger, like the 120B models touted for the latest generation of so-called AI PCs, is not practical for interactive use on a single consumer machine. VRAM is the unshakable bottleneck, and no amount of software trickery has changed that fundamental fact.

The 2B to 7B Range: The Entry Point

This is the starting line, and it's easily accessible. If you have a gaming PC from the last few years, you can run these models. An NVIDIA RTX 3060 with 12GB of VRAM is the ideal budget card here. An 8GB card will work, but you'll be tight on space for larger context windows.

Take a model like Amazon's Strands Decider 2B, which we looked at in October 2026. At full FP16 precision, it requires about 4GB of VRAM for the weights alone. Add context and overhead, and an 8GB card handles it comfortably. Quantize it down to 4-bits, and the VRAM usage becomes almost trivial. The same math applies to popular 7B models.

Performance in this bracket is excellent. You can expect fast, conversational token generation speeds. This is the perfect territory for learning the ropes, running local RAG pipelines, and fine-tuning specific tasks. There is no need to overspend on hardware if this is your goal. It just works.

The 13B to 70B Range: The Enthusiast Tier

Here, the hardware requirements ramp up severely. VRAM is everything. Don't even attempt this with less than 24GB. This effectively means your only real option in the consumer space is the NVIDIA RTX 4090. Anything else is a waste of time and money.

Quantization is mandatory. A 70B model at FP16 precision is around 140GB. It is not fitting into consumer hardware. Using a 4-bit quantization scheme like Q4_K_M in a GGUF container, that 70B model shrinks to roughly 40GB. This is still larger than the 4090's 24GB of VRAM.

The solution is GPU layer offloading. The model's layers are split between your fast GPU VRAM and your slower system RAM. The inference engine loads as many layers as it can into VRAM and pages the rest in from system RAM as needed. This works, but it kills performance. The PCI-Express bus is a crawl compared to on-chip memory. Expect a few tokens per second, not a fluid conversation. It's useful for generating text non-interactively, but it feels slow.

This is the class of model that drives enthusiasts. The push to run fine-tunes like the Qwen variant from September 2026, which aimed for a more humanlike chat style, is why people are willing to deal with the cost and complexity. It's possible, but it is not seamless.

The 120B+ Fantasy: Marketing vs. Physics

This category is almost entirely aspirational on consumer hardware. The October 2026 launch of Microsoft's Surface Laptop Ultra and its accompanying line of "AI PCs" came with bold claims of running 120B models locally. I tested this. The reality does not match the marketing slides.

A 120B model, even with aggressive 4-bit quantization, occupies more than 60GB. No laptop has this much VRAM. These new machines from Microsoft rely on a unified memory architecture and heavy offloading to system RAM. While the direct access to system memory is faster than going over a PCIe bus, it's still orders of magnitude slower than dedicated VRAM.

The result is a slideshow. Tokens generate so slowly that the model is unusable for any practical purpose. It's a technical demonstration that proves the gap between hardware specs and real-world utility remains vast. You can say it runs, but you can't say it's usable.

The only currently viable path for running models of this size locally is a multi-GPU, or even multi-PC, setup. I ran tests on NVIDIA's PAIR framework in September 2026. It allows you to link multiple PCs over a fast network to pool their VRAM for a single inference task. It's finicky, demands a low-latency network, and is a nightmare to configure. But it proves the point: running huge models requires a distributed approach that goes far beyond a single consumer-grade machine.

The Software Stack Problem

Hardware is only half the battle. The software environment is a house of cards. Getting the correct NVIDIA drivers to play nice with the right version of CUDA, which then needs to match what your inference server was compiled with, is a constant struggle. Add in Python dependency hell, and you will spend more time managing your environment than running models.

New tools and libraries appear constantly, each promising to unify or simplify the process. They rarely deliver, instead adding another layer of abstraction and another potential point of failure. This is the reality of building on a rapidly moving foundation.

Projects are attempting to build better abstraction layers. Google's ML Drift, which came to light in October 2026, wants to create a universal API across different GPU backends. It's a noble goal, but one that has been tried before with limited success. For now, you are the integration layer.

Verdict: What to Actually Buy

For tinkering and running capable models up to 13B: An NVIDIA RTX 3060 with 12GB of VRAM offers the best performance per dollar. It has enough memory for a wide range of tasks without requiring you to sell a kidney.

For serious local inference up to 70B: Get an NVIDIA RTX 4090 with 24GB. There is no other choice. Accept that you will be dealing with slow generation speeds due to RAM offloading, but it is the peak of what is possible on a single consumer card today.

For 120B models and beyond: Do not try this on a single machine. The promises of the new 'AI PCs' from Microsoft and others are functionally hollow. Your options are a dedicated multi-GPU server or a cloud provider's API. The local AI dream has a hard ceiling, and that ceiling is made of 24GB of VRAM.

Sources on this site