How Much GPU VRAM Does an LLM Need?

The amount of GPU memory an LLM needs depends on much more than the model name.

Parameter count sets the baseline, but precision, quantization, context length, batch size, KV cache, framework overhead and whether you are training or only running inference can all change the real requirement.

The safest approach is to treat weight memory as the starting point and then leave headroom for everything else.

Start with the model weights

A simple first estimate is:

model parameters × bytes per parameter

For weight storage alone, common rough values are:

• FP32: about 4 bytes per parameter
• FP16 or BF16: about 2 bytes per parameter
• INT8: about 1 byte per parameter
• 4-bit quantization: roughly half a byte per parameter before metadata and runtime overhead

That means a 70-billion-parameter model stored entirely in BF16 has a theoretical weight footprint of about 140 GB before KV cache, temporary tensors, framework overhead or other runtime memory.

The real number needed to run it will be higher.

Use the LLM VRAM calculator for a quick estimate.

Inference needs more than the weights

During inference, the model also stores temporary activations and a KV cache.

The KV cache grows with context length and the number of concurrent sequences. This is why a model that fits comfortably at a short prompt length can suddenly run out of memory when you increase the context window or serve many users at once.

Long-context inference therefore benefits from GPUs with large memory capacity, such as NVIDIA H200, NVIDIA B200, NVIDIA B300 and high-memory AMD accelerators.

Quantization can change the hardware requirement

Quantization stores weights using fewer bits.

Moving from BF16 to 8-bit can roughly halve the raw weight footprint. Moving to 4-bit can reduce it further.

The trade-off is not simply memory versus quality. Different quantization methods have different accuracy, throughput and hardware characteristics. Some models tolerate aggressive quantization extremely well; others lose useful quality.

Quantization also does not eliminate KV cache and runtime memory, so a model whose weights technically fit may still exceed the GPU's capacity in production.

Context length matters

A larger context window lets the model process more tokens in a single request, but it can dramatically increase memory use.

The exact relationship depends on the model architecture and attention implementation. Modern techniques such as grouped-query attention can reduce KV-cache requirements, but long context still consumes memory.

If you are sizing hardware for a production service, test the maximum context and concurrency you actually intend to offer rather than benchmarking only short prompts.

Batch size and concurrency

Serving several requests together can improve GPU utilisation, but it also requires more memory.

Higher batch sizes hold more active tokens and inference state at the same time. Production systems therefore balance throughput, latency and VRAM rather than simply maximizing batch size.

This is also why two providers using the same GPU can produce different real-world economics: software, batching and serving infrastructure change how much useful work each GPU completes.

One GPU or multiple GPUs?

A model does not always have to fit on one GPU.

Tensor parallelism, pipeline parallelism and other distributed approaches can split the model across several accelerators. That gives access to a larger combined memory pool, but communication between GPUs becomes important.

High-bandwidth links such as NVLink can reduce the performance penalty of splitting work across devices. For H100 deployments, see H100 PCIe vs SXM for why interconnect matters.

Multiple GPUs also do not behave exactly like one giant GPU. Software and communication overhead mean eight 80 GB GPUs are not simply equivalent to one hypothetical 640 GB accelerator.

Training uses far more memory

Training is much more demanding than inference.

In addition to model weights, training may need gradients, optimizer states, activations and temporary buffers. Depending on the optimizer and precision, these can consume several times the raw weight memory.

Techniques such as gradient checkpointing, mixed precision, optimizer sharding and distributed training reduce memory requirements, but frontier-scale training still depends on large GPU clusters.

For this reason, do not use an inference VRAM estimate to size a training system.

Fine-tuning sits in the middle

Full fine-tuning can approach the memory demands of training because many or all model weights are updated.

Parameter-efficient techniques such as LoRA and QLoRA can dramatically reduce the number of trainable parameters and make fine-tuning possible on much smaller systems.

Even then, the base model still has to be loaded, and context length, batch size and quantization continue to matter.

Choosing a GPU

Start with the memory requirement, then consider bandwidth, compute, interconnect and price.

A GPU with enough VRAM but poor economics may not be the best choice. A more expensive GPU can sometimes finish the workload so much faster that its total cost is lower.

Compare specifications in the GPU database, then check current provider rates in the GPU Price Tracker.

For cloud deployment, also compare providers in the GPU cloud directory.

A practical sizing workflow

First, identify the exact model and parameter count. Decide the precision or quantization you will actually use.

Next, estimate raw weight memory. Add headroom for KV cache, runtime overhead and your target context length and concurrency.

If the workload does not fit on one GPU, decide whether quantization is acceptable or whether the model should be split across multiple accelerators.

Finally, benchmark the real workload before committing to a large reservation or hardware purchase.

VRAM tells you whether a workload can fit. It does not by itself tell you how fast or economical the workload will be.