LLM VRAM Requirements for 7B, 13B, and 70B Models

For a local LLM, plan for 8–12 GB of VRAM for a 7B model in Q4, 16–24 GB for 13B, 24–48 GB for 34B, and 48–80 GB for 70B; FP16 increases memory usage by approximately 3–4×.

The self hosted llm vram requirements question cannot be answered by parameter count alone: the final VRAM requirement depends on the weight format, context length, KV-cache, number of concurrent requests, and the headroom required by the runtime. A model may fit into 24 GB of VRAM in a 2K-context test but fail with an out-of-memory error at 16K tokens and five parallel conversations.

How to calculate VRAM requirements for an LLM

To determine how much VRAM for LLM workloads you need on a specific server, estimate memory for the model weights first, then add KV-cache, CUDA working buffers, and at least 10–15% headroom.

VRAM required for model weights

The simplified formula for model weights is:

VRAM for weights ≈ number of parameters × bytes per parameter × service-data overhead factor

Q4    ≈ 0,55–0,65 bytes per parameter
Q8    ≈ 1,05–1,20 bytes per parameter
FP16  = 2 bytes per parameter
FP32  = 4 bytes per parameter

For example, a 70B model in FP16 requires approximately 140 GB just for its weights. In Q4_K_M format, its files typically occupy 40–45 GB, but for a stable deployment with the runtime and context loaded, it is better to plan for at least 48 GB of VRAM.

Quantization does not reduce quality linearly: Q4 is generally suitable for chatbots, RAG, classification, and agent workflows; Q8 preserves more accuracy for complex instructions and code; and FP16 is primarily needed for training, fine-tuning, and workloads where result degradation is unacceptable. Review the detailed self-hosted AI server requirements before ordering a GPU server, not after transferring the weights.

Why context length and KV-cache change the calculation

After loading the weights, the inference engine allocates a KV-cache—the cache of attention layers used to store conversation history. Its usage depends on the architecture, including the number of layers, KV heads, head dimension, cache type, and context length. In modern models with GQA, the KV-cache is considerably smaller than in older architectures with full multi-head attention.

• 7B/8B with GQA: approximately 0,3–0,8 GB of KV-cache per 4K tokens for one request in FP16;
• 13B: typically 0,6–1,5 GB per 4K tokens;
• 34B: approximately 1–3 GB per 4K tokens;
• 70B: around 1,3–3 GB per 4K tokens and 3–6 GB per 8K tokens, depending on the architecture and runtime.

With a batch size of 8, the cache grows almost eightfold. That is why a 48 GB GPU can comfortably serve a 70B Q4 model with one conversation and a 4K context, but a queue of several requests with a 16K context may already require 80 GB or multiple GPUs.

LLM VRAM requirements by model size: 7B, 13B, 34B, and 70B

LLM VRAM requirements by model size are best estimated using a safe operating capacity rather than the bare minimum: a minimum-size deployment does not guarantee acceptable speed, a long context, or freedom from CUDA out-of-memory errors.

VRAM requirements for Q4, Q8, and FP16

The ranges include a small amount of runtime overhead, but they do not replace a KV-cache calculation for the intended context length. MoE models require a different calculation: the total parameter count may be 47B, but only some experts are active at a time; nevertheless, all weights still need to be stored in memory.

How much VRAM for 7B 13B 70B in real-world workloads

The question how much VRAM for 7B 13B 70B has a practical answer: 7B Q4 runs comfortably on 12–16 GB, 13B Q4 needs 16–24 GB, and 70B Q4 should be planned for at least 48 GB, or 80 GB for a long context. A 24 GB card is a versatile starting point for 7B, 13B, and some 34B models; a 48 GB card enables 70B Q4 with moderate concurrency.

For local development, 7B/8B models on 16 GB of VRAM typically generate 30–80 tokens per second with a short context. 13B models on 24 GB more often deliver 20–50 tokens per second. A 70B Q4 model on a single 80 GB GPU may run at 10–30 tokens per second on one stream, depending on the engine, quantization, and batch size.

What GPU do you need for a local LLM 70B or smaller models?

For the gpu for local llm 70b use case, the optimal starting point is 48 GB of VRAM for single-GPU Q4 inference, but 80 GB remains the more reliable configuration for an API with a long context, RAG, and multiple users.

When is a single GPU enough?

• 12–16 GB VRAM: 7B/8B in Q4, embeddings, rerankers, and small code models.
• 24 GB VRAM: 13B Q4, 34B Q4 with a strict context limit, and Q8 for 7B/8B.
• 48 GB VRAM: 34B Q8 or 70B Q4 with a 4K–8K context and 1–2 active requests.
• 80 GB VRAM: 70B Q4 with headroom for KV-cache, batching, and API load; 34B in FP16.

When choosing an accelerator, consider more than VRAM capacity. LLM workloads depend on memory bandwidth, PCIe or NVLink for multi-GPU setups, BF16/FP16 support, driver stability, and compatibility with vLLM, TensorRT-LLM, llama.cpp, or Ollama. Practical rental scenarios are covered in which GPU to rent for 7B and 70B LLM inference.

When do you need multi-GPU and tensor parallelism?

Multi-GPU is required in three situations: the model does not physically fit on one card, you need more parallelism, or you require FP16/BF16 instead of Q4. A 70B FP16 model is typically distributed across at least two 80 GB GPUs, while retaining headroom for the runtime. A Q4 version can be split across two 24 GB GPUs, but performance depends on the interconnect: NVLink is preferable, while PCIe can add latency between layers.

# Example of launching 70B Q4 through llama.cpp with two GPUs
./llama-server \
  -m /models/llama-70b-q4_k_m.gguf \
  -ngl 99 \
  -c 8192 \
  -np 2 \
  --tensor-split 0.5,0.5 \
  --host 0.0.0.0 \
  --port 8080

Do not confuse model sharding with doubling performance. Two 24 GB cards can provide enough capacity for the weights, but without a fast interconnect they will not necessarily outperform one 48 GB GPU. Multi-GPU is justified when it solves a real capacity or batch-throughput problem.

Quick pick

Need a dedicated server?

Bare metal with NVMe in 70+ locations — configure and order in minutes.

Browse servers

Quantization VRAM requirements: Q4, Q8, or FP16?

Quantization VRAM requirements affect not only server cost, but also output quality, model loading speed, and the maximum practical context. For most self-hosted APIs, Q4_K_M or AWQ/GPTQ 4-bit is a sensible starting point, provided it is tested on your own data.

How to choose a weight format

1. Q4: Choose it for chatbots, internal search, RAG, and cost-efficient deployment of 7B–70B models. VRAM savings compared with FP16 reach approximately 65–75%.
2. Q8: Use it when 4-bit quantization noticeably harms SQL, code, strict JSON generation, or answers based on domain-specific data. VRAM usage is nearly twice that of Q4.
3. FP16/BF16: Required for training, LoRA/QLoRA pipelines, high-accuracy inference, and models without a validated 4-bit quantization.

Weight quantization and KV-cache quantization are separate settings. If the runtime stores the KV-cache in FP16, a long context can become the primary VRAM consumer even with Q4 weights. FP8 or Q8 KV-cache support can sometimes increase the available context, but quality must be tested on long documents.

How much RAM, CPU, and storage does a self-hosted LLM need?

In addition to VRAM, a self-hosted LLM needs system memory for the loader, file cache, containers, and CPU offload: 32 GB of RAM is enough for 7B, 48–64 GB is better for 13B, 64–96 GB for 34B, and at least 128 GB for 70B.

CPU and RAM for inference

If all layers are placed on the GPU, the CPU rarely becomes the bottleneck. One GPU generally needs 8 modern vCPUs clocked at 3,0 GHz or higher; two GPUs, vLLM, and several concurrent requests require 16 vCPUs. CPU offload saves VRAM but reduces speed: moving even a few 70B layers over PCIe can reduce generation speed from tens of tokens per second to single digits.

RAM should cover the model file size by at least 1,5–2× during loading, conversion, and container operation. For 70B Q4, it is sensible to allocate 128 GB of ECC RAM; for 70B FP16, plan for at least 256 GB if the weights are stored and prepared locally.

NVMe storage, networking, and headroom

Storage must accommodate the weights, multiple quantizations, the Hugging Face cache, Docker images, logs, and RAG data. Allocate at least 200 GB of NVMe for one 70B Q4 model; for multiple model formats and a vector database, use 500 GB–1 TB of NVMe. SATA SSD is acceptable for storage, but NVMe significantly reduces cold-start loading time for 40–150 GB of weights.

For an external API, use a port of at least 1 Gbps and rate-limit requests through a reverse proxy. Text traffic is small: a response of 1 000 tokens typically occupies only a few kilobytes, but RAG attachments, logs, and model downloads can consume tens or hundreds of gigabytes per month.

Load scale → vCPU, RAM, storage, bandwidth, and price

For 5 concurrent users, a 70B Q4 deployment needs 16 vCPUs, 128 GB of RAM, 500 GB of NVMe storage, and 48–80 GB of VRAM.

These prices are broad market estimates for GPU infrastructure in March 2025 and vary by region, accelerator model, interconnect type, rental term, and included bandwidth. For 7B and 13B models, the budget is substantially lower: a single GPU with 16–24 GB of VRAM usually covers most internal services. When selecting infrastructure, compare your actual workload with this VRAM and RAM estimate for an LLM server.

Quick pick

Need a dedicated server?

Bare metal with NVMe in 70+ locations — configure and order in minutes.

Browse servers

How to check VRAM before loading a model

Before deployment, measure available GPU memory, the size of the GGUF/Safetensors file, and peak usage at the target context length. Do not use file size alone as the criterion: the runtime may require an additional 2–10 GB depending on the model and batch size.

Commands for diagnosing GPU resources

# GPU status, VRAM, and active processes
nvidia-smi

# Update every 2 seconds
watch -n 2 nvidia-smi

# Check weight-file size
du -sh /models/*.gguf
du -sh /models/*.safetensors

# Monitor container memory
docker stats --no-stream

For testing, launch the model with a minimum context of 2048, then increase it to 4096, 8192, and the target limit. Increase the number of concurrent requests at the same time. This reveals the actual limit of a specific engine rather than a theoretical estimate. When selecting an accelerator, compare your results with the recommendations in this review of GPUs for local 7B–70B inference.

Frequently Asked Questions

Can you run a 70B model on a GPU with 24 GB of VRAM?

Yes, but usually only with aggressive quantization and CPU offload for some layers. A 70B Q4 model uses approximately 42–48 GB with the runtime, so a 24 GB card cannot hold the entire model. It can run with 24 GB of VRAM and 128 GB of RAM, but speed often falls to a few tokens per second because of PCIe transfers.

How much VRAM does a 13B model need in Q4?

A 13B model in Q4 requires approximately 9–12 GB of VRAM with a short context, but a practical choice is a 16 GB GPU. This headroom covers CUDA buffers, KV-cache for 4K–8K tokens, and one or two concurrent requests. For the Q8 version of 13B, it is better to have 20–24 GB of VRAM.

What matters more for an LLM: VRAM or system RAM?

For fast inference, VRAM matters more because placing all layers on the GPU provides substantially higher generation speed. RAM determines whether you can load the weights, use CPU offload, and run services around the model. For 70B Q4, the recommended configuration is 48–80 GB of VRAM and at least 128 GB of system RAM.

Do you need an 80 GB GPU for 70B Q4?

Not always: 70B Q4 can run on 48 GB of VRAM with one active conversation and a context of approximately 4K tokens. However, an 80 GB GPU is better suited to APIs, RAG, 8K–16K contexts, and batching. The additional 32 GB provides more room for KV-cache and reduces the risk of out-of-memory errors during peak requests.

Key takeaways

For 7B and 13B models, choose a GPU with 16–24 GB of VRAM; for 34B Q4, plan for 24–48 GB; and for 70B Q4, use at least 48 GB, with 80 GB preferred for API workloads. Plan VRAM together with KV-cache: a practical server configuration for 70B Q4 is 16 vCPUs, 128 GB of RAM, 500 GB of NVMe storage, and a GPU with 48–80 GB of VRAM.

SSD NVMe

Ready to launch your VPS?

NVMe VPS activated in 60 seconds: full root access, 20+ locations, and payment by card or crypto.

Choose a plan