Self hosted AI stack hardware: what the stack includes
For a self-hosted AI stack with an LLM, RAG, and vector database for a team of 10–20 employees, you need a server with 16 vCPU, 64 GB RAM, 1 TB NVMe, and a GPU with at least 24 GB VRAM.
A complete on-premises AI stack includes more than a model running in Ollama or vLLM. A typical setup includes an LLM for generation, an embedding model for document indexing, a vector database, an interface such as Open WebUI, an API layer, a task queue, and sometimes a reranker. Resource use is uneven across these services: the GPU determines LLM response speed, RAM affects caching and index size, while fast NVMe storage is critical for the vector database and bulk file uploads.
The basic flow is simple: a user submits a question in Open WebUI, the orchestrator finds relevant chunks in Qdrant, Milvus, or pgvector, passes the context to the LLM, and returns an answer. If PDF, DOCX, and knowledge-base reindexing is running at the same time, the server takes additional CPU, RAM, disk, and embedding-model load.
Which services use the most resources
- LLM inference: the main consumer of VRAM, GPU compute, and RAM when using CPU offload.
- Embedding model: can run on CPU, but a GPU significantly speeds up processing tens of thousands of documents.
- Vector database: actively uses RAM for indexes, NVMe for segment storage, and CPU for search.
- Open WebUI, Flowise, Dify, Langfuse: usually have low resource requirements but create connections, logs, and chat history.
- PostgreSQL, Redis, object storage: are needed for metadata, queues, files, and backups.
When choosing a model and estimating VRAM, refer to the separate self-hosted AI and LLM server requirements guide: model size, context length, and quantization format affect the configuration more than the web interface itself.
Why you cannot size a server based on the model alone
A Qwen, Llama, or Mistral model in Q4 format may use 5–6 GB VRAM at 7–8B parameters, but that is not the full requirement. You also need headroom for context, KV-cache, CUDA buffers, concurrent requests, and the embedding model. An 8B model can run on a GPU with 8 GB VRAM, but latency becomes unpredictable with 16–32K token context and multiple users.
A practical rule: reserve 20–30% VRAM beyond the loaded model size, 25% RAM beyond estimated container usage, and at least 30% free NVMe capacity for index compaction, temporary files, and image updates.
AI stack server requirements by workload scale
AI stack server requirements depend on the number of simultaneous chats, model size, knowledge-base volume, and indexing frequency—not just the number of registered accounts.
Fact block: workload scale → specs
The estimated prices below apply to the VPS and GPU server market as of February 2025; for continuous enterprise workloads, confirm the configuration with Valebyte before ordering.
For 10–20 concurrent users of a RAG system, 16 vCPU, 64 GB RAM, 1 TB NVMe storage, and a GPU with 24 GB VRAM are sufficient.
| Workload scale | vCPU | RAM | Storage | Network port | Price, $/month |
|---|---|---|---|---|---|
| 1–2 users, LLM 7–8B, up to 20 GB of documents | 8 vCPU | 32 GB | 500 GB NVMe | 1 Gbps | approximately from $45 |
| 10–20 users, LLM 8–14B, up to 200 GB of documents | 16 vCPU | 64 GB | 1 TB NVMe | 1 Gbps | approximately from $300 with GPU 24 GB VRAM |
| 20–50 users, LLM 32B Q4, up to 1 TB of documents | 24 vCPU | 128 GB | 2 TB NVMe | 1–10 Gbps | approximately from $700 with GPU 48 GB VRAM |
| Department of 50–100 users, multiple models and RAG | 32 vCPU | 256 GB | 4 TB NVMe RAID 1 | 10 Gbps | approximately from $1,400 with 2×48 GB VRAM |
How to interpret these requirements
The first row suits a personal assistant, internal documentation search, and testing with Qwen2.5 7B, Llama 3.1 8B, or Mistral 7B in Q4. The model will respond on CPU, but performance is usually limited to a few tokens per second; for interactive chat, use a GPU with at least 12 GB VRAM.
The configuration for 10–20 people assumes not 20 continuous generations, but 2–5 simultaneous requests. If all employees use AI at the same time, add GPU capacity rather than only vCPU. For a department, it makes sense to separate components: a dedicated GPU server for the LLM, a separate node for Qdrant/PostgreSQL, and document storage.
Looking for a reliable server for your projects?
VPS from $10/month and dedicated servers from $9/month with NVMe, DDoS protection, and 24/7 support.
View offers →Self hosted AI stack hardware for LLMs: GPU, VRAM, and CPU
Self hosted AI stack hardware starts with VRAM selection, because it determines the maximum model size, context length, and number of simultaneous responses.
Which models fit in VRAM
A quantized 7–8B model in Q4 typically requires 5–6 GB of memory, 14B requires around 10–12 GB, 32B Q4 requires 20–24 GB, and 70B Q4 requires from 40 GB without much context headroom. Actual usage depends on the engine, GGUF/AWQ/GPTQ quantization, context length, and KV-cache.
- 12 GB VRAM: 7–8B Q4, one or two interactive users, limited context.
- 24 GB VRAM: 8–14B with reasonable headroom or 32B Q4 with context trade-offs.
- 48 GB VRAM: 32B Q4/Q6, long context, multiple simultaneous sessions.
- 2×48 GB VRAM: 70B Q4, workload separation between models, or high concurrency.
A detailed Ollama sizing calculation, including RAM for partial CPU layer offloading, is available in the Ollama RAM, VRAM, and CPU sizing guide.
What can run on CPU
You can run Open WebUI, PostgreSQL, Redis, Qdrant, ingestion pipelines, and smaller embedding models on CPU. For example, bge-small or multilingual-e5-small can index documents on 8–16 vCPU without a GPU, although processing will be slower. CPU-only LLM inference is reasonable for personal use, overnight jobs, and infrequent requests, but generation latency is usually unacceptable for a team.
For a CPU-only server, choose modern cores with clock speeds from 3.0 GHz and fast NVMe storage. 16 vCPU and 64 GB RAM are a practical minimum for an LLM 7–8B Q4 plus RAG, but expect around 3–10 tokens/s depending on the processor and engine. A GPU usually delivers tens of tokens/s for models in the same class.
LLM RAG vector DB server requirements: RAM, indexes, and NVMe
LLM RAG vector DB server requirements depend on the number of chunks, embedding dimensions, HNSW index type, and whether hot segments need to remain in memory.
How to estimate vector database size
One 768-dimension float32 embedding uses approximately 3 KB without overhead. One million vectors require around 3 GB for the values array alone; with the HNSW graph, payload, replication, and internal segments, plan for 8–15 GB RAM and 10–20 GB storage per million vectors. At 1536 dimensions, the base figures roughly double.
Documents become chunks. If text is split into 500-token chunks with 50-token overlap, 1 GB of plain text can generate hundreds of thousands of chunks. Scanned PDFs also require OCR, while tables and images increase ingestion time.
# Minimal Qdrant configuration example for Docker Compose
services:
qdrant:
image: qdrant/qdrant:v1.12.5
volumes:
- /srv/qdrant:/qdrant/storage
ports:
- "6333:6333"
deploy:
resources:
limits:
memory: 16GQdrant, pgvector, or Milvus
Qdrant is convenient as a dedicated RAG vector database: it starts quickly in a container and works well for most knowledge bases up to tens of millions of vectors. pgvector is a practical choice if PostgreSQL is already used by the application and transactional metadata is needed alongside embeddings. Milvus is more often selected for large volumes, high concurrency, and a dedicated engineering team.
Do not place LLM files, Docker images, Qdrant, PostgreSQL WAL, and backups on a single full SSD. For a database up to 200 GB, 1 TB NVMe is sufficient, but with 1 TB of source documents, plan at least 2–4 TB NVMe and separate S3-compatible storage. For source files and snapshots, use the approach described in the MinIO and S3 object storage requirements guide.
Self hosted RAG hardware sizing for documents and users
Self hosted RAG hardware sizing should begin with an inventory of your documents: total volume, formats, update rate, and expected number of simultaneous search requests.
Planning formula
For an initial estimate, use four values: number of documents, average chunks per document, embedding vector size, and index factor. For example, 100 000 documents with 20 chunks each produce 2 million vectors. At 768 float32, this is around 6 GB of raw vectors; with HNSW, payload, and headroom, allocate 32 GB RAM and 100 GB NVMe specifically for vector storage.
If documents are updated daily, capacity is not the only factor—reindexing speed also matters. An embedding model on a GPU with 24 GB VRAM can process large queues much faster than CPU. For regular ingestion, create a separate worker, limit parallel jobs, and run bulk indexing outside peak hours.
Context, reranking, and answer quality
RAG should not pass all retrieved text to the LLM. A typical configuration uses top_k=20 for initial retrieval, a reranker keeps 4–8 chunks, and 3 000–8 000 context tokens are sent to the model. This reduces VRAM usage and generation time while keeping answers grounded in source material.
# Variables for a conservative RAG pipeline
TOP_K=20
RERANK_TOP_N=6
CHUNK_SIZE=500
CHUNK_OVERLAP=50
EMBEDDING_BATCH_SIZE=64
OLLAMA_NUM_CTX=8192When increasing OLLAMA_NUM_CTX from 8K to 32K, the KV-cache can consume several additional GB of VRAM. Do not increase context “just in case”: first measure average document size, the actual prompt, and retrieval quality.
OpenWebUI Ollama vector DB server specs and deployment
OpenWebUI Ollama vector DB server specs for a production environment require separate containers, persistent volumes, resource limits, and restricted access to internal ports.
Basic container architecture
You can start on one server with Docker Compose: Ollama serves the model, Open WebUI provides SSO or local accounts, Qdrant stores vectors, and PostgreSQL stores metadata and history. For a team deployment, do not expose Qdrant and Ollama directly to the internet: expose only a reverse proxy with TLS and authentication.
services:
ollama:
image: ollama/ollama:latest
volumes:
- /srv/ollama:/root/.ollama
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
openwebui:
image: ghcr.io/open-webui/open-webui:main
volumes:
- /srv/openwebui:/app/backend/data
environment:
- OLLAMA_BASE_URL=http://ollama:11434Required settings
- Enable TLS on the reverse proxy and disable anonymous registration.
- Limit upload size, for example to 50–100 MB, so one PDF cannot fill the OCR queue.
- Configure Docker limits: Qdrant must not evict Ollama from RAM, and Open WebUI should not access the GPU unnecessarily.
- Store API secrets and PostgreSQL passwords in environment variables or a secret store, not in a public Compose file.
- Separate roles: chat users should not have administrative access to vector database collections.
When deploying multiple internal services, use the general principles from this self-hosted application hardware sizing guide: resource headroom, container isolation, and measuring real workload matter more than the nominal account count.
Self-hosted AI stack hardware: network, storage, and backups
Self-hosted AI stack hardware must provide more than GPU performance: it also needs predictable I/O, secure document storage, and reliable vector database recovery.
Storage and networking without bottlenecks
A personal server only needs a 1 Gbps port and 500 GB NVMe. For a department with regular file uploads, separate object storage, and multiple nodes, use 1–10 Gbps. External chat traffic is small: a 1 000-token response is only a few kilobytes, but downloading 5–50 GB models, datasets, and backups quickly consumes bandwidth.
Do not use HDD as active storage for Qdrant or PostgreSQL. Hard drives are suitable for cold archives, but HNSW search, WAL, and segment compaction require low-latency NVMe. RAID 1 protects against a single drive failure, but it does not replace backups.
What to include in backups
Back up PostgreSQL, configurations, Qdrant collections, and source documents daily. Ollama models can be downloaded again, so they can be excluded from daily backups if tags and versions are retained. For 1 TB of active data, maintain at least 2–3 TB of backup capacity with version history.
A separate server or object storage reduces the risk of losing the database due to administrator error, ransomware, or volume corruption. Snapshot, deduplication, and capacity-planning practices are covered in the Proxmox Backup Server requirements guide.
How to test self hosted AI stack requirements
Self hosted AI stack requirements should be validated with load testing: measure tokens per second, TTFT, search latency, VRAM usage, and IOPS during concurrent requests.
Which metrics are acceptable
For internal chat, target TTFT is up to 2–4 seconds for a typical request, generation speed is from 15 tokens/s per user, and vector database search should take up to 300–500 ms without a reranker. These benchmarks depend on the model and answer quality: a 32B model will be slower than an 8B model but may handle complex instructions better.
During testing, run at least 3–5 parallel chats, test bulk document uploads separately, and monitor the queue. If VRAM is above 90% utilization and p95 TTFT grows several times over, reduce context, lower concurrency, or add a GPU.
# Monitor GPU and containers
watch -n 1 nvidia-smi
docker stats --no-stream
iostat -xz 1For production, Prometheus and Grafana are useful: collect GPU utilization, temperature, VRAM, CPU steal, free space, p95 latency, and request count. An alert at 80% disk usage lets you expand NVMe before Qdrant stops compacting segments.
Frequently Asked Questions
Can I run an LLM, RAG, and vector database without a GPU?
Yes. For 1–2 users, a CPU-only stack works on 8–16 vCPU, 32–64 GB RAM, and an NVMe drive. A 7–8B Q4 model will respond more slowly than on a GPU, often in the 3–10 tokens per second range. CPU is suitable for testing, a personal knowledge base, and periodic indexing, but not for active team chat.
How much VRAM do I need for a RAG system for 20 employees?
For 10–20 employees, a practical minimum is a GPU with 24 GB VRAM, 16 vCPU, and 64 GB RAM. This capacity can serve an 8–14B model with headroom for KV-cache and embedding tasks. If you need a 32B model, context above 16K tokens, or 5+ simultaneous generations, choose 48 GB VRAM.
Do I need a separate machine for Qdrant or pgvector?
Up to 1 million vectors, Qdrant or pgvector can run alongside the LLM on a server with 64 GB RAM and 1 TB NVMe, provided GPU inference is not CPU-bound. At 5–10 million vectors, with active reindexing or more than 20 concurrent users, it is better to move the vector database to a separate node with 64–128 GB RAM.
What disk do I need for document storage and the vector index?
For a personal RAG stack, 500 GB NVMe is usually enough; for a team with 100–200 GB of source documents, plan for 1 TB NVMe. For 1 million 768-dimension vectors, allow approximately 10–20 GB of disk space for the index and metadata. Keep at least 30% free capacity for compaction and backups.
Conclusion
For a personal RAG setup, 8 vCPU, 32 GB RAM, and 500 GB NVMe are sufficient. For a team of 10–20 people, choose 16 vCPU, 64 GB RAM, 1 TB NVMe, and a GPU with 24 GB VRAM. If the knowledge base exceeds 1 million vectors or you need a 32B model, move to 48 GB VRAM and 128 GB RAM or split the LLM and vector database across separate servers.
NVMe VPS activated in 60 seconds: full root access, 20+ locations, card or crypto payments.
Choose a plan
