Local RAG Sizing: Embeddings, Vector DBs & VRAM (8B-70B)

RETRIEVAL-AUGMENTED GENERATION, MEMORY BUDGETING & LOCAL AI SILICON

Local RAG Architecture: Sizing Embedding Models, Vector DBs & Generator VRAM (8B to 70B)

# Test Methodology: Air-gapped RAG pipeline querying 500 enterprise PDF technical specs
# Vector DB: Qdrant Local (HNSW Index in Host System DDR5 RAM, 0 MB GPU VRAM)
# Workstation: Intel i9-14900K | 64GB DDR5 | Single RTX 4090 24GB | Ubuntu 24.04 LTS

[Pipeline Phase 1] Query Embedding Ingestion:
    • Model: BAAI/bge-m3 (Dense + Sparse Multi-Lingual, FP16)
    • VRAM Footprint: 1.18 GB
    • Ingestion Latency: 14.2 ms

[Pipeline Phase 2] Initial Vector Retrieval & Cross-Encoder Reranking:
    • Candidates Retrieved: Top-25 Chunks from Qdrant
    • Reranker Model: BAAI/bge-reranker-large (Cross-Encoder, FP16)
    • Reranker VRAM Footprint: 1.34 GB
    • Re-ranking Latency: 42.8 ms (Top-5 Chunks selected)

[Pipeline Phase 3] Primary Generator Model & Augmented Context Synthesis:
    • Generator Model: Qwen/Qwen2.5-Coder-14B-Instruct (AWQ 4-bit)
    • Static Weights Footprint: 9.42 GB
    • Retrieved Context Buffer: 8,192 tokens (5 retrieved chunks + system prompt + conversation)
    • Dynamic KV Cache Allocation (vLLM PagedAttention, FP16): 1.82 GB
    • CUDA Context & PyTorch Overhead: 1.35 GB

========================================================================================
[Total Pipeline VRAM Footprint]: 1.18 + 1.34 + 9.42 + 1.82 + 1.35 = 15.11 GB
[Remaining RTX 4090 Headroom]: 8.89 GB (Safe operational buffer for concurrent queries)
[End-to-End Pipeline Latency]: 312 ms (Sub-second retrieval to first generated token)

1. The 4 Co-Existing Memory Consumers of an Offline RAG Stack

When transitioning an application from cloud APIs to an air-gapped on-premise workstation, memory budgeting must account for the entire computational graph. Unlike conversational chat where a single LLM owns the entire GPU memory pool, a production RAG system demands concurrent memory for four distinct architectural layers:

For detailed calculations on base generator parameter footings, cross-reference our capacity planning audit on Exact VRAM & GPU Sizing for 8B, 14B, 32B & 70B LLMs.

2. Dense Embedding Models: Dimension, Context & VRAM Allocations

Dense embedding models convert raw text chunks into continuous vector representations. While their parameter counts are modest compared to modern LLMs, their memory footprint during inference cannot be ignored:

• Nomic Embed Text v1.5 (137M parameters): A lightweight workhorse operating with an 8,192 token context window. In standard FP16 precision, its static weights occupy approximately 0.55 GB of VRAM. During batch vectorization of incoming queries, its transient activation footprint rarely exceeds 0.25 GB.
• BAAI/bge-m3 (567M parameters): The enterprise standard for multi-lingual and hybrid dense-sparse retrieval. Sized at 567M parameters in FP16, it consumes roughly 1.18 GB of static VRAM. When processing long document contexts up to 8k tokens, its dynamic memory spikes to 1.8 GB – 2.1 GB.
• GTE-Qwen2-7B-Instruct (7 Billion parameters): High-end dense embedding models based on full LLM backbones deliver unprecedented semantic matching, but require ~14.5 GB in FP16 or ~4.8 GB in 4-bit quantization. For consumer workstations, running a 7B embedding model alongside a 14B or 32B generator model is mathematically impossible on a single 24GB GPU.

3. Cross-Encoder Rerankers: The Hidden Latency & Memory Tax

Bi-encoder embedding models calculate embeddings for documents and queries independently via cosine similarity. While ultra-fast, this approach misses subtle cross-attention nuances.

A Cross-Encoder Reranker takes the user query and the top-K retrieved document candidates, feeding both into the transformer simultaneously. This enables full token-to-token attention across query and document, boosting retrieval precision (NDCG@10) by 15% to 30%.

However, rerankers carry a dual penalty:

1. Memory Footprint: Flagship models like bge-reranker-large (560M parameters) require approximately 1.34 GB of VRAM in FP16.
2. Compute Latency: If you pass 50 candidate chunks into a cross-encoder, the model must execute 50 individual forward passes. To maintain sub-500ms pipeline latency, cross-encoders should be restricted to evaluating between 15 and 25 candidate chunks, or replaced with ultra-fast CPU-based quantized rerankers like FlashRank.

4. Vector Database Architecture: System RAM vs. GPU VRAM (Qdrant vs. Chroma)

A persistent myth among machine learning engineers is the belief that vector databases (such as Qdrant, ChromaDB, Milvus, or FAISS) require substantial GPU VRAM.

In local workstation deployments, vector databases reside almost entirely in host system RAM (DDR4/DDR5) and NVMe storage, consuming zero GPU VRAM:

Unless you are running specialized GPU-accelerated nearest-neighbor libraries (like FAISS-GPU or RAPIDS cuVS) across 50 million+ vectors, your vector database will not compete with your LLM for graphics memory.

5. Generator Sizing (8B to 70B): Balancing Quantized Weights & Retrieved Context

In conversational chat, prompts average 200 to 500 tokens. In Retrieval-Augmented Generation, injecting 5 to 10 retrieved chunks immediately expands your prompt buffer to 4,000 to 16,000 context tokens.

As demonstrated in our deep dive on Context Window VRAM Scaling from 32k to 128k, the Key-Value (KV) cache memory expands linearly with every retrieved token:

• 8B Parameter Generator (Llama 3.1 8B Q4): Static weights occupy ~5.2 GB. An 8k retrieved context window requires ~1.0 GB of FP16 KV cache. Total VRAM footprint sits at ~7.5 GB (comfortably viable on 12GB GPUs).
• 14B Parameter Generator (Qwen 2.5 Coder 14B Q4): Static weights occupy ~9.5 GB. A 16k retrieved context buffer requires ~2.2 GB of KV cache. Total VRAM footprint sits at ~13.2 GB. Adding an embedding model and reranker brings total pipeline demand to ~16.8 GB (the undisputed sweet spot for 24GB GPUs).
• 32B Parameter Generator (Qwen 2.5 32B Q4): Static weights occupy ~19.8 GB. Adding a 16k retrieved context window (~3.8 GB) and pipeline overhead demands ~25.8 GB total VRAM—causing an out-of-memory crash on a single 24GB RTX 4090 unless KV cache quantization (FP8) is strictly enforced. Compare quantization formats in our Comprehensive AWQ vs. EXL2 vs. GGUF Guide.
• 70B Parameter Generator (Llama 3.3 70B Q4): Static weights occupy ~42.8 GB. A 32k retrieved context buffer demands ~10.7 GB of KV cache. Total requirements sit at ~56.5 GB, demanding a multi-GPU array or high-RAM unified memory workstation.

6. Co-Location vs. Dual-Device Topologies: Single GPU vs. Multi-GPU Setups

When engineering your workstation hardware, you must choose between two pipeline placement strategies:

Topology A: Monolithic Single-GPU Co-Location (e.g., Single RTX 4090 24GB)

All components (Embedding, Reranker, Generator) share a single physical VRAM pool. To prevent OOM crashes, models must either remain permanently resident in memory (demanding strict parameter limits like 14B Q4) or be dynamically loaded and unloaded sequentially using libraries like Ollama. However, dynamic swapping introduces an unacceptable 2 to 4-second cold-load latency penalty per query.

Topology B: Asymmetric Multi-GPU Pipeline Partitioning (e.g., Dual RTX 3090 48GB)

GPU 0 (24GB) hosts the embedding model, reranker, and auxiliary tools permanently resident in VRAM. GPU 1 (24GB) is dedicated entirely to the generator model and its active KV cache. Alternatively, both cards pool 48GB via tensor parallelism to host massive 70B models. For bus bandwidth considerations when clustering dual consumer GPUs, review our audit on PCIe 4.0 x8/x8 vs x16/x16 for Dual GPUs.

To compare on-premise hardware capital expenditures against cloud rental pricing across H100 and A100 instances, explore our comprehensive Cloud GPU Pricing Comparison Guide.

8. Production Python Blueprint: Shared Memory Air-Gapped RAG Pipeline

To deploy a dependable, zero-OOM local RAG pipeline, use this asynchronous Python implementation. It orchestrates a local vector database, a cross-encoder reranker, and an OpenAI-compatible local serving engine (such as vLLM or Ollama), enforcing strict memory separation:

from sentence_transformers import SentenceTransformer, CrossEncoder
from qdrant_client import QdrantClient
from openai import OpenAI
import torch

# 1. Initialize Vector DB (Host System RAM - 0 GB GPU VRAM)
qdrant = QdrantClient(location=":memory:")

# 2. Load Embedding & Reranker onto GPU (Restricted Memory Footprint)
device = "cuda" if torch.cuda.is_available() else "cpu"
embed_model = SentenceTransformer("BAAI/bge-m3", device=device)
reranker = CrossEncoder("BAAI/bge-reranker-large", device=device)

# 3. Connect to Local High-Throughput Generator (vLLM on localhost:8000)
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

def run_local_rag(query: str, top_k_retrieve=20, top_n_rerank=5):
    # Phase A: Dense Vector Search
    query_vector = embed_model.encode(query)
    search_results = qdrant.search(
        collection_name="enterprise_docs",
        query_vector=query_vector,
        limit=top_k_retrieve
    )
    candidate_chunks = [hit.payload["text"] for hit in search_results]

    # Phase B: Cross-Encoder High-Precision Reranking
    pairs = [[query, chunk] for chunk in candidate_chunks]
    scores = reranker.predict(pairs)
    scored_chunks = sorted(zip(candidate_chunks, scores), key=lambda x: x[1], reverse=True)
    top_context = [chunk for chunk, score in scored_chunks[:top_n_rerank]]

    # Phase C: Generator Synthesis with Augmented Context Buffer
    augmented_prompt = f"""Context Information:
{"\n---\n".join(top_context)}

User Query: {query}
Answer concisely using only the context provided above."""

    response = client.chat.completions.create(
        model="qwen2.5-coder-14b-instruct",
        messages=[{"role": "user", "content": augmented_prompt}],
        temperature=0.1,
        max_tokens=512
    )
    return response.choices[0].message.content

When combining this retrieval workflow with autonomous multi-step software development loops, consider deploying cyclic graph state machines as detailed in our guide to LangGraph vs. CrewAI vs. AutoGen for Local Coding Agents.

9. Frequently Asked Technical Questions

What is the primary memory bottleneck in a local offline RAG pipeline?

The primary memory bottleneck is not the vector database, but the simultaneous coexistence of multiple neural models: the embedding model, the cross-encoder reranker, and the primary LLM generator with its expanded retrieved-context KV cache. Failing to account for the cumulative VRAM of all pipeline components causes instant CUDA Out-Of-Memory exceptions.

How much VRAM does an embedding model and reranker consume?

Standard dense embedding models (like BAAI/bge-m3 or nomic-embed-text) consume between 0.6 GB and 2.2 GB of VRAM in FP16. Cross-encoder rerankers (like bge-reranker-large) consume an additional 1.2 GB to 2.4 GB during batch scoring, requiring roughly 2.5 GB to 4.5 GB of dedicated VRAM before the generator model even initializes.

Do vector databases like Qdrant, Chroma, or Milvus require GPU VRAM?

In the vast majority of local developer setups, vector databases run entirely in host system RAM (DDR4/DDR5) and NVMe storage using HNSW (Hierarchical Navigable Small World) indexing. Unless you are running GPU-accelerated index algorithms like cuVS or FAISS-GPU across tens of millions of vectors, vector databases consume 0 GB of GPU VRAM.

Can an 8B model with local RAG run reliably on an 8GB or 12GB GPU?

Yes, with strict quantization constraints. An 8B model quantized to Q4 requires ~5.2 GB of VRAM. Pairing it with a lightweight embedding model like Nomic v1.5 (0.5 GB) and an 8k retrieved context window (1.0 GB KV cache) totals approximately 7.8 GB of VRAM, making it fully functional on an RTX 3060 12GB or RTX 4060.

✓

Abdulrahman Maslmany

Lead Productivity & Systems Architect

"Retrieval-augmented intelligence is a pipeline engineering challenge, not an isolated model question. I architect memory-bounded embedding topologies, cross-encoder rerankers, and air-gapped inference clusters."

Calculate your system VRAM allocations live on our Local LLM VRAM Calculator, inspect datasets on Hugging Face, or read our complete Editorial Standards.