How to build a RAG pipeline for question answering

What RAG is and why it matters

Retrieval-augmented generation (RAG) lets a large language model (LLM) answer questions using information it was never trained on. Instead of relying only on the model's parameters, you retrieve relevant passages from your own data and place them in the prompt. This reduces hallucination, keeps answers current, and lets you cite sources.

A RAG pipeline has two phases. The ingestion phase converts documents into searchable vectors. The query phase retrieves the most relevant vectors and feeds them to the model.

Prerequisites

• Python 3.10 or later
• An embedding model and an LLM (hosted API or local)
• A vector store; this guide uses PostgreSQL with the pgvector extension

Steps

1. Set up the project and dependencies

Create an isolated environment and install the core libraries.

python -m venv .venv
source .venv/bin/activate
pip install pgvector psycopg[binary] numpy

Enable the vector type in PostgreSQL:

CREATE EXTENSION IF NOT EXISTS vector;

2. Load and chunk documents

LLMs have a context limit, so split documents into overlapping chunks of roughly 300-800 tokens. Overlap preserves context across boundaries.

def chunk(text, size=800, overlap=100):
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i+size]) for i in range(0, len(words), step)]

3. Generate embeddings

An embedding maps text to a numeric vector where similar meanings sit close together. Call your embedding model for each chunk and collect the vectors.

vectors = embed_model.embed([c for c in chunks])  # list of float arrays

4. Store vectors in an index

Create a table with a vector column and insert each chunk with its embedding.

CREATE TABLE docs (id serial PRIMARY KEY, content text, embedding vector(1536));

Add an approximate-nearest-neighbor index for fast search:

CREATE INDEX ON docs USING hnsw (embedding vector_cosine_ops);

5. Retrieve relevant chunks at query time

Embed the user's question with the same model, then find the closest chunks by cosine distance.

SELECT content FROM docs ORDER BY embedding <=> %s LIMIT 5;

6. Assemble the prompt and call the LLM

Concatenate the retrieved chunks into a context block and instruct the model to answer only from that context.

Use only the context below to answer. If unsure, say you don't know.
Context:
{retrieved_chunks}
Question: {question}

7. Verify and tune retrieval quality

Test with questions whose answers you know. If answers are wrong, inspect which chunks were retrieved before blaming the model.

Verification

Ask a question whose answer appears in a known document. Confirm the retrieved chunks contain the source text and that the final answer matches. Then ask something not in your corpus and confirm the model declines rather than inventing an answer.

Next Steps

Add metadata filters, experiment with chunk size and overlap, introduce a reranking step, and log retrieved sources so answers are auditable.