Building a RAG helpdesk that cites its sources

A helpdesk bot that sounds confident but invents answers is worse than no bot. When I built an enterprise IT helpdesk agent, the goal was simple: every answer must point to a document a human wrote.

The pipeline

1. Ingest knowledge-base articles and split them into overlapping chunks.
2. Embed each chunk and store it in a vector database (Qdrant).
3. Retrieve the top candidates for a question.
4. Rerank them with a cross-encoder so the best three really are the best three.
5. Generate an answer from only those chunks, and return the chunk ids as citations.

Why reranking matters

Vector search finds similar text, not necessarily relevant text. A question about "VPN drops on Wi-Fi" can pull in a page about "VPN setup on Wi-Fi". A reranker reads the question and each candidate together, which fixes most of these near-misses.

A prompt that refuses to guess

Answer ONLY from the context below.
If the context does not contain the answer, say you don't know
and offer to create a ticket. Cite sources as [KB-id].

That last line is the product: when the bot is unsure it escalates instead of improvising.

What I'd do next

• Add feedback buttons and log the questions that end in "I don't know".
• Use those logs to find gaps in the knowledge base.
• Stream tokens so answers feel instant.

Trust comes from showing your work. Citations are cheap to add and change how users feel about the whole system.