Internal Handbook Q&A With Hybrid Search and Source Citations
Built a RAG pipeline that cites its sources instead of just generating plausible-sounding text.
Loading live demo — first visit can take a minute…
The Challenge
Most "chat with your docs" demos either hallucinate answers that aren't actually in the source material, or use plain vector search alone, which misses exact keyword matches (a policy name, a specific term) that a pure embedding similarity search can rank too low. Businesses wanting an internal-knowledge-base bot need answers they can actually trust, not confident-sounding guesses.
The Solution
I built a hybrid retrieval pipeline: dense embeddings (bge-small-en-v1.5) and BM25 keyword search run in parallel over heading-aware chunked docs, merged with reciprocal rank fusion, then reranked with a cross-encoder (ms-marco-MiniLM-L-6-v2) down to the top 5 most relevant chunks. A local LLM (Qwen2.5) answers strictly from those 5 chunks, a system prompt forces it to say "Not found in the docs" when the answer isn't there, and the UI shows the source file and heading for every chunk the answer drew from, so every claim is traceable back to the actual document. The demo runs over a real public company handbook (communication norms, values, and compensation policy) rather than placeholder text.
The Results
The retrieval side is verified against the real bundled corpus with an automated test, not just eyeballed. The whole pipeline runs on free-tier CPU hosting with no paid API keys, and citations mean an answer's grounding is checkable at a glance instead of taken on faith. Ask it something specific to the handbook — e.g. "What is the Power of the Pause?" — and it answers from the actual policy text, not generic knowledge.
Tools & Technologies Used
How To Create This
- Chunk source docs by heading (not fixed token windows) so each chunk stays semantically coherent
- Embed chunks with a small sentence-transformer and index them in an in-memory Chroma vector store
- Build a parallel BM25 keyword index over the same chunks
- On a query, run dense + BM25 retrieval in parallel and merge the two ranked lists with reciprocal rank fusion
- Rerank the fused candidates with a cross-encoder to get the true top-5 most relevant chunks
- Prompt the LLM to answer strictly from those 5 chunks, refusing when the answer isn't present
- Display the source file and heading for each chunk the answer cites, next to the answer
Related Case Studies
AI-Powered Ticket Routing From a Single Slack Mention
An AI agent that reads a plain Slack @mention and turns it into a fully assigned ClickUp ticket, across IT, EA, and other departments, with no form to fill out.
Closed-Loop Sales Automation System Across Calendly, Close CRM, ClickUp & Slack
A suite of connected n8n workflows that carry a booked call all the way through pre-call prep, CRM sync, audit QA, and daily reporting, so no rep has to touch five separate tools by hand.
Automated Recruitment Pipeline Across Workable, Culture Index, ClickUp & Slack
The recruiting team's hiring pipeline as one connected suite of n8n workflows, from a candidate's first application through personality-assessment scoring, Slack-native review, and daily reporting, with no data re-entered by hand.