Off the clock · Jan 2026
Building a document Q&A platform and its first eval suite
A self-hosted RAG app over long documents that grew from plain vector search into hybrid search with a reranker, one eval run at a time, and what the judge made me keep or drop.
Can I just ask the manual?
As a first personal project, I started with a naive RAG platform. I wanted something that would give me easy answers from long documents when asked questions in natural language. Working on it introduced me to embeddings, semantic search, RAG and evals. This was also my first eval suite.
Long documents hide answers. A 400-page PLC hardware manual, a book, a dense employment agreement: Ctrl+F finds a word but not an answer, and a general chatbot gives a fluent answer without saying whether it actually came from the document. I wanted to ask a normal question, get a direct answer, see which page each claim came from, and know with numbers whether a change made the answers better or worse.
TODO(tarang): what prompted it, and why these documents (the FX5U manual, Atomic Habits, a separation agreement, Nonviolent Communication).
Today it's a web app that searches a document by meaning and by keyword, reranks what it finds, and has Gemini answer with citations back to the page, with an eval suite next to it where another Gemini model grades the answers. The last run on 77 questions across three documents scored 4.62 out of 5, with a typical answer arriving in about 5 seconds.
How it works today
flowchart TD
A["Upload a document: PDF, Word, web page or text"] --> B["Split into chunks along tables, lists and paragraphs, tagged with document, page and section"]
B --> C["Gemini turns each chunk into an embedding"]
C --> D[("Qdrant: search by meaning")]
C --> E[("Keyword index")]
F["A question"] --> G["Gemini rewrites it two ways and splits hard ones into parts"]
G --> H["Search by meaning and by keyword, merge the rankings"]
D --> H
E --> H
H --> I["Small local model reranks the best matches"]
I --> J["Drop duplicates and noise, keep the strongest few"]
J --> K["Gemini Flash-Lite writes the answer with numbered citations"]
K --> L["Answer streams in, each citation opens its page"]
M["Benchmark questions with reference answers"] --> G
K --> N["Gemini judge scores each answer out of 5"]
Reading the document
I pull the text out page by page and cut it into chunks that follow the layout, because so many of my answers lived in tables and lists: a table or a list stays whole, and prose is split on paragraphs. No chunk crosses a page boundary, since I wanted every citation to point at the exact page. Each chunk is labelled with its document, page and section, embedded by Gemini into Qdrant, and also put in a plain keyword index. I wanted the keyword side mostly for the hardware manual, where an answer can hang on an exact part number like FX5U-32MR/ES that a search by meaning can blur.
Answering a question
A question first goes to a fast Gemini model that writes two paraphrases and breaks a complex question ("how do I wire the inputs and what are the power specs?") into smaller ones that get answered separately and then merged. Each version is searched by meaning and by keyword, and I merge the two rankings by position rather than by score, so I never have to make two very different kinds of score agree. A small reranker then rereads the top candidates against the question. I pay for Gemini's embeddings because they made the biggest difference in my evals, but the reranker is small enough to run on my own CPU for free. After a few filters, what survives goes to Gemini Flash-Lite as numbered sources with their pages, and I ask it to use only those and cite them inline. Every citation in the answer turns into a small numbered pill that opens the page it came from.
The eval suite
Every benchmark question has a reference answer. For three documents I got these by asking Claude.ai the same questions with the document attached, and for Nonviolent Communication I had Gemini write both the questions and the answers with the whole book in front of it. The main test is 77 questions with all three of the other documents in one index, so it also has to find the right document.
Every question goes through the live pipeline, and a Gemini judge scores each answer from 1 to 5 on relevance, completeness, accuracy and organisation, alongside how close it is to the reference and how long it took. Every run is saved with its full settings, so I can see exactly what changed between two runs.
How I got here
Most of it happened in five weeks between late January and late February 2026, about 76 commits. For the first ten days I added retrieval features without measuring any of them. From the start of February I had an eval suite, and from then on every change had a number next to it and was kept or dropped because of that number. The timeline below shows retrieval growing from a single vector search into hybrid search, fusion, reranking and filtering, and which pieces came and went.
1. Naive RAG
The first version was the textbook one. Upload a PDF, pull out the text, cut it into 1000-character pieces, embed each piece with a small model running locally, and store them in Qdrant. A question was embedded the same way, the ten closest pieces came back, and Gemini Flash answered from them with a four-line prompt. The whole engine was about 140 lines, and the next day it got a React chat frontend. It worked, in the sense that it answered, but I had no way to tell whether an answer was good except reading it.
2. Keyword search and a reranker
Also on day two, I added a keyword index next to the vectors, a cross-encoder reranker, and a guard that refused to answer when the best match scored too low. The two searches were combined with a simple weighted average of their scores. My only written reason was "better recall than pure semantic search", which is true in general, but I had nothing to check it against for my documents.
TODO(tarang): what made you add keyword search and a reranker on day two? The part numbers in the FX5U manual?
3. Rank fusion and paraphrases
A week later I borrowed three ideas from a project called QMD. Gemini would write two paraphrases of every question and each would be searched, with the original counting double. The weighted average was replaced by reciprocal-rank fusion, which merges the two lists by position instead of by score, so I no longer had to make a similarity score and a keyword score mean the same thing. I even wrote down "expected improvement" percentages for each. They were guesses.
TODO(tarang): one line on what QMD is, for attribution.
The next day I switched the reranker off, because the backend kept timing out at startup while downloading it. So by the end of January I had added a lot and could not say whether any of it helped.
4. The first eval suite
This was the turning point. I wanted to stop guessing, so I built an eval suite that measured my answers against Claude.ai's answers to the same questions, starting with just four questions on the manual, and that same day I ran one change after another through it.
The first thing it caught was a bug. Turning the reranker back on made the answers worse, not better: the reranker's scores live on a completely different scale, so the low-confidence guard kept firing, and the app said "I don't have enough information" to questions it could answer. With that fixed, it scored above the baseline and stayed on.
The biggest jump was swapping the small local embedding model for Gemini's, which took similarity to the reference answers from roughly 0.72-0.8 to about 0.9 on all three documents. Each side was scored with its own embedder, so part of that jump is a different ruler, but it was clear enough and I kept it.
Then I tried semantic chunking, which cuts the text wherever the meaning shifts instead of at a fixed length. It costs a second embedding pass on every upload, and it lost to plain fixed-size chunks on the manual and tied or lost on the other two. It hadn't earned its cost, so I switched it off. Splitting complex questions into sub-questions did the opposite: it lifted the manual's score noticeably, barely touched the other two, and stayed on.
5. Tuning to the judge
Similarity said my answers were close to Claude's. The judge disagreed. It gave relevance 4.0 but completeness 2.3 out of 5: the answers were on topic but thin, missing amounts, conditions and the secondary details that make an answer actually useful.
So I went after that one number in three rounds: boosting tables, lists and specs when a question asked for one, filtering out noise, keeping more sources when they were close to the best match, and telling the model never to claim something is missing unless the source says so. Completeness went from 2.0 to about 2.8 to about 3.2. Then the question sets grew from about ten to 80, and the first full judge run averaged 4.49. Then I built the 77-question set with all three documents in one index, and it scored 2.88. The pipeline wasn't the problem: the eval matched each question set to its document by a fuzzy name match, and the manual's questions were being searched against Atomic Habits. With that fixed, and proper table extraction added at the same time, it scored 4.72. That was the first time the eval lied to me, and it's when I started saving every run with its full settings.
This stretch also had a mistake the evals couldn't catch. I added "adaptive alpha", which searched with three different weightings between meaning and keywords and kept the best. But rank fusion had been the default for two weeks, and rank fusion never looks at that weighting. So it ran three identical searches and kept the first. A later change made it faster at "equivalent quality", which in hindsight was the clue: of course it was equivalent.
Alongside, I added retries for when Gemini was overloaded (those errors were landing in eval scores as text), thumbs up and down, and chat history.
6. Making it fast
By now an answer took over 20 seconds, and I wanted it faster without losing what the judge had given me. For a fair before and after, I put the engine from early February back and ran the 77 questions: 3.69 out of 5 at a median of 23 seconds. Then I made nine changes, mostly about not waiting for things one at a time, like one model call instead of three and answering sub-questions in parallel.
That brought it to 16 seconds with the score slightly up, at 3.81. A faster run came in at 7.7 seconds for 0.14 fewer points, and the next day Flash-Lite became the answer model and has stayed there.
TODO(tarang): confirm the Flash-Lite switch was the latency trade, and whether 7.7 s vs 16 s mattered in use.
All three of those runs scored the separation agreement around 2 out of 5, while the manual sat above 4 and Atomic Habits near 5. Not one retrieved chunk came from the agreement, which turned out to be the eval lying a second time.
7. Chunks that follow the layout
The completeness rounds had added structure detection after retrieval. Now I moved it into chunking itself, so that a table or a list would never be cut in half in the first place. Unlike the semantic chunker, this one just reads the layout, with fixed-size pieces kept as the fallback.
The first run on it scored 3.76, and then I found out why the score was still low: the separation agreement wasn't uploaded to that instance at all, so a third of the questions had nothing to find. With it uploaded, the same code scored 4.62 out of 5 at about 5 seconds. On the other two documents the new chunker was a small gain and a wash, and most of the speed came from Flash-Lite. So the dips in the middle of February were mostly a missing document, not the pipeline. Twice, my eval setup moved the score more than any RAG change did. A fourth book, with Gemini-written questions, scored 4.58 on its first run.
8. Logins and hardening
The last stretch made it something other people could log into. I made logins optional: switched on, there are users, roles and a login page, and switched off it's a single-user app. The idea, from my very first plan, was one codebase for both a business version and a consumer one. Around it went rate limits and the usual security basics, a deploy script for a VPS, a redesigned UI and about 180 unit tests.
TODO(tarang): was the business/consumer split aimed at a real deployment, and did it ever run on the VPS?
Looking back
The biggest thing this made me realise is that my eval suite was a product too, and it could be wrong. It caught a real bug on its first day and talked me out of semantic chunking, but its two biggest swings had nothing to do with RAG: a fuzzy name match and a missing document. The judge also rewards long answers, and my prompt drifted towards "be exhaustive" to keep it happy.
Reading the code again later, I found things the evals couldn't have shown me. I only ever scored the final answers, because none of the questions list the pages their answers live on. The way the fusion is wired, the keyword index can reorder what vector search found but can't add a chunk it missed, which was the whole reason for having it. If I started again, I'd write down the expected pages for every question on day one and score retrieval on its own, before touching anything else.
TODO(tarang): whether you still use it, or anything you'd add about what you'd do differently.