Retrieval over engineering project data
Ten years of project documents lived in a system that could only find them if you already knew where they were. This is the retrieval layer I built on top of it.
The problem
The document control platform held roughly 125,000 documents across 1,500 projects, with 350,000 versions behind them. Everything was there. Finding it was the problem.
Search meant knowing the document number, or the project, or which discipline had produced it — and usually all three. Anyone who didn't already know asked a colleague who did. A typical "has this been done before, and where is it" question cost about fifteen minutes of manual searching, and the answer depended entirely on who you asked.
That is the expensive failure mode. Not that the data was missing, but that a decade of institutional knowledge was only reachable by people who already had it.
Why ordinary search didn't fix it
The obvious first move — better keyword search — falls down on this corpus for reasons specific to it:
- The text isn't in the database. Content lives inside PDFs, drawings and office documents, not in indexed columns. Metadata search only ever searched the filing, never the substance.
- Every document exists many times. Revision chains mean one deliverable is thirty near-identical versions. Naive search returns all thirty, ranked arbitrarily, and the reader has to work out which one is current.
- Access is not uniform. What a given user may open depends on project and role. Search that ignores this is a data leak with a nice interface.
So the requirement was never "add a chatbot." It was: answer a question in natural language, from the actual content, honouring permissions, and say which document the answer came from.
What I built
Two paths that share one index — ingestion on a schedule, retrieval per request.
Ingestion pulls documents, extracts text, splits it, embeds it, and writes vectors plus metadata to the index. Retrieval takes a user's question through the API, fetches candidate chunks, and composes an answer that cites its sources.
Decisions that mattered
Hybrid retrieval, not pure vector. Engineering documents are full of exact identifiers — document numbers, tag numbers, revision codes. Semantic similarity is actively bad at these: a query for a specific document number returns things that are about similar topics rather than the one exact match. Combining vector search with keyword matching covers both the "find me something like this" and the "find me exactly this" cases.
Permission filtering during retrieval, not after. The filter is applied when candidates are fetched, so the model is never handed a chunk the caller could not already open. Filtering after generation is not a fix — by then the content has already been read, summarised, and potentially leaked into an answer.
Chunking follows document structure. Fixed-size windows cut across section boundaries and produce chunks that read as fragments. Splitting on the document's own structure keeps each chunk self-contained enough to be quoted.
Version awareness is part of retrieval, not the prompt. The current revision is what the user means unless they say otherwise. Deciding that at retrieval time is deterministic; asking the model to reason about revision numbers is not.
Citations are mandatory. Every answer links back to a source document and version. This is what makes it usable in an engineering context — nobody is going to act on an unattributed claim about a deliverable, nor should they. It also makes the system falsifiable: when it's wrong, you can see immediately why.
What I'd do differently
The first version chunked on fixed windows because it was fast to build, and the retrieval quality was mediocre in a way that took a while to diagnose — the answers looked plausible, which is the worst failure mode to debug. I'd build the evaluation set before the pipeline next time, not after. Being able to measure "did this change make retrieval better" is worth more than any individual retrieval improvement.