What production deployments taught us about evaluation
Notes from running question answering systems inside real organizations: what held up, what did not, and what we now test before handover.
Over the past year we put question answering systems into production inside three organizations, on their own documents and their own infrastructure. This note collects what the deployments taught us about evaluation, because most of what we believed in the lab turned out to be true, and the exceptions were expensive.
Where the numbers come from
Every deployment starts with a golden set: real questions from the people who will use the system, with answers checked by someone who knows the material. Fifty questions from actual staff beat five hundred written by us, because the distribution of real confusion never looks like the distribution we imagine. The set is versioned and re-run on every change, before anyone is allowed to notice a regression by accident.
What held up
Cited answers changed how people trusted the system. Users checked citations heavily for the first two weeks, found them accurate, and then settled into trusting the system the way they trust a careful colleague. Refusals mattered as much as answers: an honest "the documents do not say" did more for adoption than any accuracy improvement we shipped.
What did not
Scanned documents were worse than our test sets suggested. A page that looks readable to a person can yield garbled text, and the system cannot cite what it cannot read. We now measure extraction quality on a sample of the actual archive before promising anything about answer quality. Stale documents were the other surprise: the system cited a superseded policy often enough that we now treat document hygiene as part of the engagement, not the client's problem.
What we test before handover
Three things, every time. Answer quality on the golden set, with a human reading a sample. Refusal behavior on questions the documents cannot answer, because a system that always answers is worse than one that sometimes declines. And operations: the client's team runs the panel, re-indexes a knowledge base, and reviews a quarantine queue without us in the room. If any of the three fails, the deployment is not done.