What production deployments taught us about evaluation

Over the past year, we deployed question-answering systems inside three organizations, using their own documents and infrastructure. This note records what those deployments taught us about evaluation.

Most of our laboratory findings held up in production. The exceptions, however, were significant enough to change how we assess systems before deployment.

Evaluation data

Each deployment begins with a representative evaluation set: real questions submitted by the people who will use the system, with answers reviewed by someone familiar with the relevant material.

Fifty questions from actual staff were more informative than 500 questions written by the development team. Real users introduced ambiguities and patterns of confusion that were difficult to anticipate in the laboratory.

The evaluation set is versioned and re-run after every material change. This allows regressions to be identified before they become production incidents.

What held up

Cited answers influenced how users developed trust in the system. During the first two weeks, users examined citations closely. Once they found the citations to be accurate, they began to use the system in much the same way they would consult a careful colleague.

Refusals were equally important. A clear statement that the available documents did not support an answer contributed more to adoption than several marginal improvements in answer accuracy. Users valued the distinction between an answer supported by evidence and one that was merely plausible.

What did not

Scanned documents performed worse than our test sets had suggested. A page that is legible to a person may produce incomplete or corrupted text during extraction. A system cannot retrieve or cite information that it has not extracted correctly.

We now measure extraction quality on a sample of the actual archive before making claims about downstream answer quality.

Document freshness was another recurring problem. In several cases, the system retrieved a superseded policy. Document management and version control therefore became part of the engagement rather than an issue that could be assigned solely to the client.

Tests required before handover

Every deployment is assessed in three areas:

• Answer quality on the representative evaluation set, including human review of a sample
• Refusal behavior on questions that cannot be answered from the available documents
• Operational readiness, including whether the client's team can use the administrative interface, re-index a knowledge base, and review quarantined content without assistance

If any of these tests fails, the deployment is not complete.