Prompting strategies that did not improve calibration
A negative result: five commonly recommended prompting techniques produced no reliable improvement in the relationship between model confidence and accuracy.
We are publishing this result because the techniques examined here are widely recommended and, in our experiments, did not work.
Five prompting strategies intended to improve calibration, the correspondence between stated confidence and actual accuracy, produced no reliable improvement in any of the configurations we tested.
Methods
We evaluated five strategies:
- Asking the model to state its confidence as a percentage before answering
- Asking the model to list reasons it might be wrong
- Using a two-pass prompt in which the model answers and then reassesses its confidence
- Framing the model as a cautious expert
- Providing few-shot examples with calibrated confidence annotations
Each strategy was evaluated on 6,000 questions covering mathematics, factual recall, and reading comprehension, across three open models.
We measured calibration using expected calibration error and the Brier score, with bootstrap confidence intervals based on 1,000 resamples.
Results
None of the strategies produced a reliable change. Expected calibration error shifted by less than half a percentage point in either direction across every strategy, model, and domain combination. No observed shift remained statistically distinguishable from noise under the reported confidence intervals.
Two techniques changed the form of the model's confidence statements. The persona prompt produced rounder numerical values, while the two-pass prompt produced higher values. Neither change improved the relationship between confidence and correctness.
A model that reports 80 percent confidence while being correct 55 percent of the time is not calibrated, regardless of how carefully the prompt is worded.
Why this matters
These techniques appear in many deployment guides. They are often adopted on the assumption that they provide epistemic discipline at minimal cost.
Our results do not support that assumption. Teams that require calibrated confidence should measure it on their own task, identify the relevant bias, and correct for it using an evaluated procedure. Prompt wording alone is not a substitute for measurement.