AI confidence estimators can falter as model knowledge changes

A research team introduces “persistent calibration”: keeping a model’s confidence aligned with what it knows as its knowledge changes, without repeated recalibration. Testing confidence estimators trained on earlier open-model checkpoints against later ones, the team reports that inference-time and fine-tuning methods fell short on questions that exposed changes in knowledge, even when they were well-calibrated on the full question set. Training on multiple checkpoints helped, the team says.