Testing a fix for inflated value estimates in reinforcement learning

One user says their Q-values—estimates of future returns—often exceed observed returns, sometimes substantially. They describe using ChatGPT to simplify relative value learning into subtracting the mean temporal-difference error from each sample’s error, but report no performance improvement on their tasks. A reply proposes a diagnostic test using cloned states: compare taking an untaken action with the highest predicted value against taking the usual action, then follow the same fixed policy and compare predicted and realized return gaps. If the predicted advantage systematically disappears, the reply argues, max-Q is feeding generalization error into its training targets.