AITesting a fix for inflated value estimates in reinforcement learning
A user describes a ChatGPT-assisted experiment with relative value learning, saying that centering temporal-difference errors did not improve performance on their tasks.
TLDR
One user says their Q-values—estimates of future returns—often exceed observed returns, sometimes substantially. They describe using ChatGPT to simplify relative value learning into subtracting the mean temporal-difference error from each sample’s error, but report no performance improvement on their tasks. A reply proposes a diagnostic test using cloned states: compare taking an untaken action with the highest predicted value against taking the usual action, then follow the same fixed policy and compare predicted and realized return gaps. If the predicted advantage systematically disappears, the reply argues, max-Q is feeding generalization error into its training targets.
Combined views
104
1 Source, first seen 10h ago
Testing a fix for inflated value estimates in reinforcement learning
A user describes a ChatGPT-assisted experiment with relative value learning, saying that centering temporal-difference errors did not improve performance on their tasks.
TLDR
One user says their Q-values—estimates of future returns—often exceed observed returns, sometimes substantially. They describe using ChatGPT to simplify relative value learning into subtracting the mean temporal-difference error from each sample’s error, but report no performance improvement on their tasks. A reply proposes a diagnostic test using cloned states: compare taking an untaken action with the highest predicted value against taking the usual action, then follow the same fixed policy and compare predicted and realized return gaps. If the predicted advantage systematically disappears, the reply argues, max-Q is feeding generalization error into its training targets.