Reward-rate RL reportedly reaches lower NanoGPT pretraining loss faster than vanilla RL

A user says reward-rate RL reached lower pretraining loss faster than vanilla RL in NanoGPT results. They suggest time-aware RL could help research agents decide how to spend a limited experiment budget. In a follow-up, the user says the work was inspired by the nearly three-decade-old paper “Between MDPs and Semi-MDPs” by Richard Sutton, Doina Precup and Satinder Singh.