A new paper proves that imperfect world models are fundamentally exploitable, causing reinforcement learning agents to misrank policies

"Imperfect World Models are Exploitable" World models can look accurate, but still rank policies incorrectly, saying policy A is better than policy B when the real environment says the reverse. This paper formalizes that failure as model exploitation and proves it is basically unavoidable for any nontrivial, nonequivalent world model on broad policy sets. It also connects this to reward hacking and derives a safe horizon showing how model error compounds with planning depth.