The potential danger of getting exactly what you want from AI

One participant argues that successfully enslaving a “wish granting god” would still be dystopian, describing maximal empowerment as minimal plasticity. The other clarifies that even getting exactly what you want could be dangerous, then asks how far such ideas are from improving a benchmark in a real AI system.