PuzzleJAX Benchmark Tests LLMs on Puzzle Games

Julian Togelius posted about a talk at the IEEE Conference on Games. Co-author Smearle presented PuzzleJax, created with several collaborators. The project supplies a GPU-accelerated engine and language for benchmarking tree search, reinforcement learning, and LLM reasoning on puzzle games. The post states that LLMs perform poorly on PuzzleScript games while search performs well and reinforcement learning performs decently. An arXiv paper titled PuzzleJAX: A Benchmark for Reasoning and Learning describes the same engine and its intended uses for game generation, testing, and understanding experiments.