Meta's Lucas Beyer tests LLMs on shifting Dota 2 strategy, finding Claude recommended item builds outdated since 2022

@giffmana sounds like a great idea for a public eval

@omarsar0 I was thinking so too, but the non-Anthro were already surprisingly good, probably need a more niche game.

@giffmana I like the new jungle changes

@giffmana Wdyt, is there any hope the model learns to infer the compound effect from multiple patches without reading it explicitly from e.g reddit? Is it able to understand that old strats no longer apply despite having seen them in training (assuming the data had a date label?).