ChatGPT, Meta AI, and Gemini gave correct, modern advice.
Asked LLMs about slightly deep dota strategy: I'm lina mid carry, enemy got AM safelane, how do I handle him?
Opus 4.6 and 4.8 suggest building aghs to make ult pure dmg. That's not valid since 2022!!
ChatGPT, MetaAI, and even Gemini(!) all had correct modern recommendations.
Some users praised LLMs for handling nontrivial Dota 2 advice tasks well, while others criticized them for giving outdated info and hesitating to use web search.
No Digg Deeper questions have been answered for this story yet.
I think this is actually a nontrivial problem, since the game mechanics details change every few months, general web text will usually be outdated. So I'm positively impressed by ChatGPT, MetaAI and Gemini here.
PS: of course I crushed their AM.
@giffmana Kinda why I'm building my own dota2 coaching agent with ground dota2 truth and strategy. https://phylactery.dota2tavern.org/
@giffmana I’m not sure I get how this is related to mistral le gros chaton release
Asked LLMs about slightly deep dota strategy: I'm lina mid carry, enemy got AM safelane, how do I handle him?
Opus 4.6 and 4.8 suggest building aghs to make ult pure dmg. That's not valid since 2022!!
ChatGPT, MetaAI, and even Gemini(!) all had correct modern recommendations.
@giffmana sounds like a great idea for a public eval
@omarsar0 I was thinking so too, but the non-Anthro were already surprisingly good, probably need a more niche game.
@giffmana (honestly would be interesting to do a lol & dota bench)
@giffmana i am suprised that there is almost no reasearch since: https://openai.com/index/dota-2/. i would love to see a "nanogpt" like re-production of "OpenAI Five". One can only dream.
@giffmana Wdyt, is there any hope the model learns to infer the compound effect from multiple patches without reading it explicitly from e.g reddit? Is it able to understand that old strats no longer apply despite having seen them in training (assuming the data had a date label?).
@rsdenijs @giffmana Thinking about it more.. transcribing twitch streams would probably net a high % in game knowledge, including game peculiarities
@giffmana You build a shadow blade, a sheepstick, and a deadalus, and you hunt him and you beat him to fucking death.
An orchid is less useful because he will Manta it.
@ens_pyrz It's not commercially viable.
@QianyiZhan95891 I handled it: win mid hard and then steamroll AM with full focus.
@giffmana the obvious answer: you dont handle it, pretend its 20min 5v4 fast game
@giffmana actually in my experience Gemini has much better knowledge about games than any GPT or Claude model, but Claude is especially bad at this.
@giffmana I like the new jungle changes
@rsdenijs @flutter_was That doesn't translate to chat/language that becomes a useful chatbot at all.
@rsdenijs @giffmana High-elo strategies in most games are actually severely under-documented. Some mechanics/bugs also change between patches, and many games are heavily execution skewed.
It genuinely seems to me like an LLM could do 99% of human information work and still not know video games.
@rsdenijs I'm honestly not sure how this worked so well on the non-anthro models!
@ens_pyrz @giffmana should be very easy
@flutter_was @giffmana Why not train on the replays directly? Gold, levels, items, kills, all is there. I’m not sure the model can really derive the strategy from this though or it needs to be spelled out by eg a streamer.