9h ago

Follow-up study presented at ACL 2026 shows reasoning large language models solve multiple-choice questions without question text by relying on memorized answer patterns from options

Behavior matches prior findings in standard models and human test-taking strategies.

——0——
Original post
OPalth0u🧶#1114alth0u🧶@ALTH0U

not surprising i took 10 practice SATs and it became like this

8:06 AM · May 19, 2026 View on X
QUOTE POSTgavin leech (Non-Reasoning)#1480gavin leech (Non-Reasoning)@GLEECH

memorising question-answer pairs: i sleep

memorising just the answers and inferring the masked question from them alone: REAL SHIT

Nishant BalepurNishant Balepur@NishantBalepur

🚨 New Paper! 🚨 One of my first Ph.D. papers found that LLMs can answer multiple-choice questions without seeing the question 🤔 At #ACL2026, I'm presenting a follow-up showing that current reasoning LLMs can still do this! And quite similarly to a clever test-taker 🧑‍🎓🧵

1:02 PM · May 18, 2026 · 813.7K Views
10:51 PM · May 19, 2026 · 2K Views