Researcher Questions Gary Marcus On Pure LLMs Solving Novel Math Problems
Reactions from ranked influencers
4 posts
@GaryMarcus For my edification, could you unpack this a bit? 1) are you claiming that a pure auto regressive LM (i.e., no harness, no tools) cannot solve novel math problems? Or just that it's not as "good" at it as with tools/harness? 1/3
@GaryMarcus 2) if a no- harness LM writes a program or calls a solver when writing a proof, is that neurosymbolic? Is the intelligence of that system partly in the classical symbol part? Would that change if a human wrote the same program, and if so, what evidence could convince you 2/4
@GaryMarcus that the LM is meaningfully using the symbolic tool in the same way? 3) or is your claim that harnesses are the key symbolic part? If a harness like https://github.com/karpathy/llm-council that has multiple LMs answer + review each other's answers and vote for the best, and that improves 3/4
@GaryMarcus math performance, how much of the intelligence would you say is in the harness? If mathematicians did a peer review + rank process, and it helped them, would you say they are "not good at math" alone? (Sorry for the long thread, just curious about the boundaries!) 4/4