GPT-5.5 achieves 99.46 percent accuracy on multi-digit multiplication across a 20-by-20 grid of problems with up to 20 digits per number

I still occasionally hear people claim that LLMs are hilariously bad at arithmetic. Another reminder that it's not 2022 anymore.

@cozyblaze265065 @yuntiandeng What makes you think model wasn’t using tools on the backend?:) they can hide whatever tools they want and produce fake thinking traces:)

@olafwillocx Symmetric in digit count. So there are 7 samples of (d1, d2) = (3, 2), and 7 samples of (d1, d2) = (2, 3), but every sample is randomly generated.

@cozyblaze265065 You tested my exact March 2024 hypothesis oddly enough

@davemeyers322 @cozyblaze265065 I think you misread. It's digit.

So, 5x5 would be 63528*28360 for example.

50x50 is 50 digits times 50 digits.

this shows prove that CoT is still underrated and undervalued.

the only way this is possible is that the CoT naturally knows how to decompose the multiplication into bits that can be computed purely from "raw intuition" in semantic space.

intuitively, there is no reason this should work so well

@cozyblaze265065 This is surprising to me

1) I cant imagine that multiplying 5 by 5 would be impossible for GPT5.5 w/o reasoning

2) I cant imagine that multiplying 50 by 50 should take 100k tokens, its just 5 by 5 with two zeros added, id be interested to see the reasoning trace

@cozyblaze265065 Are the attempts symmetric? Like if you tried 123*12 for a 2-3 digit test, did you also try 12*123?

@cozyblaze265065 Does 5.5 not have access to a calculator with a tool call?

@cozyblaze265065 Llms can never learn multiplication by design. Its mathematically impossible with the way their nodes are currently constructed.

@vardtaken @cozyblaze265065 Why? They can just reason the process

@vardtaken @cozyblaze265065 “The user wants to know what 2628638271 times 826381652 is… lets start with 1 times 2. Thats 2. Now lets do 7 times 2. Thats 14….Blah blah blah

@cozyblaze265065 Does the chain of thought carry out a multiplication algorithm step by step?

@cozyblaze265065 What about Low reasoning?

@bellmantd @cozyblaze265065 i mean it's just doing the multiplication "by hand" like we would, no? Though I'm curious if it did long multiplication or partial products.

What about xhigh reasoning? For the 100x100 digits test, it came back with 1/3 correct (and a large bill💸). I believe with enough reasoning budget, it can perform long multiplication beyond 50x50, but I'll stop here since I'm seeing the bottom of my wallet. (3/4)

@PitaAndJelly @cozyblaze265065 How?

I tested medium reasoning with 30x30 and 50x50 digits multiplication, and each came back with 7/7 correct. As expected, reasoning increased with digit count. I tried to push it to 100x100 digits with no success (0/7 correct). (2/4)

@cozyblaze265065 Are you sure it didn’t sneak a python code construction in there

@cozyblaze265065 @fentasyl Chat GPT will eventually create better IQ tests than humans can create