Daniel Litt, University of Toronto mathematician, says AI excels at local optimizations but cannot create novel concepts

The problem with using and evaluating LLMs for any work is that the intuition “AI can do tasks of type X but not of type Y” is completely wrong.

It can do task X perfectly for 99% of the domain and fail completely at 1%. For that part of the domain it can never find the right answer no matter how many times it tries.

That 1% isn’t harder and doesn’t look different to a human. It just happens to be less like the training data in some vector representation the AI “learned”.

So you start using the AI for tasks of type X. Get amazed at its performance. Start delegating that sort of task to the AI.

Then it fails catastrophically and surprises you and sabotages your work.

It does that **for every single type of task**. Just at different rates. The only exception is domains it has completely memorized from training.

For tasks combining multiple steps the domain coverage shrinks exponentially with the number of steps.

But even at very long tasks with thousands of steps it will do amazingly well for a small part of the domain. And you will look at the impressive results and be tempted to generalize.

But the generalization will never work. Or worse, it will seem to work for a while then fail.

You will then be tempted to see it as some random failure. Humans make mistakes too!

But it isn’t random.

It’s what happen when something actually didn’t “learn” anything except a large number of superficial heuristics that happen to coincide with the actual algorithm in many cases.

“Machine learning” isn’t actual learning. “Artificial intelligence” isn’t actual intelligence.

It’s not “intern intelligence”, “grad student intelligence”, or even “alien intelligence”.

It simply isn’t intelligence at all.

It’s an entirely different phenomenon.

A complex, impressive, in many ways surprising phenomenon. Potentially useful.

But it’s not intelligence or understanding or learning.

And unless you completely abandon all anthropomorphic analogies you’ll not be able to see it for what it is and use it for what it is actually good at.