Heuristics And Post-Hoc Reasoning Drive All Fluid Intelligence
Source Posts on X
Cluster sources
However, infinite computation limit theories (reflective oracles, logical induction, AIXI) are important! Rationalization is far from the only safety problem, and the fastest route to a tractable bounded rationality theory may be via weakening an infinite computation theory.
This combination of reasoning and guesses is how humans and LLMs work now, and it is how the ASIs of 1000 years from now will also work. Guessing is very powerful! The only way to not make ASIs that rely on guessing is to not make ASIs (a good plan, but out of thread scope).
First, it does not work to say "don't do any rationalization". All fluid intelligence (human or machine) is built on heuristics: our thoughts are a combination of explicit reasoning and wild guesses, with the guesses filtered with post-hoc reasoning where promising.
There are least two routes one could imagine to solve this safety challenge: 1. Trace the causal story of the heuristic back through train so that in addition to expanding into rationalization, we can "expand into training" in a fully causal way (learning dynamics, SLT, etc.).