Critic Calls Mechanistic Interpretability Misleading, Favors Control Engineering Approach
Commentary on X
Highest ranked
@_onionesque much agree on the operational viewpoint. i always found the choice of the black box trope unfortunate. ml models are transparent to measurement up to numerical instability. the complexity of measurements, not their absence, is the bottleneck to interpretation
You see this category error in the dominant strains of ML theory as well. They are always aiming for ex post rationalizations. Is this how engineering theory works? (take any of them, like communication theory, microwaves or antennas; theory there leads the way).
There is this mistaken viewpoint that studying such complex models is akin to studying complex objects in natural science. If you study enough of the parts, the whole will eventually make sense. But they are synthesized objects, coercive to an operational viewpoint instead.
There is this old scientific romance about "opening the black box" that has animated interpretability research. Mech interp mixes that with (for lack of a better description) essentially gnostic exegesis (circuits as sacred objects). Not very useful for an operational viewpoint. https://twitter.com/_onionesque/status/2074694081742307343