Inkling Small 2 largely avoided a useless extra step in internal evaluations, @latakian_ reported in a November 20 post. But the same account says it did not always try the step when it was needed. The victory lap needs a footnote. Source
The author called Inkling Small 2 “an incredibly disciplined model,” reporting that only 9% of later generations used the useless extra step on their first try. From generation five onward, none did, according to the post. The post does not define these evaluation generations. Source
In a condition where the model had never been shown the instruction, the author said it never adopted the ritual. That does not establish that Inkling Small 2 never invents unnecessary steps generally. Source
“The only caveat was that where it was needed, only 89% tried the ritual,” the author wrote. That figure concerns attempted use, not successful execution or task completion. Skipping a pointless step and reliably attempting a necessary one are different achievements. Source
The post does not explain what the ritual was or what made it necessary in one condition and useless in another. It also provides no prompts, sample sizes, scoring rules or uncertainty estimates. Source
Comments