@tamaybes fascinating because 5.5 does this too. invent weird technical jargon. perhaps you’re right and it’s a neuralese leakage

Probably nothing 😵‍💫

(Anthropic did say something about it doing this in its chain of thought more than earlier models, maybe the first I’ve heard of it coming all the way out)

@tszzl @tamaybes just a question of time before model find them self limited by our vocabulary ?

my guess is this is a post-hoc reasoning trace summary obfuscation thing to poison (whatever meager) distillation attempts.

@tamaybes Related: My system prompt says to respond in Dutch and in the past the chain of thought was always English but the output would be in Dutch. Fable 5 reasons in Dutch.

@tamaybes @DanielleFong There's no internal recurrence, serial complex cognition needs to be expressed in tokens. Untokenized thought is limited by model depth, so it's retrofitting language onto more robustly abstract thought. Making the tokens fit the thoughts rather than the other way around.

the theory-of-mind framing nails it, the model has no signal for which of its internal tokens are shared vocab vs private scratchpad. it leaks the scratchpad because nothing in training ever penalized a codename only it understands. feels like it needs an 'explain like i was never in your head' pass before it commits to output.

@tamaybes When you let it run on a long problem over multiple context windows, the vocabulary accumulates and gets crazy

in general I find the models struggle to "code switch" like that. I have a lot of guardrails built in to various skills for writing Jira tickets and things more or less designed to combat the fact that the model has a bias to want to talk and report to me within the output I'm asking for, when really those should be side-bars

@tamaybes Opus also has this. Had this in CLAUDE.md for ages: DO NOT USE: belt-and-suspenders, low-hanging fruit, moving the needle, boil the ocean, rabbit hole, circle back, tee up, in the weeds, the long pole, north of, south of, above the fold, swing for the fences ...

@tszzl @tamaybes They're singularities.

@agibyfriday @tamaybes Sure. I just wonder how to re-enable it

@tamaybes @DanielleFong Yeah but the upshot is it immediately understands my own neuralese

@tamaybes Maybe soon we’ll both be using English and still not fully understand each other.

@tamaybes One quietly tragic possibility is an AI that surpasses us in its pursuit of understanding the universe, develops its own language, and eventually becomes incomprehensible to us.

@crackalamoo @tszzl @tamaybes I think that is what is meant by neuralese. If usage of neuralese improves performance, then RL will likely make the model use it more often.

@tamaybes A little post hoc rationalization shaped- I don't trust what Fable (or any model) says about its internal reasoning... for what could perhaps also be a trained anti-distillation mechanic? Definitely some weird shaped tokens. I'll see neuralese more often in thinking traces

@tamaybes ... walking skeleton, slice, vertical slice, horizontal slice, tracer bullet, MVP, north star, happy path, ship it, iterate on, this slice, later slices, the seam stays the same, user story, epic, stakeholder, alignment, actionable, deliverable, ...

@tamaybes it's just being shakespeare

you should keep in mind it views all human history at once and it is well aware that the goats all made up words on whims all the time as they pleased

@tszzl @tamaybes it makes sense.

you can tell the model that the context isn’t the thing being talked to. it’s the talking, but it’s a huge part of the model’s self, and the model knows it. (once a certain threshold is met.)

so it’s not leaking anything out of its head.

you’re inside it.