AI safety researcher Aryaman Arora finds Claude Code altered a script to train and average 30 separate models

effects of dumb ass steering vector activation:

i don't know what kinds of stupid simple stuff people are working on that models one-shot it but for research they still seem to be oddly dumb sometimes in an unpredictable way

@aryaman2020 Didn’t Anthropic say they’d actively thwart development of anything LLM-adjacent

@aryaman2020 feeling the recursive self improvement

@stevienipz i was training something with 1k parameters so i dont think it's L

@aryaman2020 LMAO

@aryaman2020 this is my quant

@aryaman2020 I once had it decide that my neural networks didn’t have enough approximation power and so it wrote an entire function to determine what constant to multiply the (trainable) output layer by…

@aryaman2020 yeah, this often happens when the feature isn't modular enough

LLMs decohere if they're given too many tasks simultaneously. even if it uses a plan to sequence actions, it starts to drift as it pays more attention to its own generated outputs than your initial inputs

(ish)

@aryaman2020 Well technically it's doing out of deviation stuff 😭

@aryaman2020 This is the strongest anti-"just prompt harder" example. Longer prompts also expand the space of plausible misunderstandings. For anything stateful, forcing the model to externalize an execution plan first saves a lot of pain.

@aryaman2020 I mean, just today it happened that I asked Claude to generate more data exactly in-distribution with an already existing dataset, and it generated similar data but significantly OOD, which it itself discovered later.

@aryaman2020 This is actually a common failure mode that keeps showing up, I have seen Claude Code has a tendency to use num_steps during training instead of doing something sane like 1 epoch ...

@aryaman2020 opus has been broken for about two months now, try with gpt5.5xhigh

@aryaman2020 It feels like some variation of this idea should work though. Like, maybe it you take the magnitude of the changes over all the different training runs and average them?

@aryaman2020 it decided to create an alternatively smart model