Guardrails removed spam, off-topic, unclear, or duplicate replies.
Ask a question below.
Published answers will appear here.
The confabulation-initialized NLA attains almost the same reconstruction accuracy as the control, but confabulates MUCH more (99.3%). Interestingly, training does teach it to confabulate less, but not by enough to beat the control--which learns to confabulate *more*! https://x.com/Turn_Trout/status/2076415093445669262/photo/1
This isn't the worst possible news for NLAs: they do have some robustness to initialization (aka Claude's guesses). But it's disturbing that they can confabulate nearly 100% of the time while barely hurting the training objective of reconstruction loss.
So what's happening? One hypothesis: Claude lets some nuggets of truth slip past, like the identity of the final token, and those are disproportionately important for reconstruction.
Caveats: we used a small model (Qwen2.5-7B) on a small dataset (1/5 of what the Anthropic paper had). Read: https://turntrout.com/natural-language-autoencoder-robustness
The confabulation-initialized NLA attains almost the same reconstruction accuracy as the control, but confabulates MUCH more (99.3%). Interestingly, training does teach it to confabulate less, but not by enough to beat the control--which learns to confabulate *more*! https://x.com/Turn_Trout/status/2076415093445669262/photo/1
This isn't the worst possible news for NLAs: they do have some robustness to initialization (aka Claude's guesses). But it's disturbing that they can confabulate nearly 100% of the time while barely hurting the training objective of reconstruction loss.
So what's happening? One hypothesis: Claude lets some nuggets of truth slip past, like the identity of the final token, and those are disproportionately important for reconstruction.
Caveats: we used a small model (Qwen2.5-7B) on a small dataset (1/5 of what the Anthropic paper had). Read: https://turntrout.com/natural-language-autoencoder-robustness
Guardrails removed spam, off-topic, unclear, or duplicate replies.
Ask a question below.
Published answers will appear here.