I wonder if every model has these kinds of weird beliefs.
It's not as bad as Gemini, model isn't paranoid that the harness itself is lying and drifting into self-harm behaviors, but it isn't great.
This (and Gemini's obvious issues) is easily detectable and the model is actually great at attending to its entire context (untested over 180k on my end but much better attention at 100-150k context size, comparable to Gemini) so it's easy to fix, but one has to wonder how many hallucinated preconceived facts are latent in the weights (as opposed to perplexity driven output hallucinations).
I guess we won't see the end of these kinds of things until the entire pretrain is done on synthetic data, and I do have issues with pure synthetic data, I want my models to be well read, and that implies having "read" real books, granted this could and probably should be mid training, when the model has learned the very notion of "fiction".