Grateful to @sebkrier for putting this paper on my radar. It helped me frame a persistent frustration with using LLMs in philosophy-related tasks – the persistent reach for a grand synthesis.
My experience talking to Claude about Tocqueville scholarship:
When models are asked to simulate the views of philosophers, they can struggle to reproduce the messy, cross-cutting structure of real philosophical disagreement.
Often they'll end up compressing the views into cleaner, more stereotyped/correlated packages, giving the impression of artificial consensus where the actual philosophers would have shown more heterogenous views.
Not too surprised: this is basically directly prompting the latent space: "here's a philosophical profile, PhD background etc - now express your stance on this PhilPapers question."
Some people will wrongly read this as an indictment on language models when in fact a better scaffold with search, retrieval over actual writings, evidence grading etc would likely do much better. https://arxiv.org/abs/2604.23575v2