I had to explain this many times last week...
A Critique of Aumann’s Agreement Theorem
Aumann’s Agreement Theorem does not apply to actual minds.
The theorem says that two Bayesian agents with a common prior cannot knowingly maintain different posterior probabilities. If their posteriors are common knowledge, they must agree. But the common-prior assumption is doing almost all the work.
1. Priors are not shared objects
For real minds, priors are not explicit probability tables. They are compressed products of accumulated experience: education, culture, training, social feedback, past successes, failures, and emotional salience.
In neural network terms, a mind’s “prior” is encoded in its weights. Those weights are a lossy compression of vast quantities of past data. This data is not recoverable or shareable.
So two agents cannot simply exchange the evidence that produced their priors. They can exchange arguments and conclusions, but not the full training history that shaped their world-models.
2. Evidence is digested, not stored
The Bayesian picture treats evidence as discrete observations that can be conditioned on hypotheses. Real cognition is different. Much of the relevant evidence has been absorbed into perception, intuition, pattern recognition, and judgment.
A physicist, a lawyer, a soldier, and an investor may see the same facts but extract different features. Their disagreement may lie not in any single missing datum, but in the internal models that decide what matters.
This is also true of AI systems. A trained neural net does not store its training data in explicit form. It compresses statistical regularities into weights. Its later judgments are products of that compression, not transparent Bayesian updates over a preserved evidence ledger.
3. Knowing another posterior is not enough
Suppose Alice assigns 80% probability to X and Bob assigns 20%. Aumann says this cannot persist if they share a prior and know each other’s posteriors.
But Alice and Bob do not merely have different posteriors. They may have different learned priors, different likelihood models, different feature maps, and different estimates of each other’s reliability.
Alice may rationally treat Bob’s disagreement as evidence, but not decisive evidence. Bob may do the same. Their disagreement need not disappear, because each is evaluating the other through a different internal model.
4. Disagreement can reflect model heterogeneity
Persistent disagreement is not necessarily irrational. It often reflects heterogeneous inference systems.
Actual minds differ in training data, architecture, incentives, expertise, calibration, and conceptual vocabulary. They do not merely update differently; they often represent the problem differently.
Thus “agreeing to disagree” is not a paradox. It is what we should expect from bounded, path-dependent, lossy learning systems.
5. Real priors are learned
The common-prior assumption is especially artificial because real priors are themselves the residue of previous evidence. Yesterday’s posteriors become today’s priors.
Over a lifetime, a mind’s prior becomes a compressed autobiography of experience. Two agents with different priors may simply have processed different histories.
Asking them to adopt a common prior is like asking two neural networks trained on different corpora to erase their training and begin from the same initialization.
6. AI makes the problem clearer
Modern AI systems illustrate the point sharply. Their judgments arise from weights produced by massive training runs. The underlying data, optimization path, architecture, fine-tuning, and reinforcement signals are not fully recoverable from the final model.
Two AIs may disagree even when given the same prompt. They can produce rationales, but those rationales are not the full causal history of their weights. They are verbal reconstructions, not the evidence itself.
AI therefore does not rescue the Aumann ideal. It makes vivid why that ideal fails for real learning systems.
7. Empirically, minds agree to disagree
In practice, intelligent and informed people often know each other’s views and still disagree. Economists, physicists, investors, historians, political analysts, and AI researchers all do this.
The disagreement is often common knowledge. Yet convergence does not follow, because the agents do not share priors in Aumann’s strong sense.
Bottom line
Aumann’s theorem proves that ideal Bayesians with common priors cannot agree to disagree. But actual minds are not such agents.
Human and artificial minds are neural networks whose weights encode lossy compressions of past evidence. That evidence cannot, in general, be fully recovered or exchanged. Therefore persistent disagreement is not an anomaly. It is the normal outcome of heterogeneous minds trained on different histories.
