The central thesis of the paper is simple:

Cooperation is not an additional capability to be scaled or a task to solve, but an equilibrium property that emerges from multiple intelligences navigating their irreducible interdependence.

The solipsistic paradigm fails to account for the structure that makes cooperation possible or fragile.

Unilateral optimization for task success can therefore become self-undermining, pushing the coupled system toward degraded equilibria. 2/n

📄 Paper: https://arxiv.org/abs/2606.03237

Work done in collaboration with my wonderful coauthors @natashajaques, @locross, Sasha Vezhnevets, and @jzl86.

Very excited to present this at #ICML 2026. If you are visiting, come say hi at our poster session. We would love to discuss!

Deployment itself induces non-stationarity through structured responses, resulting in a train–test–deploy gap.

The paper identifies three channels of adaptation: 👤 Behavioral: humans alter behavior

🏛️ Institutional: organizations revise rules and procedures

🤖 Algorithmic: other AI systems retrain, fine-tune, and co-evolve

5/n

The paper concludes by tackling several counterarguments such as:

- multi-actor designs may have worse failure modes - competitive pressure may produce cooperation naturally - the empirical track record may not justify alarm - scale may solve interaction dynamics - RLHF may already train cooperative behavior

These are serious objections. Our response is that each misses how deployment changes the game.

12/n

We outline three key directions for progress on this non-solipsistic research path:

🔁 Dynamic evaluation: evaluate systems under test distributions that depend on adaptive counterparties.

🏛️ Institutions as design primitives: design mechanisms, incentives, and governance structures that shape cooperative behavior.

🧑‍🤝‍🧑 Preserving human agency: treat human learning, judgment, option space, and meaningful control as core design constraints.

11/n

When outcomes depend on the joint behavior of multiple adaptive agents, good performance is no longer an optimization output of any single policy in isolation.

It becomes an equilibrium property of the coupled system.

This is why cooperation becomes central.

By cooperation, we mean the process by which society coordinates to select beneficial equilibria and avoid harmful ones.

This raises the questions the rest of the paper takes up. 6/n

Why is cooperation difficult?

🌐 Externalities & equilibrium shifts: optimizing entities create spillovers no single objective accounts for.

⏱️ Temporal asymmetry: AI systems may adapt faster than humans, organizations, laws, and norms.

⚖️ Legitimacy: coordination mechanisms work only while affected actors accept them as legitimate and retain meaningful control.

7/n

Why are current approaches unlikely to produce cooperative AI?

Self-undermining property: the more aggressively a unilateral optimizer exploits historical patterns, the faster it induces adaptations that render those patterns obsolete.

Exploit historical regularity → others best-respond → regularity no longer holds

For more capable systems, deeper exploitation can create stronger incentives to adapt and sharper regime shifts.

8/n

Why isn’t cooperation just another capability for powerful predictors?

Because prediction is not participation.

The paper identifies two independent grounds, either of which may suffice to block unilateral prediction as a solution path.

🧠 Epistemic horizons: deployment creates novelty, predictions become part of the game, and the space of interacting agents rapidly becomes intractable.

⚖️ Legitimacy and participation: even where prediction improves, open societies require legal order, value pluralism, attention to preference endogeneity, and safeguards against Goodhart dynamics.

9/n

The paper calls for a paradigm shift: treating interdependence among actors as a first-order design principle.

Instead of asking only what do humans want?, we should ask:

What arrangements are sustainable when all parties — human and artificial — adapt in response to each other, and what processes select beneficial arrangements from the many possibilities?

10/n

The dominant pathway to advanced AI has been remarkably successful at advancing capabilities.

But the development pipeline often looks like:

pretraining on static corpora → post-training against frozen reward models → hill-climbing (“benchmaxxing”) on fixed evaluation suites → deployment

Each stage treats the external world as a stationary distribution, and measures progress on targets that do not respond.

A world with many humans and many AIs does. 3/n

This is the solipsistic assumption behind the pipeline:

🌍 Environment: exogenous to the agent’s policy

📈 Distribution: stationary from training to deployment

👥 Other agents: absorbed into the state space rather than treated as strategic actors

AI is built as if deployment were insertion into a fixed world. The paper argues that deployment is better understood as intervention into a coupled system. 4/n

@rstriv @natashajaques @locross @jzl86 Looking forward to this!