AI Models Struggle Most with Opening Legal Negotiations

The most interesting result is that every model is weakest on the opening move. In late negotiation turns, the models consistently score between 50-59%, but in turn 1, those scores are just 17-31%. Models are better at continuing an existing negotiation than starting a new one.

What makes this fascinating is that starting a negotiation is where human lawyers had the most consensus. Human attorneys can follow playbooks closely to start a negotiation with consensus priority issues and stances; models could not.

Our key takeaways: Models are good at restraint, yet still weak at judgment. When the right move is to leave things alone, they do well (~85%). Where it takes real legal judgment, and asking “is this clause right” or “is this worth it”, they fall to ~45%.

Models are also too agreeable. They accept a counterparty’s edits to keep a deal moving. At first that seems good, but it turns out they concede too much. Good lawyering isn’t about blindly saying yes. It’s about being clever enough to protect your client without slowing the deal.

A hard non verifiable RL problem - what does “good” look like in opening legal negotiation 👇

*We built custom contracts and had expert human lawyers redline them specifically for this benchmark; no client data was used in the creation of this benchmark.

If these problems sound interesting, we're hiring! http://intelligence.crosby.ai

A misconception we hear a lot is, "contract review is commoditized legal work."

In fact, it requires very clever lawyering. Tact, strategy and creativity. It's complex negotiation.

We spent the last few months building an authoritative benchmark with @micro1_ai

negotiationBench just dropped

@ryanjdaniels @micro1_ai very cool

@jsarihan @micro1_ai Very cool

@jsarihan @aliansarinik @micro1_ai Good researcg

@jsarihan @micro1_ai awesome!

@jsarihan @micro1_ai have you tested grok models?

@jsarihan @micro1_ai Are you giving the agents enriched data about the parties involved in the negotiation?

@ryanjdaniels @micro1_ai The split is what matters. Most of what a growing company signs is commodity, NDAs and MSAs, the same paper over and over, and agents handle that end to end now. The clever lawyering you describe is the 20% of deals where it actually counts. That's where humans belong.