Crosby Launches RedlineBench Legal AI Dataset on Hugging Face

Contract negotiations are like poker games. The right answer depends on knowing your opponent, as much as knowing the law/rules. How good are frontier models at closing deals?

With @micro1_ai we benchmarked frontier models on multiple contract negotiations, across several turns. Rather than individual edits, we assessed the full sequence of judgment calls a lawyer makes across a deal lifecycle.

The headline: no model is close, and there are no standout winners yet.

We’re announcing 3 things as part of the Crosby Intelligence launch today: 1/ RedlineBench with @micro1_ai – publishing the first benchmark measuring how frontier models handle multiple steps of a complex, real world contract negotiation*, hosted on @huggingface 2/ The Crosby Intelligence Research Fellowship – funding two fellows pursuing frontier research with support from @OpenAI: $25K + $12.5K Codex credits each 3/ Hosting the most interesting conversations in applied AI at our Soho office, featuring @paraga, @rahulgs, @PeterHndrsn, @NeelGuha, and more

*built externally with no client data

Read more about all three at http://intelligence.crosby.ai

Today, we're launching Crosby Intelligence to push the frontier of legal AI forward.

Our key takeaways: Models are good at restraint, yet still weak at judgment. When the right move is to leave things alone, they do well (~85%). Where it takes real legal judgment, and asking “is this clause right” or “is this worth it”, they fall to ~45%.

Models are also too agreeable. They accept a counterparty’s edits to keep a deal moving. At first that seems good, but it turns out they concede too much. Good lawyering isn’t about blindly saying yes. It’s about being clever enough to protect your client without slowing the deal.

The most interesting result is that every model is weakest on the opening move. In late negotiation turns, the models consistently score between 50-59%, but in turn 1, those scores are just 17-31%. Models are better at continuing an existing negotiation than starting a new one.

What makes this fascinating is that starting a negotiation is where human lawyers had the most consensus. Human attorneys can follow playbooks closely to start a negotiation with consensus priority issues and stances; models could not.

explore RedlineBench, apply to our fellowship, and see conversations we’re having at http://intelligence.crosby.ai

Very cool new legal benchmark on HF: https://huggingface.co/datasets/crosbylegal/RedlineBench

Law is an interesting use case for subjective AI, due to judgment and other players playing significant roles in the best outcomes. Most of this judgment is built on the collective experiences of the law firms' partners and employees.

Crosby is working to codify this intelligence and announced Crosby Intelligence today - a dedicated org to expand what their AI-powered law firm is capable of. Looking forward to seeing how the frontier models continue performing on their new RedlineBench benchmark, and the convos in their new series with leading scholars and practitioners.

*We built custom contracts and had expert human lawyers redline them specifically for this benchmark; no client data was used in the creation of this benchmark.

If these problems sound interesting, we're hiring! http://intelligence.crosby.ai

Contract negotiation is multi-agent, multi-turn, partially observable, and non-stationary. Cool benchmark launch by Crosby, important problem for models + agents to hillclimb.

@jsarihan huge congrats. great to work with you all!

@jsarihan @alexkaplan0 Let’s goo!

@jsarihan Good stuff, love the site @zachkrall @emilyzsh

Major product announcement from @crosbylegal called Crosby Intelligence, focused on providing more transparency and intelligence to legal work by adding better measurement of outcomes of negotiations and more. Read more:

A huge launch for @crosbylegal - pioneering the future of what frontier AI contracts look like 👇

there’s so much subtlety and craft that goes into defining the right benchmark and getting an agent framework to perform well throughout the process, like fine tuning taste in redlining, which is a signature of a good lawyer. i think @crosbylegal’s team nailed it, because of the way they approach the problem from the practitioner standpoint - they don’t just give you a souped up claude cowork, but they’ve built the learning framework from scratch while using it in-house with product development being done by the new breed of lawyers - legal engineers! definitely double click into how this team is changing the entire legal practice space.

introducing Crosby Intelligence!

while AI agents are new, human negotiation has been the backbone of society for thousands of years. the Crosby Intelligence branding is a simulation of the Gale-Shapley stable matching algorithm, instantiated with random preferences and solved in O(n^2) steps every time. the study of how humans reach agreement is one that has deeply meaningful and rich with history.

Crosby is excited to be pushing its next frontier.

@ClementDelangue Thank you! Best place to host it 🤝

http://intelligence.crosby.ai

@jsarihan launch of the century