AI law firm Crosby launches Crosby Intelligence to develop subjective legal reasoning models and releases its RedlineBench benchmark

Law is an interesting use case for subjective AI, due to judgment and other players playing significant roles in the best outcomes. Most of this judgment is built on the collective experiences of the law firms' partners and employees.

Crosby is working to codify this intelligence and announced Crosby Intelligence today - a dedicated org to expand what their AI-powered law firm is capable of. Looking forward to seeing how the frontier models continue performing on their new RedlineBench benchmark, and the convos in their new series with leading scholars and practitioners.

We’re announcing 3 things as part of the Crosby Intelligence launch today: 1/ RedlineBench with @micro1_ai – publishing the first benchmark measuring how frontier models handle multiple steps of a complex, real world contract negotiation*, hosted on @huggingface 2/ The Crosby Intelligence Research Fellowship – funding two fellows pursuing frontier research with support from @OpenAI: $25K + $12.5K Codex credits each 3/ Hosting the most interesting conversations in applied AI at our Soho office, featuring @paraga, @rahulgs, @PeterHndrsn, @NeelGuha, and more

*built externally with no client data

Read more about all three at http://intelligence.crosby.ai

Contract negotiations are like poker games. The right answer depends on knowing your opponent, as much as knowing the law/rules. How good are frontier models at closing deals?

With @micro1_ai we benchmarked frontier models on multiple contract negotiations, across several turns. Rather than individual edits, we assessed the full sequence of judgment calls a lawyer makes across a deal lifecycle.

The headline: no model is close, and there are no standout winners yet.

Our key takeaways: Models are good at restraint, yet still weak at judgment. When the right move is to leave things alone, they do well (~85%). Where it takes real legal judgment, and asking “is this clause right” or “is this worth it”, they fall to ~45%.

Models are also too agreeable. They accept a counterparty’s edits to keep a deal moving. At first that seems good, but it turns out they concede too much. Good lawyering isn’t about blindly saying yes. It’s about being clever enough to protect your client without slowing the deal.

The most interesting result is that every model is weakest on the opening move. In late negotiation turns, the models consistently score between 50-59%, but in turn 1, those scores are just 17-31%. Models are better at continuing an existing negotiation than starting a new one.

What makes this fascinating is that starting a negotiation is where human lawyers had the most consensus. Human attorneys can follow playbooks closely to start a negotiation with consensus priority issues and stances; models could not.

*We built custom contracts and had expert human lawyers redline them specifically for this benchmark; no client data was used in the creation of this benchmark.

If these problems sound interesting, we're hiring! http://intelligence.crosby.ai

@jsarihan huge congrats. great to work with you all!

@jsarihan Good stuff, love the site @zachkrall @emilyzsh

@jsarihan launch of the century

@jsarihan bars

@jsarihan incredible release & such a pleasure working with you all

Crosby Intelligence 🔥

@jsarihan Huge congrats on launch! 🐐

@jsarihan Let’s go!!!

@jsarihan @ryanjdaniels Congrats John! Such a cool problem + benchmark to be tackling

@jsarihan Sick!

@jsarihan sickkk

@jsarihan incredible

@jsarihan Woooooo

@jsarihan major, congrats!

@jsarihan Congrats!!