We're excited to announce the @crosbylegal Intelligence Fellowship: $50K in grants and $25K in Codex credits supported by @OpenAI
Legal is AI's next trillion dollar frontier. The hardest problems won't be solved by just one team.
We've shared some ideas that fascinate us and we're excited to see what you come up with:
Today, we're launching Crosby Intelligence to push the frontier of legal AI forward.
Users are excited about Crosby Intelligence's RedlineBench and AI Research Fellowship launches because they see them as valuable additions with more benchmarks and research opportunities.
No Digg Deeper questions have been answered for this story yet.
Contract negotiations are like poker games. The right answer depends on knowing your opponent, as much as knowing the law/rules. How good are frontier models at closing deals?
With @micro1_ai we benchmarked frontier models on multiple contract negotiations, across several turns. Rather than individual edits, we assessed the full sequence of judgment calls a lawyer makes across a deal lifecycle.
The headline: no model is close, and there are no standout winners yet.
Our key takeaways: Models are good at restraint, yet still weak at judgment. When the right move is to leave things alone, they do well (~85%). Where it takes real legal judgment, and asking “is this clause right” or “is this worth it”, they fall to ~45%.
Models are also too agreeable. They accept a counterparty’s edits to keep a deal moving. At first that seems good, but it turns out they concede too much. Good lawyering isn’t about blindly saying yes. It’s about being clever enough to protect your client without slowing the deal.
The most interesting result is that every model is weakest on the opening move. In late negotiation turns, the models consistently score between 50-59%, but in turn 1, those scores are just 17-31%. Models are better at continuing an existing negotiation than starting a new one.
What makes this fascinating is that starting a negotiation is where human lawyers had the most consensus. Human attorneys can follow playbooks closely to start a negotiation with consensus priority issues and stances; models could not.
We’re announcing 3 things as part of the Crosby Intelligence launch today: 1/ RedlineBench with @micro1_ai – publishing the first benchmark measuring how frontier models handle multiple steps of a complex, real world contract negotiation*, hosted on @huggingface 2/ The Crosby Intelligence Research Fellowship – funding two fellows pursuing frontier research with support from @OpenAI: $25K + $12.5K Codex credits each 3/ Hosting the most interesting conversations in applied AI at our Soho office, featuring @paraga, @rahulgs, @PeterHndrsn, @NeelGuha, and more
*built externally with no client data
Read more about all three at http://intelligence.crosby.ai
*We built custom contracts and had expert human lawyers redline them specifically for this benchmark; no client data was used in the creation of this benchmark.
If these problems sound interesting, we're hiring! http://intelligence.crosby.ai
@jsarihan @micro1_ai Very cool
@jsarihan @aliansarinik @micro1_ai Good researcg
@jsarihan @micro1_ai awesome!
@jsarihan @micro1_ai have you tested grok models?
@GallagherBilly @crosbylegal @OpenAI Heyya. I buy these credits though
@jsarihan @micro1_ai @huggingface more benchmarks! yay! 🤓
@jsarihan @micro1_ai @huggingface ⏫
@jsarihan @micro1_ai Are you giving the agents enriched data about the parties involved in the negotiation?
@jsarihan @micro1_ai @huggingface AI nails code, but bombs contract opens? RedlineBench just proved negotiation is the last human superpower.
@jsarihan When I built http://wargame.esq, I found that adding a third well-prompted agent to supervise and shepherd the negotiation kept it from devolving into stalemate, but also kept the party-agents from settling too eagerly.
@jsarihan What is the metric for a "good" opening? Remembering that, in the moment, you have no idea how the other side will respond so you can't use any post-facto information to score the opening...
