Posts from X
3 posts
Our new research: to start building towards political superintelligence by using AI to help us govern better, we created an AI that reads contracts and predicts ambiguities that will lead to disputes. We tested it on 10,000 Kalshi and Polymarket resolution rules, and it works quite well! Just by reading the resolution rules, our tool is able to assess which prediction-market contracts are likely to lead to resolution disputes, far better than random chance. So much of governance is writing good rules. But writing good rules is hard. For centuries, we've relied on human experts to try to write rules that anticipate as many confusions, ambiguities, and loopholes as possible. Inevitably, we make mistakes---from the famous contract law example of "the two ships Peerless" that I explore in today's piece, to unclear content moderation policies, ambiguous legislation, or the recent dustup around the US-Iran cease fire agreement wording, this is a perennial challenge. Can AI help us do it better? At Free Systems, a big part of our vision is figuring out how AI can improve how we govern, so we were eager to put this to the test. We collected a sample of 10,000 prediction-market contracts with their stated resolution rules, along with info on which ones ended up disputed. Then, we worked with our buddy Claude to develop a 10-point rubric for contract clarity, covering elements like whether the key question is well defined, whether the entities are identifiable, whether the time window is clearly specified, and so on. We had an LLM grader apply the rubric to the contracts, then built a simple machine-learning model that uses the 10-dimensional rubric score to predict subsequent disputes. The resulting scores allow us to provide overall grades to prediction-market contracts which reflect how clearly written they are and how likely they are to fall into dispute later. The contracts we grade "CCC" are 3.4x more likely to fall into dispute than the ones we grade "A." There's a lot more work to do here---we need to make sure our predictions hold in a truly out-of-sample test where we grade contracts now and see if they get disputed in the future, which we'll be working on next---and we need to expand this beyond prediction market contracts as well. But we're super excited about this direction. Tools like this will help us to identify contract ambiguities before they become disputes, allowing us to write better rules, improve governance, and eventually, get to political superintelligence. There's lots more info in our write-up, here: https://freesystems.substack.com/p/superintelligent-governance-and-the Joint work with @elliotjpaschal
Our new research: to start building towards political superintelligence by using AI to help us govern better, we created an AI that reads contracts and predicts ambiguities that will lead to disputes. We tested it on 10,000 Kalshi and Polymarket resolution rules, and it works quite well! Just by reading the resolution rules, our tool is able to assess which prediction-market contracts are likely to lead to resolution disputes, far better than random chance. So much of governance is writing good rules. But writing good rules is hard. For centuries, we've relied on human experts to try to write rules that anticipate as many confusions, ambiguities, and loopholes as possible. Inevitably, we make mistakes---from the famous contract law example of "the two ships Peerless" that I explore in today's piece, to unclear content moderation policies, ambiguous legislation, or the recent dustup around the US-Iran cease fire agreement wording, this is a perennial challenge. Can AI help us do it better? At Free Systems, a big part of our vision is figuring out how AI can improve how we govern, so we were eager to put this to the test. We collected a sample of 10,000 prediction-market contracts with their stated resolution rules, along with info on which ones ended up disputed. Then, we worked with our buddy Claude to develop a 10-point rubric for contract clarity, covering elements like whether the key question is well defined, whether the entities are identifiable, whether the time window is clearly specified, and so on. We had an LLM grader apply the rubric to the contracts, then built a simple machine-learning model that uses the 10-dimensional rubric score to predict subsequent disputes. The resulting scores allow us to provide overall grades to prediction-market contracts which reflect how clearly written they are and how likely they are to fall into dispute later. The contracts we grade "CCC" are 3.4x more likely to fall into dispute than the ones we grade "A." There's a lot more work to do here---we need to make sure our predictions hold in a truly out-of-sample test where we grade contracts now and see if they get disputed in the future, which we'll be working on next---and we need to expand this beyond prediction market contracts as well. But we're super excited about this direction. Tools like this will help us to identify contract ambiguities before they become disputes, allowing us to write better rules, improve governance, and eventually, get to political superintelligence. There's lots more info in our write-up, here: https://freesystems.substack.com/p/superintelligent-governance-and-the Joint work with @elliotjpaschal
Our new research: to start building towards political superintelligence by using AI to help us govern better, we created an AI that reads contracts and predicts ambiguities that will lead to disputes. We tested it on 10,000 Kalshi and Polymarket resolution rules, and it works quite well! Just by reading the resolution rules, our tool is able to assess which prediction-market contracts are likely to lead to resolution disputes, far better than random chance. So much of governance is writing good rules. But writing good rules is hard. For centuries, we've relied on human experts to try to write rules that anticipate as many confusions, ambiguities, and loopholes as possible. Inevitably, we make mistakes---from the famous contract law example of "the two ships Peerless" that I explore in today's piece, to unclear content moderation policies, ambiguous legislation, or the recent dustup around the US-Iran cease fire agreement wording, this is a perennial challenge. Can AI help us do it better? At Free Systems, a big part of our vision is figuring out how AI can improve how we govern, so we were eager to put this to the test. We collected a sample of 10,000 prediction-market contracts with their stated resolution rules, along with info on which ones ended up disputed. Then, we worked with our buddy Claude to develop a 10-point rubric for contract clarity, covering elements like whether the key question is well defined, whether the entities are identifiable, whether the time window is clearly specified, and so on. We had an LLM grader apply the rubric to the contracts, then built a simple machine-learning model that uses the 10-dimensional rubric score to predict subsequent disputes. The resulting scores allow us to provide overall grades to prediction-market contracts which reflect how clearly written they are and how likely they are to fall into dispute later. The contracts we grade "CCC" are 3.4x more likely to fall into dispute than the ones we grade "A." There's a lot more work to do here---we need to make sure our predictions hold in a truly out-of-sample test where we grade contracts now and see if they get disputed in the future, which we'll be working on next---and we need to expand this beyond prediction market contracts as well. But we're super excited about this direction. Tools like this will help us to identify contract ambiguities before they become disputes, allowing us to write better rules, improve governance, and eventually, get to political superintelligence. There's lots more info in our write-up, here: https://freesystems.substack.com/p/superintelligent-governance-and-the Joint work with @elliotjpaschal