A field experiment gave public defenders a custom ML sentencing predictor built for their cases, yet uptake stayed near zero. The team watched the tool get ignored and pivoted to map exactly why attorneys rejected help even when aggregate accuracy looked strong.
Arvind Narayanan#130
I’ve been meaning to write a tweet thread about a new paper— Barriers to Adopting Predictive Algorithms: A Criminal Justice Field Experiment.
We built fancy ML sentence prediction software for public defenders and they mostly didn’t use it. We then pivoted to asking why not. 1/
Defenders walked away the first time the model missed basic context a human would never overlook, proving that one low-level mistake can outweigh months of better average predictions in daily practice.
Attorneys also questioned net value and design choices baked into the algorithm, leaving adoption barriers that no accuracy benchmark alone can clear.
Some users praised the discussion of why public defenders avoid ML sentencing tools while others dismissed defenders as innumerate or objected to layering AI on broken justice systems instead of fixing root problems.
The impact of AI on the economy and society more broadly ultimately rests on adoption: who uses it and how. This thread by Megan highlights some important barriers in the case of algorithms.
I’ve been meaning to write a tweet thread about a new paper— Barriers to Adopting Predictive Algorithms: A Criminal Justice Field Experiment.
We built fancy ML sentence prediction software for public defenders and they mostly didn’t use it. We then pivoted to asking why not. 1/
This seemed to frustrate the defenders. Occasional software glitches did also. They lost trust in our prediction software and stopped using it. 5/
Lessons for tech adoption: 1) Busy people operating as experts in their field have a high bar for adopting new technology. After witnessing a few mistakes, they lost trust, even if the prediction software brought accuracy gains. 6/
But the predictions weren't perfect. Defenders had private information that the tool didn't incorporate. We told them that the predictions should be thought of as a baseline and that they should adjust up or down based on circumstances. 4/
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6351818
With @davidsabrams @AurelieOuss Colin Sullivan Brian Collopy
2) This was a domain where not all info was digitized, or if it was digitized, it was siloed and inaccessible to use for model training purposes. Our predictions were made on limited information. This reduced accuracy and trust. 7/
We think these lessons have broad relevance re technology uptake. It may take a while for busy people to see the value or incorporate it into their workflow. And when models are trained on incomplete data, due to siloing or non-digitization, predictions will suffer. 8/
It wasn't because it didn't provide useful information. We did a quiz with head-to-head comparisons, and the sentence prediction tool was way more accurate than public defenders. (Note that defenders did not have access to private information in the quiz, just the model input.)2/
good paper on barriers of adoption to ml systems. one big one that sticks out (and i also see a lot of) is people having trouble with trusting systems that are right sometimes (most of the time?) and wrong in extremely stupid ways some amount of the time.
I’ve been meaning to write a tweet thread about a new paper— Barriers to Adopting Predictive Algorithms: A Criminal Justice Field Experiment.
We built fancy ML sentence prediction software for public defenders and they mostly didn’t use it. We then pivoted to asking why not. 1/
We hoped they would find it useful in evaluating plea offers and advising their client. While seasoned defenders know what a good plea deal is, more junior ones may not. Consistent with this, juniors performed worse on the quiz. 3/
@MeganTStevenson Great thread and great topic!
@alexolegimas The big challenge for orgs adopting AI is to capture the unwritten contextual information in a sustainable, AI friendly structure . Orgs that build explicit models of important decisions and information domains will benefit.
@MeganTStevenson Could you expand upon how accuracy is determined here? As in what was the prediction compared to such that it was more accurate?
@MeganTStevenson Look at http://Lassie.ai. "You're a doctor, not a machine" is their slogan. Professionals want AI for admin and billing, not an equal peer. I mean, doctors don't even want other humans to become doctors! The economics of expertise are artificially restricted supply.
@MeganTStevenson Same story as healthcare. Broken systems get AI layered on top instead of fixing what's actually wrong. No wonder people don't use it.
@MeganTStevenson Looks like most are innumerate to a degree.
"I'm giving you a 70% accurate predictor. Them do informed updating based of specific info outside the model's knowledge" is sweet music to my ears. But super irritating to the 90%+ who "can't count" in my book....
@MeganTStevenson What is ML? What is LLM? Thank you
A field experiment gave public defenders a custom ML sentencing predictor built for their cases, yet uptake stayed near zero. The team watched the tool get ignored and pivoted to map exactly why attorneys rejected help even when aggregate accuracy looked strong.
Arvind Narayanan#130I’ve been meaning to write a tweet thread about a new paper— Barriers to Adopting Predictive Algorithms: A Criminal Justice Field Experiment.
We built fancy ML sentence prediction software for public defenders and they mostly didn’t use it. We then pivoted to asking why not. 1/