Professionals distrust algorithms that fail in obvious, simplistic ways
Arvind Narayanan#267
I’ve been meaning to write a tweet thread about a new paper— Barriers to Adopting Predictive Algorithms: A Criminal Justice Field Experiment.
We built fancy ML sentence prediction software for public defenders and they mostly didn’t use it. We then pivoted to asking why not. 1/
Users praised the thread on public defenders avoiding ML sentencing tools because they viewed the topic as insightful and important.
The impact of AI on the economy and society more broadly ultimately rests on adoption: who uses it and how. This thread by Megan highlights some important barriers in the case of algorithms.
I’ve been meaning to write a tweet thread about a new paper— Barriers to Adopting Predictive Algorithms: A Criminal Justice Field Experiment.
We built fancy ML sentence prediction software for public defenders and they mostly didn’t use it. We then pivoted to asking why not. 1/
This seemed to frustrate the defenders. Occasional software glitches did also. They lost trust in our prediction software and stopped using it. 5/
Lessons for tech adoption: 1) Busy people operating as experts in their field have a high bar for adopting new technology. After witnessing a few mistakes, they lost trust, even if the prediction software brought accuracy gains. 6/
But the predictions weren't perfect. Defenders had private information that the tool didn't incorporate. We told them that the predictions should be thought of as a baseline and that they should adjust up or down based on circumstances. 4/
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6351818
With @davidsabrams @AurelieOuss Colin Sullivan Brian Collopy
2) This was a domain where not all info was digitized, or if it was digitized, it was siloed and inaccessible to use for model training purposes. Our predictions were made on limited information. This reduced accuracy and trust. 7/
We think these lessons have broad relevance re technology uptake. It may take a while for busy people to see the value or incorporate it into their workflow. And when models are trained on incomplete data, due to siloing or non-digitization, predictions will suffer. 8/
It wasn't because it didn't provide useful information. We did a quiz with head-to-head comparisons, and the sentence prediction tool was way more accurate than public defenders. (Note that defenders did not have access to private information in the quiz, just the model input.)2/
We hoped they would find it useful in evaluating plea offers and advising their client. While seasoned defenders know what a good plea deal is, more junior ones may not. Consistent with this, juniors performed worse on the quiz. 3/
@MeganTStevenson Great thread and great topic!
@alexolegimas The big challenge for orgs adopting AI is to capture the unwritten contextual information in a sustainable, AI friendly structure . Orgs that build explicit models of important decisions and information domains will benefit.
@MeganTStevenson Could you expand upon how accuracy is determined here? As in what was the prediction compared to such that it was more accurate?
@MeganTStevenson Look at http://Lassie.ai. "You're a doctor, not a machine" is their slogan. Professionals want AI for admin and billing, not an equal peer. I mean, doctors don't even want other humans to become doctors! The economics of expertise are artificially restricted supply.
@MeganTStevenson Looks like most are innumerate to a degree.
"I'm giving you a 70% accurate predictor. Them do informed updating based of specific info outside the model's knowledge" is sweet music to my ears. But super irritating to the 90%+ who "can't count" in my book....
Professionals distrust algorithms that fail in obvious, simplistic ways
Arvind Narayanan#267I’ve been meaning to write a tweet thread about a new paper— Barriers to Adopting Predictive Algorithms: A Criminal Justice Field Experiment.
We built fancy ML sentence prediction software for public defenders and they mostly didn’t use it. We then pivoted to asking why not. 1/