Rishi Bommasani and Stanford HAI researchers find AI hiring tools violate civil rights standards by disproportionately rejecting Black and Asian applicants
Single-vendor screening risks blocking candidates across multiple employers
Hiring AI is high-stakes, pervasive, and poorly understood. That is a dangerous trifecta.
Over 90% of employers use hiring AI.
Yet independent research is bottlenecked by data access to deployed systems.
rishi@RishiBommasaniAI is changing how employers hire workers. Today we are publishing our research over the past four years into this high-stakes application of AI. We independently studied the impacts of deployed AI hiring tools based on the real outcomes for 3.3 million people.
We find that 18% Asian and 30% Black applicants are adversely impacted based on the relevant US federal standard.
If these groups were selected at the same rate as the most-selected group (generally White), then 40k additional applications would have been recommended.
rishi@RishiBommasaniWe acquire access to data from a hiring AI vendor for 3.3M applicants submitting 4M applications to 1700 positions at 150 different employers. What this means is every one of these real job applications was assessed by an AI system built by this vendor.
We acquire access to data from a hiring AI vendor for 3.3M applicants submitting 4M applications to 1700 positions at 150 different employers.
What this means is every one of these real job applications was assessed by an AI system built by this vendor.
rishi@RishiBommasaniHiring AI is high-stakes, pervasive, and poorly understood. That is a dangerous trifecta. Over 90% of employers use hiring AI. Yet independent research is bottlenecked by data access to deployed systems.
To reveal this adverse impact, we need to consider each position separately.
While our data comes from a single vendor, the vendor mediates decisions for 1700 positions at 150 firms.
Notably, just 23% of the positions yield 80% of the adverse impact for Blacks (Matthew effect)
rishi@RishiBommasaniWe find that 18% Asian and 30% Black applicants are adversely impacted based on the relevant US federal standard. If these groups were selected at the same rate as the most-selected group (generally White), then 40k additional applications would have been recommended.
Some applicants might get homogeneous outcomes by chance so we compare to the baseline of independence.
We find that: (i) observed outcomes in our data are more homogenous than the baseline yet (ii) the baseline predicts observed outcomes in prior data not focused on hiring AI
rishi@RishiBommasaniThe hiring AI we study is an algorithmic monoculture: similar systems built by the same vendor mediate decisions at many different employers. In prior work, we conjectured this homogenizes outcomes: applicants might get recommended/rejected everywhere due to the shared AI tool
The hiring AI we study is an algorithmic monoculture: similar systems built by the same vendor mediate decisions at many different employers.
In prior work, we conjectured this homogenizes outcomes: applicants might get recommended/rejected everywhere due to the shared AI tool
rishi@RishiBommasaniTo reveal this adverse impact, we need to consider each position separately. While our data comes from a single vendor, the vendor mediates decisions for 1700 positions at 150 firms. Notably, just 23% of the positions yield 80% of the adverse impact for Blacks (Matthew effect)
In particular, our work studies outcomes based on a single vendor from 2018-2022.
We need independent research into the other major vendors, especially as the industry begins to adopt more sophisticated frontier AI.
Data and system access bottlenecks independent research today
rishi@RishiBommasaniPolicy to govern the use of AI in hiring is emerging in multiple jurisdictions through multiple policy domains. We need more independent empirical research: even with our work, we know so little about what happens in practice in spite of hiring AI being pervasive.
Policy to govern the use of AI in hiring is emerging in multiple jurisdictions through multiple policy domains.
We need more independent empirical research: even with our work, we know so little about what happens in practice in spite of hiring AI being pervasive.
rishi@RishiBommasaniBecause of AI, we can test the counterfactual of what would happen if applicants applied more broadly. No applicant would have been rejected from all 1700 positions. But many would need to apply far more broadly than they did for the chances of systemic rejection to approach 0
Because of AI, we can test the counterfactual of what would happen if applicants applied more broadly.
No applicant would have been rejected from all 1700 positions.
But many would need to apply far more broadly than they did for the chances of systemic rejection to approach 0
rishi@RishiBommasaniSome applicants might get homogeneous outcomes by chance so we compare to the baseline of independence. We find that: (i) observed outcomes in our data are more homogenous than the baseline yet (ii) the baseline predicts observed outcomes in prior data not focused on hiring AI
This work took four years to complete: thanks to my brave coauthors @SarahHBana @KathleenACreel @jurafsky @percyliang for their determination to get this over the finish line.
The paper will be published and presented at FAccT 2026 in Montreal next month https://algorithmichiring.github.io/
rishi@RishiBommasaniIn particular, our work studies outcomes based on a single vendor from 2018-2022. We need independent research into the other major vendors, especially as the industry begins to adopt more sophisticated frontier AI. Data and system access bottlenecks independent research today
#449Kawin Ethayarajh@ETHAYARAJHIt's worth thinking about the second-order effects of this: if you can get rejected from all 1700 positions you applied to because one model is doing all the judging, then
1) People will increasingly demand heterogeneity in the models that run their lives. Expect to see policy moving in this direction.
2) People will control what they can, _the environment_. This means not only the CV, but also every trace of their online presence---anything that could conceivably be drawn upon by a frontier model. The goal is to mecha-nudge: not override or coerce a model to select you, but to subtly shift distributions to favor yourself. Even still, the Internet will start to look like a very different place.
Great work as usual by @RishiBommasani !
rishi@RishiBommasaniBecause of AI, we can test the counterfactual of what would happen if applicants applied more broadly. No applicant would have been rejected from all 1700 positions. But many would need to apply far more broadly than they did for the chances of systemic rejection to approach 0
@ethayarajh I expected the competition scenario. (Just like TCP-IP vs UDP), this might cause applicants to apply very broadly, flooding the recipients and forcing (shallow?) automatic consideration. Which will hurt employees (they get their worse stable pick) and partially employers (flood)
Kawin Ethayarajh@ethayarajhIt's worth thinking about the second-order effects of this: if you can get rejected from all 1700 positions you applied to because one model is doing all the judging, then 1) People will increasingly demand heterogeneity in the models that run their lives. Expect to see policy moving in this direction. 2) People will control what they can, _the environment_. This means not only the CV, but also every trace of their online presence---anything that could conceivably be drawn upon by a frontier model. The goal is to mecha-nudge: not override or coerce a model to select you, but to subtly shift distributions to favor yourself. Even still, the Internet will start to look like a very different place. Great work as usual by @RishiBommasani !
@ethayarajh Employers will get their best pick though (if they can find it, likely often not compensating on the biases of the automatic system. but sometimes?)
Leshem (Legend) Choshen 🤖🤗@LChoshen@ethayarajh I expected the competition scenario. (Just like TCP-IP vs UDP), this might cause applicants to apply very broadly, flooding the recipients and forcing (shallow?) automatic consideration. Which will hurt employees (they get their worse stable pick) and partially employers (flood)
@RishiBommasani really solid work. congrats!
rishi@RishiBommasaniAI is changing how employers hire workers. Today we are publishing our research over the past four years into this high-stakes application of AI. We independently studied the impacts of deployed AI hiring tools based on the real outcomes for 3.3 million people.
#1460Jiaxin Wen@JIAXINWEN22I really appreicate people who are dedicated to running large-scale human studies to understand AI's societal impacts.
rishi@RishiBommasaniAI is changing how employers hire workers. Today we are publishing our research over the past four years into this high-stakes application of AI. We independently studied the impacts of deployed AI hiring tools based on the real outcomes for 3.3 million people.