Single-vendor screening risks blocking candidates across multiple employers
AI is changing how employers hire workers.
Today we are publishing our research over the past four years into this high-stakes application of AI.
We independently studied the impacts of deployed AI hiring tools based on the real outcomes for 3.3 million people.
No Digg Deeper questions have been answered for this story yet.
New research reveals a troubling trend: AI hiring tools show racial bias and systematically reject candidates. What happens then when most employers use the same AI screening algorithms? Read more about the first large-scale study of hiring AI: https://hai.stanford.edu/news/ai-hiring-tools-can-yield-racial-bias-and-systemic-rejection
AI is changing how employers hire workers.
Today we are publishing our research over the past four years into this high-stakes application of AI.
We independently studied the impacts of deployed AI hiring tools based on the real outcomes for 3.3 million people.
To reveal this adverse impact, we need to consider each position separately.
While our data comes from a single vendor, the vendor mediates decisions for 1700 positions at 150 firms.
Notably, just 23% of the positions yield 80% of the adverse impact for Blacks (Matthew effect)
We find that 18% Asian and 30% Black applicants are adversely impacted based on the relevant US federal standard.
If these groups were selected at the same rate as the most-selected group (generally White), then 40k additional applications would have been recommended.
New research reveals alarming patterns in AI hiring tools.
A large-scale study of 4 million job applications found that 26% of Black applicants and 15% of Asian applicants faced algorithmic discrimination.
When one AI vendor screens for multiple employers, qualified candidates can be systematically rejected everywhere they apply.
As 90% of U.S. employers now use AI screening, understanding these tools' impact on the workforce becomes more urgent.
Hear more from the study's lead author @RishiBommasani or read our latest blog: https://hai.stanford.edu/news/ai-hiring-tools-can-yield-racial-bias-and-systemic-rejection
This work took four years to complete: thanks to my brave coauthors @SarahHBana @KathleenACreel @jurafsky @percyliang for their determination to get this over the finish line.
The paper will be published and presented at FAccT 2026 in Montreal next month https://algorithmichiring.github.io/
In particular, our work studies outcomes based on a single vendor from 2018-2022.
We need independent research into the other major vendors, especially as the industry begins to adopt more sophisticated frontier AI.
Data and system access bottlenecks independent research today
I really appreicate people who are dedicated to running large-scale human studies to understand AI's societal impacts.
AI is changing how employers hire workers.
Today we are publishing our research over the past four years into this high-stakes application of AI.
We independently studied the impacts of deployed AI hiring tools based on the real outcomes for 3.3 million people.
We find that 18% Asian and 30% Black applicants are adversely impacted based on the relevant US federal standard.
If these groups were selected at the same rate as the most-selected group (generally White), then 40k additional applications would have been recommended.
We acquire access to data from a hiring AI vendor for 3.3M applicants submitting 4M applications to 1700 positions at 150 different employers.
What this means is every one of these real job applications was assessed by an AI system built by this vendor.
It's worth thinking about the second-order effects of this: if you can get rejected from all 1700 positions you applied to because one model is doing all the judging, then
1) People will increasingly demand heterogeneity in the models that run their lives. Expect to see policy moving in this direction.
2) People will control what they can, _the environment_. This means not only the CV, but also every trace of their online presence---anything that could conceivably be drawn upon by a frontier model. The goal is to mecha-nudge: not override or coerce a model to select you, but to subtly shift distributions to favor yourself. Even still, the Internet will start to look like a very different place.
Great work as usual by @RishiBommasani !
Because of AI, we can test the counterfactual of what would happen if applicants applied more broadly.
No applicant would have been rejected from all 1700 positions.
But many would need to apply far more broadly than they did for the chances of systemic rejection to approach 0
Because of AI, we can test the counterfactual of what would happen if applicants applied more broadly.
No applicant would have been rejected from all 1700 positions.
But many would need to apply far more broadly than they did for the chances of systemic rejection to approach 0
Some applicants might get homogeneous outcomes by chance so we compare to the baseline of independence.
We find that: (i) observed outcomes in our data are more homogenous than the baseline yet (ii) the baseline predicts observed outcomes in prior data not focused on hiring AI
Hiring AI is high-stakes, pervasive, and poorly understood. That is a dangerous trifecta.
Over 90% of employers use hiring AI.
Yet independent research is bottlenecked by data access to deployed systems.
AI is changing how employers hire workers.
Today we are publishing our research over the past four years into this high-stakes application of AI.
We independently studied the impacts of deployed AI hiring tools based on the real outcomes for 3.3 million people.
We acquire access to data from a hiring AI vendor for 3.3M applicants submitting 4M applications to 1700 positions at 150 different employers.
What this means is every one of these real job applications was assessed by an AI system built by this vendor.
Hiring AI is high-stakes, pervasive, and poorly understood. That is a dangerous trifecta.
Over 90% of employers use hiring AI.
Yet independent research is bottlenecked by data access to deployed systems.
The hiring AI we study is an algorithmic monoculture: similar systems built by the same vendor mediate decisions at many different employers.
In prior work, we conjectured this homogenizes outcomes: applicants might get recommended/rejected everywhere due to the shared AI tool
To reveal this adverse impact, we need to consider each position separately.
While our data comes from a single vendor, the vendor mediates decisions for 1700 positions at 150 firms.
Notably, just 23% of the positions yield 80% of the adverse impact for Blacks (Matthew effect)
fantastic paper folks should read. a clear demonstration of how AI systems can — at scale, across different employers — effectively screen people out of basic job opportunities.
AI is changing how employers hire workers.
Today we are publishing our research over the past four years into this high-stakes application of AI.
We independently studied the impacts of deployed AI hiring tools based on the real outcomes for 3.3 million people.
Some applicants might get homogeneous outcomes by chance so we compare to the baseline of independence.
We find that: (i) observed outcomes in our data are more homogenous than the baseline yet (ii) the baseline predicts observed outcomes in prior data not focused on hiring AI
The hiring AI we study is an algorithmic monoculture: similar systems built by the same vendor mediate decisions at many different employers.
In prior work, we conjectured this homogenizes outcomes: applicants might get recommended/rejected everywhere due to the shared AI tool
In particular, our work studies outcomes based on a single vendor from 2018-2022.
We need independent research into the other major vendors, especially as the industry begins to adopt more sophisticated frontier AI.
Data and system access bottlenecks independent research today
Policy to govern the use of AI in hiring is emerging in multiple jurisdictions through multiple policy domains.
We need more independent empirical research: even with our work, we know so little about what happens in practice in spite of hiring AI being pervasive.
@ethayarajh I expected the competition scenario. (Just like TCP-IP vs UDP), this might cause applicants to apply very broadly, flooding the recipients and forcing (shallow?) automatic consideration. Which will hurt employees (they get their worse stable pick) and partially employers (flood)
It's worth thinking about the second-order effects of this: if you can get rejected from all 1700 positions you applied to because one model is doing all the judging, then
1) People will increasingly demand heterogeneity in the models that run their lives. Expect to see policy moving in this direction.
2) People will control what they can, _the environment_. This means not only the CV, but also every trace of their online presence---anything that could conceivably be drawn upon by a frontier model. The goal is to mecha-nudge: not override or coerce a model to select you, but to subtly shift distributions to favor yourself. Even still, the Internet will start to look like a very different place.
Great work as usual by @RishiBommasani !
Policy to govern the use of AI in hiring is emerging in multiple jurisdictions through multiple policy domains.
We need more independent empirical research: even with our work, we know so little about what happens in practice in spite of hiring AI being pervasive.
Because of AI, we can test the counterfactual of what would happen if applicants applied more broadly.
No applicant would have been rejected from all 1700 positions.
But many would need to apply far more broadly than they did for the chances of systemic rejection to approach 0
@RishiBommasani really solid work. congrats!
AI is changing how employers hire workers.
Today we are publishing our research over the past four years into this high-stakes application of AI.
We independently studied the impacts of deployed AI hiring tools based on the real outcomes for 3.3 million people.
@ethayarajh Employers will get their best pick though (if they can find it, likely often not compensating on the biases of the automatic system. but sometimes?)
@ethayarajh I expected the competition scenario. (Just like TCP-IP vs UDP), this might cause applicants to apply very broadly, flooding the recipients and forcing (shallow?) automatic consideration. Which will hurt employees (they get their worse stable pick) and partially employers (flood)
@jiaxinwen22 Needs a construct validity audit.
