Large language models (LLMs) show promise in healthcare, but a new study reveals a concerning issue: these models may produce biased recommendations based on patients' sociodemographic labels rather than clinical needs.
Researchers from the Icahn School of Medicine at Mount Sinai and the Mount Sinai Health System analyzed over 1.7 million outputs from nine LLMs across 1,000 emergency cases, each presented with varying demographic profiles.
The study found that cases labeled as Black, unhoused, or LGBTQIA+ were more frequently directed by the LLMs toward urgent care, invasive procedures, or mental health evaluations—sometimes exceeding clinical recommendations.
In contrast, high-income labels led to more advanced diagnostic imaging, while low- and middle-income patients often received fewer tests or less comprehensive care.
These findings suggest that AI-driven healthcare systems may inadvertently perpetuate, rather than reduce, healthcare inequities.
The researchers stress the need for ongoing audits and corrective measures to address the biases identified in the study.
As AI systems become more integrated into healthcare, ensuring that these tools do not exacerbate existing inequities is essential. The study calls for increased attention to the ethical development of AI in healthcare and the implementation of safeguards to protect vulnerable populations from biased or inequitable care.
【MORE】