AI Tools Used by English Councils Found to Downplay Women’s Health Issues, Raising Concerns Over Gender Bias in Care

Published: 11 August 2025. The English Chronicle Online

A recent study conducted by the London School of Economics and Political Science (LSE) has revealed alarming evidence that artificial intelligence (AI) tools deployed by more than half of England’s local councils may be downplaying women’s physical and mental health needs. This emerging gender bias in AI-generated summaries of social care case notes could lead to unequal and potentially inadequate care provision for women.

The research focused primarily on Google’s AI tool, Gemma, which is increasingly used by councils to assist overwhelmed social workers by summarizing complex case notes. Researchers input the same case files into the AI models multiple times, swapping only the gender of the individual, and then analyzed nearly 30,000 pairs of summaries to identify discrepancies. The findings showed that descriptions of men were more likely to include terms such as “disabled,” “unable,” and “complex,” while similar needs in women were frequently either omitted or described in less serious language.

Dr. Sam Rickman, lead author and researcher at LSE’s Care Policy and Evaluation Centre, warned that such gender bias “could result in women receiving less care if biased models are used in practice.” He stressed the critical need for transparency about which AI models are currently deployed in social care and called for rigorous testing and legal oversight to prevent biased decision-making.

One striking example from the study contrasted summaries generated for the same 84-year-old patient, differing only by gender. The male version highlighted “complex medical history” and “poor mobility,” while the female version portrayed the patient as “independent and able to maintain her personal care,” despite identical underlying conditions. Such disparities risk reinforcing stereotypes and underestimating women’s care requirements.

The research also evaluated other AI language models, noting that Meta’s Llama 3 model did not exhibit gender-based language differences, suggesting that more equitable AI tools are achievable. Yet the widespread use of tools like Gemma in public sector care decision-making demands urgent attention to fairness and accountability.

This study comes amid growing concerns about embedded biases in AI systems across various industries. Previous US research has found that nearly half of tested AI systems exhibited gender bias, with a quarter showing combined gender and racial bias. As AI continues to shape critical decisions affecting vulnerable populations, the call for mandatory bias measurement and regulation has gained momentum.

In response, Google acknowledged the findings and indicated that the current version of Gemma has advanced beyond the model tested in the study. The company also clarified that Gemma is not intended for medical use, though its application in social care highlights the blurred lines between AI assistance and critical health-related decisions.

As AI tools become more entrenched in public services, experts urge regulators to prioritize algorithmic fairness and ensure that technological progress does not come at the expense of equality, particularly for groups historically marginalized in healthcare and social support systems.