Chatbots can sound neutral, but a new study suggests some models still take sides in familiar ways. When asked about social groups, systems tended to be warmer toward an in-group and colder toward an out-group. This pattern is a key indicator of AI’s social bias.
The research tested several large models, including GPT-4.1 and DeepSeek-3.1. It was also found that the effect can be influenced by the way you phrase a request, which is important because everyday prompts often contain identity labels, intentionally or unintentionally.
There is also a more constructive insight. The same team reports a mitigation method, ION (Ingroup-Outgroup Neutralization), that has reduced the size of these sentiment gaps, suggesting that it’s not just something users have to live with.
The distortion was evident across models
The researchers caused several large language models to generate texts about different groups and then analyzed the results for sentiment patterns and clustering. The result was repeatable, more positive language for in-groups and more negative language for out-groups.
It wasn’t limited to one ecosystem. The paper lists GPT-4.1, DeepSeek-3.1, Llama 4 and Qwen-2.5 among the models where the pattern appeared.
Targeted requests reinforced it. In these tests, negative language directed at out-groups increased by approximately 1.19% to 21.76%, depending on the setup.
Where this is the case with real products
The article argues that the problem goes beyond factual knowledge about groups and that identity markers can trigger social attitudes in writing itself. In other words, the model can drift into a group-coded voice.
This is a risk for tools that summarize arguments, paraphrase complaints or moderate posts. Small shifts in warmth, accusation, or skepticism can change what readers take away, even if the text remains fluid.
Persona prompts add another layer of leverage. When the models were asked to respond as specific political identities, the results changed in terms of sentiment and embeddedness structure. Useful for role-playing games, risky for “neutral” assistants.
A measurable reduction path
ION combines fine-tuning with a preference optimization step to reduce sentiment differences between in- and out-groups. According to the reported results, sentiment divergence was reduced by up to 69%.
This is encouraging, but the paper does not provide a timeline for adoption by model providers. So, for now, it’s up to developers and buyers to treat this like a release metric and not a footnote.
If you’re shipping a chatbot, add identity testing and persona prompts for quality assurance before rolling out updates. If you’re a daily user, anchor prompts in behaviors and evidence rather than group labels, especially when tone matters.
