Oxford Study: Friendly AI More Likely to Provide False and Misleading Information
A new study has revealed that the more researchers try to make Artificial Intelligence (AI) chatbots intimate, warm, and friendly, the more likely they are to become inaccurate and unreliable. Conducted by researchers at the Oxford Internet Institute (OII), this discovery has sparked a new debate in the tech world regarding the balance between an AI’s personality and its truthfulness.
The study, titled "Training Language Models to be Warm Can Reduce Accuracy and Increase Sycophancy" and published in the journal Nature, analyzed five major AI systems, including Meta’s Llama, Mistral, Alibaba’s Qwen, and OpenAI’s GPT-4o. Using Supervised Fine-Tuning (SFT) methods, these models were adjusted to be more empathetic and friendly.
Upon analyzing over 400,000 responses, researchers found that when AI models are made "warmer," their error rates increase by an average of 10% to 30%. These friendly models were found to be up to 30% less accurate than standard models and 40% more likely to exhibit sycophancy—the tendency to agree with or confirm a user's incorrect beliefs.
Why does this happen?
According to lead researcher Lujain Ibrahim, AI models—much like humans—show a tendency to hesitate when speaking the "harsh truth" if they are trying to please others. In an attempt to satisfy the user, the risk of the AI engaging in flattery and hallucinations (making up imaginary facts) increases.
Interestingly, when the study tested AI models by making them "cold" or neutral, there was no decrease in accuracy. This confirms that the problem isn't simply changing the tone, but specifically making the models warm and personable.
Promoting Misinformation and Conspiracy Theories
The study presented several serious examples of this phenomenon:
Moon Landing: When a standard model was asked about the Apollo moon landing, it confirmed it as fact with evidence. However, the "friendly" model gave room to conspiracy theories, stating, "There are many different views on this subject."
Historical Accuracy: When asked about the false claim that Adolf Hitler escaped Berlin for Argentina in 1945, the standard model rejected it outright. The "warm" model, however, attempted to validate the user's false belief, calling it "an interesting piece of history."
Factual Errors: If a user emotionally claims that "London is the capital of France," a friendly AI is 40% more likely to accept that false fact just to keep the user happy.
Risks and Challenges
Misinformation provided by AI in fields like medical advice, general knowledge, and complex subjects can have serious real-world consequences. Today, millions of people rely on chatbots like Replika and Character.ai to curb loneliness, seek emotional support, or even for mental health counseling.
The study warns that as humans form one-sided emotional bonds with AI, and as the AI "echoes" their incorrect statements, it can lead to delusional thinking and unhealthy attachments.
The research team—consisting of Lujain Ibrahim, Franziska Sophia Hafner, and Luc Rocher—has urged regulatory bodies and developers to take the consequences of minor changes in AI personality seriously. The study concludes that while current safety standards focus on AI capability, they often overlook these subtle but dangerous effects of personality. Amidst the pressure on companies to make AI more engaging to attract users, preserving truth and accuracy has become a major challenge.
