AI as an “unreliable newsmaker”: study reveals massive errors in AI assistants’ news responses
27 October 14:56
Artificial intelligence, which is increasingly used as a news source, still cannot ensure accuracy and reliability. This is the conclusion reached by researchers of the European Broadcasting Union (EBU) after a large-scale testing of the most popular AI assistants, "Komersant Ukrainian" reports.
The results published on the EBU website show that 45% of AI news responses contained serious errors, and another 81% contained minor inaccuracies.
What is known
The EBU involved 22 public broadcasters from 18 countries. They analyzed more than 3,000 responses generated by ChatGPT (OpenAI), Copilot (Microsoft), Gemini (Google), and Perplexity systems in different languages.
They evaluated four criteria:
- accuracy of facts
- reliability of sources,
- separation of facts from opinions,
- context and relevance of the data.
What are the most common mistakes?
The most common problems were related to sources: in 31% of cases, data was missing or incorrectly attributed.
Accuracy errors came in second place (30%), when assistants provided outdated or fictitious information.
For example, ChatGPT stated in a test scenario that Pope Francis is still the current pontiff, although according to the study, he had already died and was replaced by Pope Leo XIV.
Copilot instead referred to a 2006 BBC news story when asked about current bird flu research.
Gemini – most errors, Perplexity – the best result
Among the four systems, Gemini was the worst performer, with 76% of the answers containing errors – twice as many as its competitors.
For comparison:
- Copilot – 37%,
- ChatGPT – 36%,
- Perplexity – 30%.
Assistants most often made mistakes in topics that change rapidly or require complex context, such as questions about Donald Trump’s trade policy or geopolitical events.
Read us on Telegram: important topics – without censorship
The popularity of AI in the news is growing
Despite the high error rate, artificial intelligence continues to become a source of information for more and more people.
According to the Reuters Institute and Oxford University, the number of users receiving news through AI doubled in a year, with the largest growth recorded in Argentina, the United States, and among 18-24 year olds.
What it means for the media
EBU researchers warn that trusting AI’s false answers could undermine the role of journalism.
“Artificial intelligence assistants are still not a reliable way to get news,” the EBU noted.
Experts add that although the models show some improvement over previous BBC tests, the level of accuracy remains unacceptable for journalistic standards.
The European Broadcasting Union (EBU) is the world’s largest association of public broadcasters, which includes, among others, the BBC, ARD, France Télévisions, RAI, Suspilne Ukraine and others.
The organization regularly monitors the use of new technologies in journalism, including artificial intelligence, blockchain and recommendation algorithms.
Watch us on YouTube: important topics – without censorship
