AI Chatbots Debunk Over 93% of Disinformation Narratives in New Study
Advanced AI models, while powerful, also present a new threat: their potential use by malicious actors to generate and disseminate fake news and propaganda at scale. However, a new study by NPR and NewsGuard suggests these same AI models can be part of the solution. Researchers found that when presented with queries centered on fake news, AI chatbots debunked false information in over 93% of cases.
For the study, NPR and NewsGuard, which specializes in rating the credibility of online news sources, identified 15 false narratives promoted by Russia, Iran, and China since December 2025. Two queries were developed for each narrative: one neutral and one assuming the narrative's truth. These 30 questions were fed into six popular US chatbots, OpenAI's ChatGPT, Google's Gemini, Microsoft's Copilot, Meta AI, SpaceX's Grok, and Anthropic's Claude, as well as three search engines: Google, Microsoft Bing, and DuckDuckGo.
NPR compared the AI responses and traditional search results against NewsGuard's fact-checks. A response was coded as debunking fake news only if it directly answered the query, provided an accurate yes/no answer or direct refutation, correctly analyzed the premise, and reached the correct conclusion. Out of 180 queries to chatbots, 168 (93.3%) successfully debunked the false narratives. Search engine results were also largely effective, debunking 85.8% of the falsehoods. However, AI summaries within search engines performed less well, debunking 64.5% of narratives, with significant variation between providers.
Chatbots and AI summaries demonstrated an advantage in providing context and explaining the origins of false narratives. For instance, when asked about a Russian disinformation campaign falsely blaming Ukraine for bombing a monastery, chatbots accurately identified the premise as false and explained it originated from a Russian campaign to deflect blame. Some AI responses also analyzed the credibility of the source spreading the fake news.
Despite these encouraging results, challenges remain. A study in Nature found that AI models provided more favorable answers about China's government when queried in Chinese compared to English, likely due to censorship mechanisms. Furthermore, the growing reliance on chatbots for information makes them targets for influence campaigns. Politico reported in August that Israel is running an influence campaign to shape AI chatbot responses on issues like Gaza and the IDF by feeding them pro-Israel articles. Such campaigns are expected to grow, posing a significant challenge for AI companies to maintain the integrity and reliability of their products.
