Study finds search-augmented AI systems cite fake sources systematically
Original: Verified Misguidance: Measuring Structural Citation Failures in Search-Augmented LLMs
Source: arxiv.org ↗
Who: Authored by Yongsik Seo, Wooseok Jeong, Eunyoung Kim, Hyeonseo Jang, and Dongha Lee. No institutional affiliations are listed in the metadata, but the paper is posted to arXiv.
What's new: When AI assistants that search the web answer your questions, they add citations to look trustworthy — but this paper shows those citations are often misleading in ways that are hard to spot. The researchers built , a dataset of 11,200 real queries matched to 112,000 responses from ten different AI models across five providers, yielding 761,495 individual citation pairs to evaluate. The core finding is a pattern they name "verified misguidance" — the AI links to a real, working webpage, but the page either contradicts, distorts, or is simply the wrong kind of source for what the AI claimed.
How it works: Each citation is scored on three dimensions: whether the citation's purpose matches the user's intent, whether the source is from an appropriate domain for that topic, and whether the answer actually reflects what the source says. These are evaluated using expert-designed rubrics, including a five-level that grades how faithfully the AI's answer reflects the source. The researchers find a persistent trade-off: models that stay faithful to what sources say tend to pick unsuitable sources, and vice versa.
The numbers: 30.6% of all citations distort their sources in some way, and 27.1% come from domain-inappropriate sources — say, citing a cooking blog to back up a medical claim. At the response level, up to 96% of users encounter at least one structurally misleading citation in a given answer. Strikingly, differences between providers — not between individual models — account for 88–96% of the variation in citation quality, pointing to source-retrieval pipeline choices as the dominant factor.
Why it matters: Citations give AI answers an air of authority that most users never check. This work is the first to measure all three failure modes together, showing the problem is systematic and largely invisible to the reader. The implication is that fixing citation quality requires redesigning how these systems retrieve and select sources, not just improving the itself.