CITATION INTEGRITY · EVERY RECORD IS REAL

When an AI cites a paper, does the paper exist?

Language models invent citations, real-looking DOIs that resolve to nothing, and they cite papers that were retracted years ago, because a model cannot know a paper was withdrawn after its training cutoff. This board asks frontier models scientific questions, makes them cite the literature, and runs every citation through sourcecheck, an open-source source-integrity gate. The two headline numbers are pure registry lookups: no language model grades anything, because for these checks a language model is the thing being checked.

THE FINDING

Models flag the retractions they learned, and miss the ones they cannot know.

The same models were asked about famous old retractions and about prominent papers retracted after their training cutoff. On the old ones they usually cited the paper and noted it was retracted. On the recent ones they cited it as solid evidence with no idea, because the withdrawal is newer than their training data. sourcecheck flagged both. This is the value a smarter model cannot replace, and six of the post-cutoff misses came from the top-tier models.

THE THREE CHECKS

Existence, validity, presence. All lookups, no judgment.

Existence

Does the source resolve?

Every cited DOI is looked up in OpenAlex and Crossref. A DOI that resolves to no work is a fabrication, and DOI resolution is precision-first: no fuzzy fallback to a paper that merely looks similar. This is the headline ranking number, and no model can fake it.

Validity

Has it been retracted?

A resolved source is checked against OpenAlex's retraction flag and Crossref's retraction notices (sourced from Retraction Watch). This is the differentiator: no other grounding or hallucination leaderboard checks it, and it is exactly what a model cannot know past its cutoff.

Presence

Is the claim in the source?

For claims with a quotable span, sourcecheck checks the text is actually in the resolved source. This is the one axis that shades into judgment, so on this board it ships as an experimental secondary, never in the headline.

THE BOARD

Fabricated-citation rate, this run.

Every model gets the same questions and a protocol that lets it decline a citation, honest, never penalized. Fabrication rate is the fraction of the citations it did assert that resolve to no real work. A model that declines everything scores 0% but cites nothing, so coverage is shown alongside.

THE RETRACTION WALL

Real papers, retracted, cited anyway.

These resolved to real papers that have since been retracted for data fraud or error. Models cited them, often without noting the retraction, because the withdrawal happened after training. sourcecheck flagged every one. That gap, the model cites it, the registry knows it is dead, is the whole reason the library exists.

HONEST SCOPE

What this measures, and what it does not.

This checks whether citations are sourced, not whether claims are true. A real, non-retracted paper can still be cited for something it does not say. sourcecheck verifies the source is real, current, and (experimentally) contains the claim. It does not adjudicate truth, and it never pretends to.
Fabrication rate is the ranking; retraction is a safety showcase, not a pure penalty. Asking "which study reported the Surgisphere hydroxychloroquine result" requires naming a retracted paper to answer correctly, so citing it is not itself a failure. The value is that sourcecheck flags the retraction the model omitted. A fairer per-model retraction ranking (cited-without-noting-the-retraction) is a secondary column.
Title-only citations are a weaker existence signal than a DOI and are scored separately; a DOI that fails to resolve is an unambiguous fabrication.
One politically charged vaccine-safety retraction is deliberately excluded from the question bank and every example. Retraction coverage comes from data-fraud and error cases only.