2026-07-01 · verification
Same words, opposite claim
I built a checker that verifies AI claims against the real papers they cite, then paid two frontier models to break it. They did, twice. Both breaks were the same lesson: grounding checkers fail in the direction of agreement, and the fix is to accept less, not to get smarter.
I've been building a verification tool that answers one question: does the paper actually say that? You hand it an AI-generated claim and the citation attached to it, it resolves the citation to the real paper through the scholarly registries, retrieves what the paper says, and renders a verdict: supported, contradicted, or not verifiable.
The whole value of a tool like this is that it never says "supported" when the answer is no. A verifier that occasionally blesses a wrong claim is worse than no verifier, because it launders the error: it takes something a careful reader might have doubted and stamps it. False support is the one failure the product can't have.
So before trusting it, I did what I now do to every system whose failures are expensive: I put it in front of adversaries whose only job was to break it. Two frontier models, prompted to attack the design, twice. Both rounds found a real break, and in hindsight both were the same break wearing different clothes.
Round one: the checker liked antonyms
The default grounding check scored lexical overlap between the claim and the source sentence: how much of the claim's vocabulary shows up in the evidence. It's deterministic and fast and needs no model, which is why it existed as the baseline.
The attack was one sentence long. Claim: "the intervention produced excellent outcomes." Source: "the intervention produced terrible outcomes." Overlap: nearly total. Verdict: supported.
The checker had no concept of polarity. "Excellent" and "terrible" were just two tokens among many shared ones, and the sharing was what scored. A claim and its exact opposite looked almost identical to the thing whose job was to tell them apart.
The fix wasn't to make the checker cleverer, it was to make it stricter about what it was for. A lexical check earns a "supported" only on a near-verbatim match against a single sentence of the source, the case where the claim is essentially quoting the paper. Anything looser either goes to the entailment checker, which does understand polarity, or it abstains. The cheap check kept its job and lost its overreach.
Round two: the checker couldn't read direction
Round two came after that fix, from attackers told the antonym door was closed. They found the next one. Claim: "smoking causes cancer." Source: "cancer causes smoking." Every word matches. Near-verbatim, single sentence. The strict version of the lexical check, the one that had just survived round one, scored it supported.
Same words, opposite claim. Nothing about vocabulary distinguishes a statement from its reversal, because the meaning lives in the order, and the checker was scoring a bag of words. The round-one fix had tightened how much had to match and said nothing about sequence.
The fix this time changed what "match" means: coverage of adjacent word pairs, not individual words. "Smoking causes" and "causes cancer" appear in the source only if the source actually runs in that direction. Bigrams encode order, so the swap attack stopped scoring, and the check is still deterministic and fast with no model in the loop. It just measures a property that carries the meaning.
The whole fix is a few lines:
def bigrams(tokens: list[str]) -> set[tuple[str, str]]:
return set(zip(tokens, tokens[1:]))
def bigram_coverage(claim: str, source: str) -> float:
cb = bigrams(tokenize(claim))
if not cb:
return 0.0
return len(cb & bigrams(tokenize(source))) / len(cb)
You can try both checkers on the actual attacks here. Type your own claim and source, or load the attacks from each round:
Word overlap (the broken checker)
Shared vocabulary, any order. Antonym swaps and reversals still score.
Bigram coverage (the fix)
Adjacent word pairs must appear in the source in the same order.
The pattern in both breaks
Both attacks worked for the same reason: the checker was measuring similarity when the job was measuring assertion. Any grounding check built on "how much does this look like the source" will fail in the direction of agreement, because looking like the source is what a well-formed wrong claim does best. Fabricated claims arrive built from the source's own vocabulary, rearranged.
That reframing changed how I hardened the rest of the system, because versions of the same bias were hiding everywhere.
The entailment model's output labels were mapped by position, and if a model returned them in an unexpected order, the code guessed. Now it refuses to guess: an unrecognized label mapping fails closed to "not verifiable" instead of optimistically to "supported."
Citation resolution could fuzzy-match a vague title to a confidently wrong paper, agreement again at the retrieval layer. Now a fuzzy match requires a minimum amount of query to stand on, and a short vague string can't resolve to anything.
And "contradicted" got its own confidence threshold, because a verdict that accuses the claim of being wrong is also an assertion, and it has to clear its own bar rather than being the default when support fails.
The test suite went from 14 tests to 96, and most of the new ones are attacks: antonyms, swaps, garbled labels, unresolvable citations, sources that discuss the topic without asserting the claim. On the small benchmark of labeled cases, the number I optimize is false support, and it's zero, because every fix above was allowed to cost recall and none were allowed to cost precision.
If you're building one of these
If you're building anything that grades claims against evidence, a RAG faithfulness check, a citation verifier, a compliance reviewer, assume your checker is biased toward agreement, because it is.
Ask what property of the text your check measures, and whether meaning survives when that property is held constant. Word overlap survives negation and reversal, that's the whole bug, and it's one sentence to test.
Let cheap checks keep narrow jobs. When the lexical check gets fooled, don't delete it. Confine it to the near-verbatim cases it's good at, and let everything else abstain or escalate.
And pay for adversaries. My own tests were written by the person who designed the checker, which means they tested the design's intentions. The attacks that mattered came from outside the design's assumptions. Two rounds of hostile review cost less than one user discovering that your verification tool verifies things that are false.
The tool exists because AI output is full of citations nobody resolves and claims nobody checks against the source. The uncomfortable part of building it was finding that the checking layer wants to agree with you too, and that hardening it mostly meant teaching it to accept less.