RAGTruth: How Unsupported Words Become Expensive Publishing Errors

RAGTruth is a reminder that supplying retrieved evidence does not eliminate unsupported or contradictory statements. A generated answer can get most of a paragraph right while changing one word that materially alters the meaning. For publishers, the practical response is to review factual spans and decision conditions, rather than accepting a draft because it contains citations or accurately describes the general topic.

The most useful review asks where each consequential claim came from, whether the evidence supports its exact scope, and what a reader might do because of it. An unsupported adjective in a casual description and an unsupported guarantee in a buying guide deserve different priorities. Both should be visible in the record, but severity determines the repair and release decision.

What the historical research provides

The RAGTruth paper presents a corpus of nearly 18,000 generated responses annotated at case and word levels to study hallucinations in retrieval-augmented generation. The authors examine unsupported and contradictory claims despite retrieved context, and evaluate detection approaches. The May 2024 revision is historical research; its dataset is not a current error-rate census of all commercial answer systems. RAGTruth primary paper.

The transferable lesson is the value of locating the defect precisely. A whole-answer label can identify that something went wrong, while a span-level record shows which wording requires attention. The workflow below is an original editorial application of that idea. Its hypothetical products, calculations, and review policies are teaching examples, not results reported by the researchers.

Small wording changes can create large factual changes

Suppose a source says, “Selected accounts may request a refund within 30 days.” A draft says, “All accounts automatically receive a refund within 30 days.” Several words changed the offer: selected became all, may request became automatically receive, and an eligibility condition disappeared. The sentence remains on topic and retains the number, but it now asserts a different policy.

A reviewer checking only names and numbers could miss the defect. Quantifiers, modal verbs, conditions, negation, and time relationships deserve attention because they define the scope of a claim. The factual unit is not simply the most visible noun or statistic. It is the complete assertion the reader will understand.

Another source might say a tool supports importing a format, while the draft says it exports that format. The vocabulary overlap is high and the function is reversed. A generic similarity score cannot establish factual equivalence. The reviewer needs to inspect the relationship expressed by the verb and the direction of the operation.

An apparently minor modifier can also imply evidence that does not exist. “Tested compatibility” asserts an actual test; “documented compatibility” asserts a source statement; “possible compatibility” expresses a hypothesis. Choose the wording that matches the available evidence. Do not let an attractive adjective invent experience or validation.

Separate unsupported claims from contradictions

An unsupported claim lacks adequate evidence in the material being used. It may be true in the world, but the current evidence does not establish it. A contradiction conflicts with the supplied evidence. Both are problems for a grounded answer, but they should not be merged if the distinction helps repair.

If a product page says nothing about a restoration deadline, a draft’s one-hour guarantee is unsupported. If the page explicitly says restoration can take two business days, the one-hour guarantee contradicts the source. The first case may require finding suitable evidence or removing the claim. The second already supplies evidence that the claim must be corrected.

Also distinguish incorrect evidence use from an unreliable source. A draft can faithfully repeat a source that is outdated or mistaken. That is not the same failure as inventing an assertion beyond the source. Editorial approval needs both support checking and source suitability, including current version and relevant market.

Keep missing decision conditions visible. An answer can avoid unsupported sentences while omitting an exclusion that the user needs. That omission may not fit the same span label as an invented guarantee, but it still affects usefulness. Use a separate completeness check rather than forcing every defect into one hallucination category.

Build a claim and span record

Save the original draft and source version before editing. For each consequential claim, record the sentence, the exact problematic words, the source URL, the supporting or conflicting passage, the defect type, and its likely effect. This provides enough context for another reviewer to assess the decision without reconstructing the entire research process.

Use spans to locate errors, not to strip away meaning. A record containing only the word “all” is difficult to interpret. Retain the sentence and relevant condition so that the reviewer can see why the quantifier is unsupported. The span is a pointer into the claim, while the claim remains the unit of understanding.

Assign severity based on plausible consequence and the article’s purpose. A wrong plan eligibility statement can cause a purchase mistake. A wrong implementation step can waste hours or damage data. A minor descriptive overstatement may be less urgent. The policy should be consistent and should not become more permissive merely because the draft supports a preferred commercial message.

Record uncertainty honestly. If the source is ambiguous or inaccessible, mark the claim unresolved. Do not manufacture a clean label to complete the spreadsheet. A held claim with a named evidence dependency is more useful than a falsely approved claim that later appears in published copy.

A worked editorial audit

Imagine a fictional buying guide with 50 assessable factual claims. A reviewer identifies five unsupported claims and two contradictions. Suppose one claim contains both an unsupported guarantee and a contradictory plan name. The total number of affected claims is six, not seven, because the categories overlap on that claim.

The equal-weight affected-claim rate is six divided by 50, or 12%. That percentage is not enough to determine publication readiness. If the six claims concern pricing, eligibility, and safety-critical steps, the draft may be unusable until repaired. If they concern minor background details, the repair burden may differ. Keep severity beside the count.

Now suppose the reviewer marks 20 problematic words in a 2,000-word article. A one-percent span rate can sound reassuring while concealing a single false “guaranteed” commitment. Word-level proportions are useful for some comparisons, but they do not measure business risk directly. A small span can carry a large consequence.

These figures are illustrative arithmetic, not RAGTruth findings or a SEOS.co content audit. Their purpose is to show why case counts, claim counts, span counts, and severity should not be substituted for one another. A useful report defines the unit and retains the underlying evidence.

Repair claims without inventing new support

When a claim is overbroad, restore the source’s condition. Replace “all accounts receive refunds” with the actual eligibility and request process. Adding a citation after the unchanged claim does not repair the meaning. The wording itself must become accurate.

When evidence is absent, remove the assertion or explain the gap. Do not generate a plausible source name, invent a test, or imply that an expert reviewed the point. Research can continue for an authoritative source, but the draft should not treat the desired conclusion as established while that work remains incomplete.

When sources conflict, investigate version, geography, plan, and publication context. Two apparently contradictory statements may apply to different offers or periods. If the conflict remains unresolved, say so where the uncertainty matters. Choosing the more convenient source without explanation is not a reliable editorial policy.

After repair, reread the surrounding paragraph. A corrected condition can change the recommendation or calculation that follows. Fixing one sentence while leaving its downstream conclusion unchanged can preserve the substantive error in a different form. Trace the claim’s consequences through tables, summaries, metadata, and image captions.

Evaluate detectors as fallible tools

An automated detector can flag likely unsupported spans, but its output requires validation on the actual content domain. Include negation, numerical transformations, product versions, conditional policies, tables, and multi-source synthesis in the calibration set. A detector that handles simple factual summaries may struggle with technical procedures or contractual language.

Inspect false negatives carefully. A detector that misses a rare consequential guarantee may be unsuitable as a sole release gate even if aggregate agreement looks strong. Inspect false positives too, because excessive alarms can consume review time and encourage users to ignore warnings. The balance should reflect the actual editorial consequences.

Keep detector version and inputs stable during a comparative audit. Changing the model or prompt between draft batches can create an apparent quality shift unrelated to the manuscripts. Retain an overlap sample when updating the evaluator and document how labels changed.

Do not use a hallucination detector as an authorship detector. Unsupported claims can appear in human and machine writing. A clean support review does not prove human authorship, and an automated writing score does not establish factual quality. Report what the tool actually measures and keep unsupported marketing claims out of the handoff.

Prevent the same error upstream

Group related evidence before drafting. A feature and its exclusion should travel together in the research packet. A price should include currency, billing period, relevant market, and effective context. A performance result should include conditions and limitations. Better source packets reduce the chance that a writer fills missing details with assumptions.

Make hypothetical examples unmistakable. Label fictional companies, invented datasets, and scenario calculations before presenting their numbers. Do not introduce a scenario as an example and later refer to it as a measured case study. The factual status should remain consistent across the article and metadata.

Keep the outline from becoming authority. An early planning document may contain tentative claims or proposed sections. Before those statements enter the manuscript, verify them against evidence. A polished outline can preserve an invented detail through several revisions if nobody revisits its origin.

Maintain a short recurrent-error register. If drafts repeatedly turn conditional statements into guarantees, add targeted checks and better examples to the writing guidance. Repair the process at the point where the defect first enters, rather than repeatedly rewriting entire articles after completion.

The GEO evidence-improvement guide provides a broader source-preparation workflow around this claim-level review.

Review the published representation

An accurate manuscript can become misleading during implementation. A table header may disappear on mobile, a caption may lose a qualification, or a summary card may truncate a condition. Verify the actual delivered page and important text-only representations where publication is authorized. Draft review and live verification are different stages.

Check metadata as factual copy. A title promising a guaranteed result can contradict a cautious body. Image alt text can invent a measured relationship that the diagram does not show. A source-supported article does not excuse unsupported promotional fields around it. Keep the claim scope consistent throughout the asset.

Preserve release evidence and later corrections. If a published claim is wrong, record the original wording, source basis, correction, affected assets, and verification. Historical evidence helps identify whether the cause was research, drafting, review, rendering, or a later source change. Deleting the record weakens future troubleshooting.

Use the internal linking checklist when verifying that readers can reach the authoritative source and the next relevant explanation.

Questions for an editorial QA policy

Can RAG still hallucinate when the correct source is present? Yes, the existence of retrieved evidence does not force correct use. A generator can omit a condition, reverse a relationship, or add an unsupported detail. Inspect the answer-source relationship rather than relying on retrieval status alone.

Should every unsupported word block publication? The release decision should follow factual consequence and the ability to repair. Consequential unsupported claims must be corrected, removed, or held. A consistent policy prevents teams from ignoring serious defects because the overall error percentage appears small.

Does a citation make a sentence safe? Only if the source is suitable and supports the assertion as written. A relevant link can still fail to establish the claim. Review entailment, source validity, and decision completeness as separate checks.

What is the most useful review artifact? A version-bound record connecting claims and problematic spans to exact evidence, repair decisions, and severity. It should let another person understand what was found and why the final wording is supported. A single approved badge cannot provide that explanation.

The AI visibility mistakes guide explains why a citation count should remain separate from the factual accuracy outcome this audit examines.

Verify arithmetic as a separate operation

A source-supported number can still be transformed incorrectly. Check units, denominators, percentage points, relative changes, rounding, and overlapping categories directly. If a source reports a monthly fee, an annual total requires explicit assumptions about billing and discounts. A model’s confident explanation should not replace the calculation. Retain the inputs and formula so that a later source update can identify which derived statements and graphics need to change.

Span-level warranty audit worksheet

This fictional warranty example makes small factual edits visible. It is a proposed review tool, not a RAGTruth corpus item or a real manufacturer’s policy.