When Retrieval Makes the Answer Worse

Retrieval-augmented QA systems revise their own draft answers, and average accuracy goes up — so the technique looks free. The average hides the trade. The same retrieval step that repairs one answer silently overwrites another that was already right.

This chapter measures the trade directly. For every question it runs both branches — answer directly, and answer again with retrieved evidence — scores both against the same answer set, and labels each outcome preserved, repaired, harmed, or unrecovered. On a 1/0 correctness scale the value of refining reduces exactly to repair minus harm.

Across 25,870 held-out questions from NQ-Open, TriviaQA and PopQA, with DPR and BM25 retrieval over a shared Wikipedia source, refinement repairs 10.8% of answers and harms 8.1%. Roughly two thirds of all repairs are found by exactly one of the two retrievers, so recoverability is not a property of the question — it is a property of the question and the stack together.

Confidence cannot substitute for this measurement. A learned verifier sorts draft accuracy from 4.4% in the lowest decile to 93.9% in the highest, so it works as a correctness signal — but the value of refining is +23.7 points in that lowest decile, crosses zero around the sixth, and is negative in the top four. Adding confidence proxies to the controller made it worse; only the retrieved passages themselves helped.

The resulting two-head policy estimates each branch separately and routes every question to answer, refine, or abstain. It holds the same fraction correct as always-refine while cutting wrong answers from 47.5% to 10.6%, closing about two thirds of the gap between always-refine and an oracle with the paired labels.

This is chapter 5 of my dissertation, defended July 2026. The paper version, “Measuring and Predicting Recoverability in Retrieval-Augmented Revision,” incorporates the ACL Rolling Review feedback and is being prepared for arXiv.