Nick Kashani Motlagh
← Back to publications

arXiv preprint / 2026

First author

Return or Revise? Learning When Revision Helps Retrieval-Augmented QA

N. Kashani Motlagh, T. Anderson, J. Gwinnup, G. Erdmann

Paper arXiv

A retrieval-augmented QA system can improve an incorrect answer, but it can also replace a correct answer with a mistake. Average accuracy alone does not describe this tradeoff.

This dissertation research evaluates both branches for the same question: a direct answer and a revision using retrieved evidence. The paired outcomes distinguish answers that are preserved, repaired, harmed, or unrecovered. With binary correctness, the net change in accuracy is repair minus harm.

The central distinction is between confidence in an initial answer and recoverability under a particular revision process. The work studies how these signals inform a choice between returning the draft and revising it.

Setup

  • Data: 25,870 held-out questions from NQ-Open, TriviaQA, and PopQA.
  • Retrieval: three setups over Wikipedia: DPR dense retrieval, BM25, and BM25 reranked with MonoT5.
  • Models: Llama 3.1 8B Instruct writes the drafts and the revisions. Llama 3.3 70B Instruct is the automated semantic judge. gpt-oss-20b and OLMo 3 7B are secondary generator families.
  • Policies: LoRA-trained scorers estimate repair, harm, and tie outcomes to choose whether to return the draft or revise it. The optional third action is a draft-free retrieval-augmented answer, not abstention.

Rigor

  • Three training seeds, compared with paired run-level t-tests.
  • 10,000-replicate bootstrap intervals for the baselines.
  • Thresholds are selected on the dev split and frozen before test.

Stack

PyTorch, Hugging Face Transformers, vLLM, FAISS, and GPU jobs on Slurm.

Status

The manuscript is available as Return or Revise?, a public arXiv preprint posted as v1 on September 24, 2026. Code and artifacts will not be released.

My contribution includes the evaluation pipeline, model training, and experiments comparing decision policies. Work spanned OSU and DCS; completed at DCS. This overview describes the public preprint, which has not been peer-reviewed.

Cite

.bib 
@misc{kashanimotlagh2026retrieval,
  title     = {Return or Revise? Learning When Revision Helps Retrieval-Augmented QA},
  author    = {Kashani Motlagh, N. and Anderson, T. and Gwinnup, J. and Erdmann, G.},
  year      = {2026},
  url       = {https://arxiv.org/pdf/2609.30087},
  note      = {arXiv:2609.30087v1; preprint, not peer-reviewed}
}