Nick Kashani Motlagh

Seeking Research Scientist / Applied Scientist roles · LLM evaluation & reliability

Nick Kashani Motlagh

I build models that know when not to answer.

  • Answer
  • Abstain
  • Refine

PhD candidate at Ohio State's Computer Vision Lab. I successfully defended my dissertation on July 8, 2026, with the degree expected in August. Answering Under Uncertainty studies three points where directly returning a model's current best answer may not be justified: abstention from an unreliable prediction, evidence use under ambiguity, and whether refining a draft is more likely to repair it than harm it.

Now Current work: when should a QA system trust its draft answer, refine it with retrieved evidence, or abstain? The manuscript is under review at ACL Rolling Review; its title and author list remain withheld during review.

Resume (PDF) Email me
Peer-reviewed
4 papers
First-author papers
4
Award
Springer Best Paper
Available
Aug 2026

Toolkit

Languages & ML

  • Python
  • PyTorch
  • Hugging Face
  • NumPy
  • scikit-learn

Systems & scale

  • Slurm
  • Singularity
  • Distributed training
  • GPU clusters
  • Git

Research areas

  • LLM evaluation
  • Retrieval-augmented generation
  • Selective prediction
  • Calibration
  • Uncertainty quantification
  • Abstention
  • Multimodal systems

PhD dissertation · successfully defended July 8, 2026

Answering Under UncertaintyAbstention, Ambiguity, and Recoverability

The dissertation asks what an AI system should measure when direct return is not yet justified: whether to withhold an unreliable output, whether available evidence supports an intended interpretation, and whether refining a draft is more likely to repair it than harm it.

  1. 01 Output uncertainty

    Is the current prediction reliable enough to return?

    Natural reject option

    Abstention when no rejection cost or coverage target is given: per-class thresholds that maximize selected accuracy while requiring the rejected region to behave like genuine confusion.

    Abstain Springer Best Paper Award at ISVC 2022 · MVA 2025 journal extension

  2. 02 Input ambiguity

    Does available evidence move the model toward the intended meaning?

    Measuring evidence use

    ImageCoMMuTE-style metrics for multimodal translation: does the correct image lower the model's uncertainty for the correct translation, relative to a misleading image? The metrics test image dependence directly instead of inferring it from aggregate scores.

    Use evidence WMT 2024

  3. 03 Post-answer recoverability

    Will a second look make the answer better or worse?

    Measuring recoverability

    Compares direct and evidence-refined answers on the same questions, distinguishing preserved, repaired, harmed, and unrecovered outcomes for a fixed QA stack before evaluating answer, refine, or abstain policies.

    Refine Manuscript under review at ACL Rolling Review

Manuscript under review

Selective RAG-QA: answer, refine, or abstain

This work compares direct and evidence-refined answers in retrieval-augmented QA, separating preserved, repaired, harmed, and unrecovered outcomes. It then evaluates answer, refine, or abstain policies without treating draft confidence as a complete estimate of recoverability. Results are reported for a fixed stack on NQ-Open, TriviaQA, and PopQA.

The submission title, author list, and preprint remain withheld during double-blind review.

Under review at ACL Rolling Review

Manuscript note Related papers

News

Latest news

View archive
  1. Successfully defended my PhD dissertation, ‘Answering Under Uncertainty: Abstention, Ambiguity, and Recoverability,’ in Computer Science and Engineering at The Ohio State University.
  2. Prepared an ARR submission on retrieval-augmented selective QA: deciding when to answer, refine, or abstain.
  3. Reported LLM reject-option training and evaluation results for DCS Corp / AFRL.
  4. Joined DCS Corp (AFRL) as Technical Analyst II working on LLM reject-option training and evaluation.
Built

Public research code and data.

Research implementations, calibration utilities, and dataset-collection tools, with their scope stated explicitly.

Repo

learning-idk

Companion code for ISVC 2022 / MVA 2025: per-class reject-option classification with binomial threshold search.

  • Python
  • PyTorch
  • selective prediction
  • Learn per-class reject thresholds from precomputed classifier logits and labels.
  • Evaluate select accuracy, reject accuracy, and coverage for the learned thresholds.
  • Run the included calibration and threshold-analysis utilities; no coverage-curve exporter is bundled.

Repo

calibration

PyTorch calibration utilities for histogram binning, global temperature scaling, and class-wise temperature scaling.

  • Python
  • PyTorch
  • calibration
  • Run global and class-wise temperature scaling on classifier logits.
  • Produce the calibration plots and expected-calibration-error summaries implemented in the repository.
  • Use the utilities as building blocks; this repository is not the WMT evaluation harness.

Repo

construction-site-satellite-imagery-collection

Companion code for OpenStreetMap-guided temporal satellite imagery collection and annotation.

  • Python
  • OpenStreetMap
  • remote sensing
  • Extract candidate construction-site polygons from OpenStreetMap data.
  • Download temporal satellite imagery for the extracted regions.
  • Inspect the released sample dataset and notebook; annotation and split-export tools are not included.
All code and data
Selected work

Papers and reports.

Peer-reviewed papers, related public code and data, and a current manuscript note with explicit review status.

Manuscript under review / 2026

Manuscript Note: Knowing When to Answer, Refine, or Abstain

Author list withheld during review

Uses paired direct and evidence-refined outcomes to evaluate answer, refine, or abstain policies for a fixed retrieval-augmented QA stack.

  • Compares direct and evidence-refined answers on 25,870 held-out questions, distinguishing preserved, repaired, harmed, and unrecovered outcomes.
  • Evaluates answer, refine, or abstain policies for a fixed model-retriever-corpus stack instead of treating draft confidence as a complete estimate of recoverability.

Why it matters

Separates draft correctness from whether refinement is likely to repair or harm the answer.

  • selective prediction
  • adaptive QA
  • retrieval-augmented generation
  • abstention

ISVC 2022 / 2022

Learning When to Say "I Don't Know"

N. Kashani Motlagh, J. Davis, T. Anderson, J. Gwinnup

Per-class reject thresholds estimated from validation statistics without requiring a rejection cost, target accuracy, or target coverage.

  • Springer Best Paper Award at ISVC 2022; later extended in the MVA 2025 journal version.
  • At the ImageNet δ=.75 operating point, B-CDF improved select accuracy by 0.4 percentage points and coverage by 1.3 points versus an uncalibrated global 0.5 threshold.
Read paper arXiv Code

Why it matters

Springer Best Paper Award

  • vision
  • reject option
  • selective accuracy
  • ImageNet

Related public code

Machine Vision and Applications / 2025

Naturally Constrained Reject Option Classification

N. Kashani Motlagh, J. Davis, T. Anderson, J. Gwinnup

Journal extension of the ISVC 2022 work, evaluating per-class binomial reject thresholds on synthetic, image, and text classification datasets.

  • Extends the natural reject-region constraint across controlled synthetic data, benchmark image classification, and text classification.
  • Reports transfer behavior on CINIC10 and long-tailed iNaturalist19 alongside explicit selective-accuracy, reject-accuracy, and coverage tradeoffs.
Read paper arXiv Code

Why it matters

ImageNet at δ=.75: +0.4 percentage points select accuracy and +1.3 points coverage versus uncalibrated global 0.5 thresholding.

  • vision
  • reject option
  • calibration
  • ImageNet

Related public code

WMT 2024 / 2024

Assessing the Role of Imagery in Multimodal Machine Translation

N. Kashani Motlagh, J. Davis, J. Gwinnup, G. Erdmann, T. Anderson

Contrastive evaluation of WMT 2024 multimodal MT systems shows measurable dependence on paired visual context.

  • Introduced imagery-aware contrastive probes for testing whether model scores change under mismatched visual context.
  • Evaluated three English-to-French multimodal model families, plus gated variants, under matched and mismatched visual context.
Read paper

Why it matters

Image evidence can affect model uncertainty without changing the final translation preference.

  • multimodal MT
  • vision-language models
  • evaluation
  • WMT

Related public code

All publications
Experience

Roles.

Research, teaching, internships, and applied evaluation work.

Role

Technical Analyst II — DCS Corp (sponsored by Air Force Research Laboratory)

Dayton, OH / May 2025 — Present

  • Train and evaluate abstention-augmented LLM policies for retrieval-augmented QA, measuring when evidence-based refinement repairs a draft answer and when it harms one.
  • Build evaluation harnesses and calibration dashboards for comparing LLM policy variants across coverage, utility, and out-of-distribution behavior.

Role

Graduate Research Associate — Computer Vision Lab

Ohio State University · Columbus, OH / Aug 2021 — Present

  • Build selective-prediction systems for vision, multimodal, and language tasks, advised by Prof. Jim Davis.
  • Designed imagery-aware contrastive metrics for multimodal machine translation (WMT 2024), measuring whether translations depend on paired visual context.

Role

Graduate Teaching Associate — Machine Learning & NLP

Ohio State University · Columbus, OH / Aug 2023 — Dec 2025

  • Supported machine learning and natural language processing courses through grading, office hours, and lab materials.
  • Served as a graduate teaching associate through May 2025 and as a course grader through December 2025.

Role

Graduate Research Intern — Air Force Research Laboratory

Dayton, OH / Summers 2022–2024

  • Summer 2024: Adapted and trained JEPA and MAE transformers in a distributed Slurm/Singularity setup for multimodal EO/SAR representation learning in low-data regimes.
  • Summer 2023: Developed Reject Option Beam Search to improve machine translation quality at large beam widths.

Work with me

Research Scientist, Applied Scientist, and ML Engineer roles.

I successfully defended my dissertation on July 8, 2026, and expect to complete my PhD in August. I am open to coordinating start timing where needed. Best fit: teams working on LLM evaluation, calibration, selective prediction, retrieval-augmented QA, or reliability infrastructure. I am comfortable with experiment design, PyTorch training code, distributed cluster runs, evaluation harnesses, and metrics/reporting layers. U.S. citizen with five summers of AFRL research experience; federal roles welcome.

Email me View resume

At a glance

  • PhD candidate, The Ohio State University · dissertation defended July 8, 2026 · degree expected August 2026
  • Research Scientist / Applied Scientist / ML Engineer · selective prediction, calibration, LLM evaluation
  • First author on 4 published papers · 1 manuscript under review · Springer Best Paper Award at ISVC 2022
  • Current manuscript on retrieval-augmented selective QA · under review at ACL Rolling Review
  • Python · PyTorch · Hugging Face · Slurm/Singularity · RAG evaluation
  • U.S. citizen · five summers of AFRL research experience · Columbus OH, open to relocation / remote
LinkedIn GitHub Scholar ORCID Email

Recent roles

  1. Technical Analyst II — DCS Corp (sponsored by Air Force Research Laboratory)

    Dayton, OH / May 2025 — Present

  2. Graduate Research Associate — Computer Vision Lab

    Ohio State University · Columbus, OH / Aug 2021 — Present

  3. Graduate Teaching Associate — Machine Learning & NLP

    Ohio State University · Columbus, OH / Aug 2023 — Dec 2025

  4. Graduate Research Intern — Air Force Research Laboratory

    Dayton, OH / Summers 2022–2024