# Nick Kashani Motlagh — full site text

Source: https://nmotlagh.github.io/ · Generated 2026-08-10


## About

I work on when a machine learning system should answer, weigh evidence, revise, or stay quiet. Every model ships with a confidence signal, and every one of them answers a slightly different question than the one the next decision actually depends on. My dissertation, *Answering Under Uncertainty*, studies three places where that gap bites: abstaining from an unreliable prediction, checking whether evidence really moved the model, and deciding whether a second pass will repair a draft answer or ruin it.

I write training code in Python and PyTorch, build the evaluation harnesses around it, and run distributed experiments on Slurm and Singularity clusters. My most recent study spans 25,870 held-out questions and roughly 400 H200-GPU-hours.

I earned my PhD at The Ohio State University in August 2026, advised by Jim Davis, with graduate minors in mathematics and high-performance computing. I did my M.S. (2025) and B.S. with Honors (2021) there too. I am first author on four published papers — reject-option classification (Springer Best Paper Award at ISVC 2022, extended in Machine Vision and Applications 2025), evidence-use metrics for multimodal translation (WMT 2024), and a temporal satellite-imagery collection framework (ICCV Workshop 2021) — with a fifth manuscript under review. I have reviewed for CVPR, ICCV, and ECCV.

I am looking for Research Scientist, Applied Scientist, and ML Engineer roles, available to start now, and I am most useful to teams working on LLM evaluation, calibration, retrieval-augmented systems, or reliability infrastructure. Based in Columbus, OH, open to relocation and remote. U.S. citizen with five summers of AFRL-sponsored research experience; federal roles welcome.

## Research

### 01. Abstention without a budget (Abstain)

Question: Should it answer at all?
Venue: Springer Best Paper Award, ISVC 2022 · extended in Machine Vision and Applications, 2025

Problem: A classifier names a class for every input, including the ones it is plainly confused by. The standard fix — threshold the softmax — needs a rejection cost or coverage target that real deployments rarely have. On binary problems it rejects nothing at all, since one of two softmax scores is always at least 0.5.

Approach: B-CDF learns per-class abstention thresholds from a validation set, post-hoc, with no retraining. The constraint is inverted: rather than targeting accuracy on what the model keeps, it requires the rejected set to look like a coin flip — so a model cannot inflate its score by quietly discarding answers it had right.

Headline result: 88.3% → 97.8% — CIFAR-100 selective accuracy, at 77.3% coverage

- Answering everything: 11.7 (mistakes per 100 predictions)
- Answering only what it keeps: 2.2 (mistakes per 100 predictions)

The same result read as errors, after abstaining on 22.7% of inputs.

ImageNet moves 88.4% → 97.4% at 79.7% coverage. Verified across four vision benchmarks, three text benchmarks and eight synthetic datasets, from 2 to 1,000 classes.

### 02. Measuring whether the image mattered (Weigh evidence)

Question: Is it really using the evidence?
Venue: WMT 2024

Problem: Multimodal translation systems get an image to resolve an ambiguous sentence — “that's lots of bucks” is about deer or about dollars. But a model that always guesses “dollars” still scores about 50%, so the benchmarks could not separate genuine evidence use from a lucky prior. The field largely concluded that images do not help.

Approach: ImageCoMMuTE holds the translation fixed and swaps the image, then asks whether the correct image lowers the model's perplexity for the correct translation. That is an intervention on the evidence channel rather than an inference from aggregate scores. Group variants give credit only when a model resolves both readings of the same ambiguity.

Headline result: 81% vs. 63% — how often VGAMT reads the image correctly, vs. how often that changes its translation

- Reads the image correctly: 81% (% of ambiguous sentences)
- …and changes the translation: 63% (% of ambiguous sentences)

VGAMT, English→French. Give it a perfect image and the gap widens: 92% vs. 34%.

Hand the model a perfect image and evidence use climbs to 92%, while translation accuracy barely moves — 26% → 34%. The images were working; the text prior was overriding them. A fusion problem, not an image problem, and aggregate BLEU hides it entirely.

### 03. When retrieval makes the answer worse (Refine)

Question: Will a second pass help or hurt?
Venue: Dissertation chapter · manuscript under review

Problem: RAG systems revise their own draft answers and average accuracy goes up, so refinement looks free. The average hides the trade: the same retrieval step that repairs one answer silently overwrites another that was already correct.

Approach: Score both branches for every question and label each outcome preserved, repaired, harmed, or unrecovered — the value of refining is then exactly repair minus harm. A two-head policy routes each question to answer, refine, or abstain.

Headline result: repairs 10.8%, harms 8.1% — across 25,870 held-out questions on NQ-Open, TriviaQA and PopQA

Confidence cannot predict which you get, so the policy reads the retrieved passages instead — holding the accuracy of always-refine while cutting wrong answers from 47.5% to 10.6%.

## Publications

### When Retrieval Makes the Answer Worse

Authors: Nick Kashani Motlagh
Venue: Dissertation chapter · manuscript under review (2026)
Page: https://nmotlagh.github.io/publications/adaptive-qa-abstention/
BibTeX: https://nmotlagh.github.io/publications/adaptive-qa-abstention.bib


Summary: Retrieval-based refinement repairs 10.8% of draft answers and destroys 8.1% that were already correct. Confidence cannot tell you which you are about to get.

- Across 25,870 held-out questions on NQ-Open, TriviaQA and PopQA, retrieval-based refinement repairs 10.8% of answers and harms 8.1% — every dataset and retriever slice contains both.
- Confidence does not predict recoverability: a verifier that sorts draft accuracy from 4.4% to 93.9% across deciles still values refinement at +23.7 points in the lowest decile and net negative in the top four.
- An evidence-aware two-head policy holds the accuracy of always-refine while cutting wrong answers from 47.5% to 10.6%, closing two thirds of the gap to an oracle.

Retrieval-augmented QA systems revise their own draft answers, and average
accuracy goes up — so the technique looks free. The average hides the trade.
The same retrieval step that repairs one answer silently overwrites another
that was already right.

This chapter measures the trade directly. For every question it runs both
branches — answer directly, and answer again with retrieved evidence — scores
both against the same answer set, and labels each outcome **preserved**,
**repaired**, **harmed**, or **unrecovered**. On a 1/0 correctness scale the
value of refining reduces exactly to repair minus harm.

Across 25,870 held-out questions from NQ-Open, TriviaQA and PopQA, with DPR and
BM25 retrieval over a shared Wikipedia source, refinement repairs 10.8% of
answers and harms 8.1%. Roughly two thirds of all repairs are found by exactly
one of the two retrievers, so recoverability is not a property of the question —
it is a property of the question and the stack together.

Confidence cannot substitute for this measurement. A learned verifier sorts
draft accuracy from 4.4% in the lowest decile to 93.9% in the highest, so it
works as a correctness signal — but the value of refining is +23.7 points in
that lowest decile, crosses zero around the sixth, and is negative in the top
four. Adding confidence proxies to the controller made it worse; only the
retrieved passages themselves helped.

The resulting two-head policy estimates each branch separately and routes every
question to answer, refine, or abstain. It holds the same fraction correct as
always-refine while cutting wrong answers from 47.5% to 10.6%, closing about two
thirds of the gap between always-refine and an oracle with the paired labels.
A 480-row blind human audit agrees with the automatic outcome labels 90.8% of
the time.

*This is chapter 5 of my dissertation, defended July 2026. A version is under
review at ACL Rolling Review; the submission title and author list are not
listed here during the anonymity period.*

---

### Naturally Constrained Reject Option Classification

Authors: N. Kashani Motlagh, J. Davis, T. Anderson, J. Gwinnup
Venue: Machine Vision and Applications (2025)
Page: https://nmotlagh.github.io/publications/naturally-constrained-reject-option-classification/
BibTeX: https://nmotlagh.github.io/publications/naturally-constrained-reject-option-classification.bib
DOI: https://doi.org/10.1007/s00138-024-01620-5
arXiv: https://arxiv.org/abs/2209.04944
Code: https://github.com/osu-cvl/learning-idk
Publisher page: https://link.springer.com/article/10.1007/s00138-024-01620-5

Summary: Journal extension evaluating per-class binomial reject thresholds across 4 vision, 3 text and 8 synthetic datasets, from 2 to 1,000 classes.

- Extends the natural reject-region constraint across controlled synthetic data, benchmark image classification, and text classification.
- Reports transfer behavior on CINIC10 and long-tailed iNaturalist19 alongside explicit selective-accuracy, reject-accuracy, and coverage tradeoffs.

We present the journal extension of our reject-option work. The method learns per-class softmax thresholds that maximize select accuracy while constraining the rejected region to behave like genuine correct-versus-incorrect confusion. Experiments span synthetic, image, and text classification, with separate reporting of select accuracy, reject accuracy, and coverage. The method does not require a user-specified rejection cost, target select accuracy, or target coverage, though its significance parameter still controls the operating tradeoff.

---

### Assessing the Role of Imagery in Multimodal Machine Translation

Authors: N. Kashani Motlagh, J. Davis, J. Gwinnup, G. Erdmann, T. Anderson
Venue: WMT 2024 (2024)
Page: https://nmotlagh.github.io/publications/assessing-imagery-in-multimodal-mt/
BibTeX: https://nmotlagh.github.io/publications/assessing-imagery-in-multimodal-mt.bib
DOI: https://doi.org/10.18653/v1/2024.wmt-1.130
PDF: https://aclanthology.org/2024.wmt-1.130.pdf
Publisher page: https://aclanthology.org/2024.wmt-1.130/

Summary: The best multimodal translation system reads the image correctly 81% of the time, but that only changes its translation 63% of the time. 'Images don't help' was partly a measurement artifact.

- Introduced imagery-aware contrastive probes for testing whether model scores change under mismatched visual context.
- Evaluated three English-to-French multimodal model families, plus gated variants, under matched and mismatched visual context.

Published at WMT 2024, this work designs imagery-sensitive contrastive metrics for multimodal machine translation and evaluates three English-to-French model families. The experiments hold a candidate translation fixed while varying matched and mismatched visual context, testing whether the image moves model uncertainty in the intended direction. These score-based diagnostics do not by themselves establish generated-translation quality or causal evidence use.

---

### Learning When to Say “I Don’t Know”

Authors: N. Kashani Motlagh, J. Davis, T. Anderson, J. Gwinnup
Venue: ISVC 2022 (2022)
Award: Springer Best Paper Award
Page: https://nmotlagh.github.io/publications/learning-when-to-say-i-dont-know/
BibTeX: https://nmotlagh.github.io/publications/learning-when-to-say-i-dont-know.bib
DOI: https://doi.org/10.1007/978-3-031-20713-6_15
arXiv: https://arxiv.org/abs/2209.04944
Code: https://github.com/osu-cvl/learning-idk
Publisher page: https://link.springer.com/chapter/10.1007/978-3-031-20713-6_15

Summary: Per-class abstention thresholds that need no rejection cost or coverage target: CIFAR-100 selective accuracy climbs from 88.3% to 97.8% at 77.3% coverage.

- Springer Best Paper Award at ISVC 2022; later extended in the MVA 2025 journal version.
- At the ImageNet δ=.75 operating point, B-CDF improved select accuracy by 0.4 percentage points and coverage by 1.3 points versus an uncalibrated global 0.5 threshold.

We introduce a reject-option classification framework that estimates per-class softmax thresholds from validation data. The method targets the coverage/selective-accuracy tradeoff directly rather than applying a single global confidence threshold across all classes.

---

### A Framework for Semi-automatic Collection of Temporal Satellite Imagery for Analysis of Dynamic Regions

Authors: N. Kashani Motlagh, A. Radhakrishnan, J. Davis, R. Ilin
Venue: ICCV 2021 Workshop on LUAI (2021)
Page: https://nmotlagh.github.io/publications/framework-for-semi-automatic-collection/
BibTeX: https://nmotlagh.github.io/publications/framework-for-semi-automatic-collection.bib
PDF: https://openaccess.thecvf.com/content/ICCV2021W/LUAI/papers/Motlagh_A_Framework_for_Semi-Automatic_Collection_of_Temporal_Satellite_Imagery_for_ICCVW_2021_paper.pdf
Code: https://github.com/osu-cvl/Construction-Site-Satellite-Imagery-Collection
Publisher page: https://openaccess.thecvf.com/content/ICCV2021W/LUAI/html/Motlagh_A_Framework_for_Semi-Automatic_Collection_of_Temporal_Satellite_Imagery_for_ICCVW_2021_paper.html

Summary: OpenStreetMap-guided imagery collection and labeling tools for building temporal satellite datasets for dynamic-region analysis.

- Combined OpenStreetMap extraction, polygon filtering, and temporal imagery download scripts in a public Python workflow.
- Released the collection code and a sample construction-site satellite-imagery dataset under the repository's GPL license.

We detail a semi-automatic pipeline for collecting temporal satellite imagery using OpenStreetMap metadata. The public repository provides scripts for extracting candidate construction-site regions and downloading imagery over time, plus a sample dataset and notebook. It does not include a general annotation application or a train/validation/test split exporter.

## Education

### Ph.D. in Computer Science and Engineering

The Ohio State University, Columbus, OH · Aug 2021 — Aug 2026
- Degree conferred August 2026. Dissertation: Answering Under Uncertainty: Abstention, Ambiguity, and Recoverability, defended July 8, 2026.
- Advised by Prof. Jim Davis. Minors in Mathematics and High-Performance Computing.

### M.S. in Computer Science and Engineering

The Ohio State University, Columbus, OH · Aug 2021 — May 2025
- GPA 3.70. Advised by Prof. Jim Davis.

### B.S. with Honors in Computer Science and Engineering

The Ohio State University, Columbus, OH · Aug 2017 — May 2021
- GPA 3.86. Minor in Mathematics.
- Maximus, Ten-Hai Lai, Ansel, and Name and Seal scholarships.

## Experience

### Technical Analyst II — DCS Corp

AFRL-sponsored · Dayton, OH · May 2025 — Present
- Built the paired-outcome evaluation behind a 25,870-question study of retrieval-augmented QA across NQ-Open, TriviaQA and PopQA, measuring when evidence-based refinement repairs a draft answer and when it destroys a correct one.
- Trained LoRA answer/refine/abstain controllers on 8× NVIDIA H200 GPUs (~400 GPU-hours for a clean reproduction), beating the strongest confidence baseline and closing two thirds of the gap to an oracle policy.
- Maintain the evaluation harness comparing LLM policy variants across coverage, utility, and out-of-distribution behavior, plus a 480-row blind human audit validating the automatic labels at 90.8% agreement.

### Graduate Research Associate — Computer Vision Lab

Ohio State University · Columbus, OH · Aug 2021 — Present
- Build selective-prediction systems for vision, multimodal, and language tasks, advised by Prof. Jim Davis.
- Developed per-class reject-option classification that lifts CIFAR-100 selective accuracy from 88.3% to 97.8% at 77.3% coverage, validated on 4 vision, 3 text and 8 synthetic datasets (Springer Best Paper Award at ISVC 2022; MVA 2025 journal extension).
- Designed the ImageCoMMuTE contrastive metrics for multimodal machine translation (WMT 2024), which separated genuine image use (81%) from final translation preference (63%) and built a 1,540-image extended evaluation set.
- Publish and maintain the public research code for reject-option classification and calibration.

### Graduate Research Intern — AFRL

Dayton, OH · Summers 2022–2024
- Summer 2024: Adapted and trained JEPA and MAE transformers in a distributed Slurm/Singularity setup for multimodal EO/SAR representation learning in low-data regimes.
- Summer 2023: Developed Reject Option Beam Search for machine translation at large beam widths.
- Summer 2022: Built the end-to-end training procedure for Naturally Constrained Reject Option Classification.

### Graduate Teaching Associate — Machine Learning & NLP

Ohio State University · Columbus, OH · Aug 2023 — Dec 2025
- Taught and supported machine learning and NLP courses through grading, office hours, and lab materials.

### Undergraduate Research Intern — AFRL

Dayton, OH · Summers 2020–2021
- Summer 2021: Studied ensemble distillation for ambiguous instances.
- Summer 2020: Built a semi-automated system for temporal satellite imagery collection (ICCV 2021 workshop), released as the Construction-Site-Satellite-Imagery dataset.

### Undergraduate Research Associate — Computer Vision Lab

Ohio State University · Columbus, OH · 2020 — 2021
- Built semi-automatic labeling workflows for remote-sensing change detection, including Python tooling for collecting and preparing temporal satellite datasets.

### Summer Research Intern — Sii Canada / Concordia University

Montreal, QC · Summer 2019
- Built anomaly-detection dashboards on behavioral telemetry to prioritize follow-up experiments.

### Undergraduate Teaching Associate — Discrete Structures & Algorithms

Ohio State University · Columbus, OH · 2018 — 2019
- Led recitations and office hours for discrete structures and algorithms.

## Professional service

- Reviewer: ICCV 2023, CVPR 2023, ECCV 2022, CVPR 2022
- Volunteer: HackOHI/O 2023

## Code and data

### calibration

https://github.com/osu-cvl/calibration

PyTorch calibration utilities for histogram binning, global temperature scaling, and class-wise temperature scaling.

Stack: Python, PyTorch, calibration

What the repository does:
- Run global and class-wise temperature scaling on classifier logits.
- Produce the calibration plots and expected-calibration-error summaries implemented in the repository.
- Use the utilities as building blocks for calibration experiments.

### construction-site-satellite-imagery-collection

https://github.com/osu-cvl/Construction-Site-Satellite-Imagery-Collection

Companion code for OpenStreetMap-guided temporal satellite imagery collection and annotation.

Stack: Python, OpenStreetMap, remote sensing

What the repository does:
- Extract candidate construction-site polygons from OpenStreetMap data.
- Download temporal satellite imagery for the extracted regions.
- Inspect the released sample dataset and notebook; annotation and split-export tools are not included.

### learning-idk

https://github.com/osu-cvl/learning-idk

Companion code for ISVC 2022 / MVA 2025: per-class reject-option classification with binomial threshold search.

Stack: Python, PyTorch, selective prediction

What the repository does:
- Learn per-class reject thresholds from precomputed classifier logits and labels.
- Evaluate select accuracy, reject accuracy, and coverage for the learned thresholds.
- Run the included calibration and threshold-analysis utilities; no coverage-curve exporter is bundled.

## Frequently asked questions

### What does Nick Kashani Motlagh work on?

Reliability of machine learning systems under uncertainty — specifically when a model should answer, weigh evidence, revise its answer, or abstain. The work spans selective prediction and reject-option classification for classifiers, evidence-use metrics for multimodal systems, and answer/refine/abstain policies for retrieval-augmented question answering with large language models.

### Is he available for hire, and when?

Yes. He is available now for Research Scientist, Applied Scientist, and Machine Learning Engineer roles. He is based in Columbus, Ohio and is open to relocation or remote work.

### What is his education?

A PhD in Computer Science and Engineering from The Ohio State University, conferred August 2026 (dissertation defended July 8, 2026), advised by Prof. Jim Davis, with graduate minors in Mathematics and High-Performance Computing. He also holds an M.S. (2025) and a B.S. with Honors (2021) in Computer Science and Engineering from Ohio State.

### What has he published?

Four peer-reviewed first-author papers: “Naturally Constrained Reject Option Classification” (Machine Vision and Applications, 2025), “Assessing the Role of Imagery in Multimodal Machine Translation” (WMT 2024), “Learning When to Say I Don’t Know” (ISVC 2022, Springer Best Paper Award), and “A Framework for Semi-automatic Collection of Temporal Satellite Imagery” (ICCV Workshop 2021). A fifth manuscript, on retrieval-augmented selective QA, is under review and not yet accepted.

### Has he won any awards?

Yes — the Springer Best Paper Award at ISVC 2022 for “Learning When to Say I Don’t Know,” the reject-option classification work later extended into the Machine Vision and Applications journal version.

### What is his engineering experience, as opposed to research output?

He writes the training code, the evaluation harnesses, and the cluster orchestration himself. Recent work includes LoRA fine-tuning of answer/refine/abstain controllers on 8× NVIDIA H200 GPUs (roughly 400 GPU-hours for a clean reproduction) and a paired-outcome evaluation harness over 25,870 held-out questions. Day-to-day stack: Python, PyTorch, Hugging Face, FAISS, Slurm, and Singularity.

### Can he work on U.S. federal or defense contracts?

Yes. He is a U.S. citizen and has completed five summers of AFRL-sponsored research, and currently works as a Technical Analyst II at DCS Corp on AFRL-sponsored LLM reliability work. Federal and cleared-adjacent roles are welcome.

### How should someone contact him?

By email at kashanimotlagh.1@osu.edu. His CV is at https://nmotlagh.github.io/resume.pdf, code at https://github.com/nmotlagh, and publication record at https://scholar.google.com/citations?user=srZXFMcAAAAJ&hl=en and ORCID https://orcid.org/0000-0001-6229-6212.

## News

- 2026-07-08: Successfully defended my PhD dissertation, ‘Answering Under Uncertainty: Abstention, Ambiguity, and Recoverability,’ in Computer Science and Engineering at The Ohio State University.
- 2026-05-19: Prepared an ARR submission on retrieval-augmented selective QA: deciding when to answer, refine, or abstain.
- 2026-04-10: Reported LLM reject-option training and evaluation results for DCS Corp / AFRL.
- 2025-05-20: Joined DCS Corp (AFRL) as Technical Analyst II working on LLM reject-option training and evaluation.
- 2024-11-26: Journal extension on naturally constrained reject-option classification published online in Machine Vision and Applications.
- 2024-11-15: Accepted WMT 2024 paper on imagery-aware multimodal MT evaluations.