Assessing the Role of Imagery in Multimodal Machine Translation

Published at WMT 2024, this work designs imagery-sensitive contrastive metrics for multimodal machine translation and evaluates three English-to-French model families. The experiments hold a candidate translation fixed while varying matched and mismatched visual context, testing whether the image moves model uncertainty in the intended direction. These score-based diagnostics do not by themselves establish generated-translation quality or causal evidence use.

On the original CoMMuTE set, VGAMT scores 81% ImageCoMMuTE (IC): a fixed candidate translation has lower perplexity with its matching image than with the mismatched image. It scores 63% TextCoMMuTE (TC): the correct candidate has lower perplexity than the alternative under a fixed image. Relative to the ambiguous mixed-image baseline, the favorable translation-preference-flip rate (IPR) is 13%, and the unfavorable rate (INR) is 0% (Tables 1–2). The 63% TC score is not a preference-flip rate or generated-translation accuracy.