Assessing the Role of Imagery in Multimodal Machine Translation

N. Kashani Motlagh, J. Davis, J. Gwinnup, G. Erdmann, T. Anderson

Published at WMT 2024, this work designs imagery-sensitive contrastive metrics for multimodal machine translation and evaluates three English-to-French model families. The experiments hold a candidate translation fixed while varying matched and mismatched visual context, testing whether the image moves model uncertainty in the intended direction. These score-based diagnostics do not by themselves establish generated-translation quality or causal evidence use.

@inproceedings{kashanimotlagh2024assessing,
  title     = {Assessing the Role of Imagery in Multimodal Machine Translation},
  author    = {Kashani Motlagh, N. and Davis, J. and Gwinnup, J. and Erdmann, G. and Anderson, T.},
  year      = {2024},
  booktitle = {Proceedings of the Ninth Conference on Machine Translation (WMT)},
  publisher = {Association for Computational Linguistics},
  pages     = {1428--1439},
  doi       = {10.18653/v1/2024.wmt-1.130},
  url       = {https://aclanthology.org/2024.wmt-1.130/}
}