Visual Context and AI Translation Accuracy in Chinese Museum Object Descriptions: An Exploratory Paired Comparison of Text Only and Text Plus Image Conditions
Abstract
A photograph can make an object feature easier to inspect without resolving how that feature should be translated. This study examines that problem in 12 Chinese museum descriptions through a paired comparison of 24 retained English outputs from the National Museum of China. An adapted Multidimensional Quality Metrics scheme separates confirmed accuracy and terminology errors from unresolved judgements, recorded as reservations. The text-only condition contained 11 confirmed errors and 7 reservations; the text-plus-image condition contained 9 errors and 13 reservations. Accuracy errors numbered 9 in each condition, and terminology errors 2 and 0. Five samples had fewer confirmed errors with an image, five had the same count, and two had more. Case analysis shows contrasting changes: some outputs retain quantities and local forms more fully, while others omit details or substitute an animal category. Technical identification still depends on terminology and curatorial evidence. Checking was assisted by artificial intelligence (AI), with condition labels visible and no second independent evaluator. Because the comparison uses single outputs under uncontrolled generation conditions and leaves some judgements unresolved, it cannot establish an overall quality ranking or attribute the differences to image input. Its contribution is an account of specific translation decisions that require review.
References
- Caglayan, O., Madhyastha, P., Specia, L., & Barrault, L. (2019). Probing the need for visual context in multimodal machine translation. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)(pp. 4159–4170). https://doi.org/10.18653/v1/N19-1422
- Wu, Z., Kong, L., Bi, W., Li, X., & Kao, B. (2021). Good for misconceived reasons: An empirical revisiting on the need for visual context in multimodal machine translation. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)(pp. 6153–6166). https://doi.org/10.18653/v1/2021.acl-long.480
- Liu, D., Kong, F., Sun, X., Patil, D., Vajpayee, A., Liu, Z., Bhat, V., & Sadoughi, N. (2025). Detect, disambiguate, and translate: On-demand visual reasoning for multimodal machine translation with large vision-language models. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)(pp. 1559–1570). https://doi.org/10.18653/v1/2025.naacl-long.74
- Martinec, R., & Salway, A. (2005). A system for image-text relations in new (and old) media.Visual Communication, 4, 337–371. https://doi.org/10.1177/1470357205055928
- Adami, E. (2023). A social semiotic multimodal approach to translation. InThe Routledge Handbook of Translation Theory and Concepts. Routledge, London.
- Valdeón, R. A. (2024). The translation of multimodal texts: Challenges and theoretical approaches.Perspectives: Studies in Translation Theory and Practice, 32, 1-13. https://doi.org/10.1080/0907676X.2024.2290928
- Liao, M.-H. (2023). Translation as a practice of resemiotization: A case study of the Opium War Museum.Translation Studies, 16, 48-63. https://doi.org/10.1080/14781700.2022.2103024
- Villaespesa, E., & Murphy, O. (2021). This is not an apple! Benefits and challenges of applying computer vision to museum collections.Museum Management and Curatorship, 36, 362-383. https://doi.org/10.1080/09647775.2021.1873827
- Calixto, I., & Liu, Q. (2019). An error analysis for image-based multi-modal neural machine translation.Machine Translation, 33, 155-177. https://doi.org/10.1007/s10590-019-09226-9
- Sulubacak, U., Caglayan, O., Grönroos, S.-A., Rouhe, A., Elliott, D., Specia, L., & Tiedemann, J. (2020). Multimodal machine translation through visuals and speech.Machine Translation, 34, 97-147. https://doi.org/10.1007/s10590-020-09250-0
- Yang, C., & Cui, F. (2023). Multimodal mediation in translation and communication of Chinese museum culture in the era of artificial intelligence.Corpus-Based Studies Across Humanities, 1, 51-77. https://doi.org/10.1515/csh-2023-0020
- Lommel, A. R., Uszkoreit, H., & Burchardt, A. (2014). Multidimensional quality metrics (MQM): A framework for declaring and describing translation quality metrics.Tradumàtica, 12, 455-463. https://doi.org/10.5565/rev/tradumatica.77