Communications in Humanities Research ISSN: 2753-7072
Home> Volumes> Volume CHR Vol.61> Article
Research Article

Evaluating ChatGPT's Chinese-English Translation Quality of Tender Documents: A Research of MTPE with MQM Scoring Models

Lin ZhouBeijing Normal University*

* Corresponding author: [email protected]

Abstract

Since Google introduced the Transformer model into natural language processing (NLP) in 2017, AI-aided translation has rapidly advanced. At the same time, translation is evolving from a solitary endeavor into a cooperative activity between human translators and machine translation systems, epitomized by the emergency of platforms with the Machine Translation Post Editing (MTPE) function. The advent of new translation modes also leads to increased research evaluating the effectiveness and quality of machine translation, for example, studies on the translation quality under the Multidimensional Quality Metrics (MQM) error typology framework. Involving AI-based translators and MTPE in their translation enables human translators to prepare the engineering documents efficiently. However, researchers notice that it is difficult for most machine translators to figure out the semantic and cultural differences in the source language and generate coherent structural translation in the target language. This research opens up ChatGPT’s application in tender document translation under the MQM framework, hoping to cast light on assessment on ChatGPT's translation quality, identification of ChatGPT's errors in translating such documents and suggestions on human translators' performance throughout MTPE.

Keywords: Machine Translation; Translation Error Typology; MTPE; MQM
Published: January 24, 2025
DOI: 10.54254/2753-7064/2025.20639
Volume: CHR Vol.61
pp. 53-62
Download PDF

References

  1. Soydaner, Derya. "Attention mechanism in neural networks: where it comes and where it goes." Neural Computing and Applications 34.16 (2022): 13371–85.
  2. Vieira, Lucas Nunes, Minako O'Hagan, and Carol O'Sullivan. "Understanding the societal impacts of machine translation: a critical review of the literature on medical and legal use cases." Information, Communication & Society 24.11 (2021): 1515–32.
  3. Lommel, Arle, et al. "The Multi-Range Theory of Translation Quality Measurement: MQM scoring models and Statistical Quality Control." arXiv preprint arXiv:2405.16969, 2024.
  4. Yang, Yanxia, et al. "Performance and perception: machine translation post-editing in Chinese-English news translation by novice translators." Humanities and Social Sciences Communications 10.1 (2023): 1–8.
  5. Gao, Yuan, Ruili Wang, and Feng Hou. "How to design translation prompts for ChatGPT: An empirical study." arXiv preprint arXiv:2304.02182, 2023.
  6. Gao, Ruiyao, et al. "Machine translation of Chinese classical poetry: a comparison among ChatGPT, Google Translate, and DeepL Translator." Humanities and Social Sciences Communications 11.1 (2024): 1–10.
  7. Liu, Huey-Ing, and Wei-Lin Chen. "X-transformer: a machine translation model enhanced by the self-attention mechanism." Applied Sciences 12.9 (2022): 4502.
  8. Ren, Jinghui. "Mining Lexical Co-occurrence Features Based Research On Construction Engineering English." IOP Conference Series: Earth and Environmental Science. Vol. 267. No. 6. IOP Publishing, 2019.
  9. Gou, Miao. "Syntactic Features and Translation Strategies of English for Construction: A Review of English for Construction." Building Structure, vol. 51, no. 14, 2021, pp. Appendix 4–5 (in Chinese).
  10. Wang, Haifeng, et al. "Progress in machine translation." Engineering 18 (2022): 143–53.
  11. Bahdanau, Dzmitry. "Neural machine translation by jointly learning to align and translate." arXiv preprint arXiv:1409.0473, 2014.
  12. Wang, Shanshan, and Xiaohui Wang. "Cross-Language Translation and Comparative Study of English Literature Using Machine Translation Algorithms." Journal of Electrical Systems 20.6s (2024): 2221–29.
  13. Feng, Zhiwei. "Term Translation in Era of Large Language Models." China Terminology, vol. 26, no. 3, 2024, p. 93 (in Chinese).
  14. Mossop, Brian. "Lynne Bowker and Jairo Buitrago Ciro. Machine Translation and Global Research: Towards Improved Machine Translation Literacy in the Scholarly Community. Bingley, Emerald Publishing, 2019, 111 p." TTR: traduction, terminologie, rédaction 32.2 (2019): 250–55.
  15. Yang, Yanxia, and Xiangling Wang. "Predicting student translators' performance in machine translation post-editing: interplay of self-regulation, critical thinking, and motivation." Interactive Learning Environments 31.1 (2023): 340–54.
  16. Pang, Yaxin, and Xiaoyan Wang. "A Study on the Translation Quality of ChatGPT in the Context of Large Language Model." Modern English, vol. 22, 2023 (in Chinese).
  17. Al-Khalifa, Hend, Khaloud Al-Khalefah, and Hesham Haroon. "Error Analysis of Pretrained Language Models (PLMs) in English-to-Arabic Machine Translation." Human-Centric Intelligent Systems, 2024: 1–14.
  18. Popović, Maja. "Error classification and analysis for machine translation quality assessment." Translation Quality Assessment: From Principles to Practice, 2018: 129–58.
  19. Tsuji, Kayo. "Identifying MT Errors for Higher-Quality Second Language Text." International Journal of Translation, Interpretation, and Applied Linguistics (IJTIAL) 6.1 (2024): 1–12.