Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

Researchers from Apple have published findings showing that both human annotators and large language models struggle to consistently assess the quality of machine translation outputs. While automated models are being explored as a cost-effective replacement for human evaluators, this study indicates that neither method currently provides a reliable standard for measuring translation accuracy across diverse languages and domains. This underscores the technical difficulty of establishing objective benchmarks for linguistic quality in automated translation systems.