TY - GEN
T1 - Tagged Span Annotation for Detecting Translation Errors in Reasoning LLMs
AU - Yeom, Taemin
AU - Ryu, Yonghyun
AU - Choi, Yoonjung
AU - Bak, Jin Yeong
N1 - Publisher Copyright:
© 2025 Association for Computational Linguistics.
PY - 2025
Y1 - 2025
N2 - We present the submission of the AIP team to the WMT 2025 Unified MT Evaluation Shared Task, focusing on the span-level error detection subtask. Our system emphasizes response-format design to better harness the capabilities of OpenAI's o3, the state-of-the-art reasoning LLM. To this end, we introduce Tagged Span Annotation (TSA), an annotation scheme designed to more accurately extract span-level information from the LLM. On our refined version of WMT24 ESA dataset, our reference-free method achieves an F1 score of approximately 27 for character-level label prediction, outperforming the reference-based XCOMET-XXL at approximately 17.
AB - We present the submission of the AIP team to the WMT 2025 Unified MT Evaluation Shared Task, focusing on the span-level error detection subtask. Our system emphasizes response-format design to better harness the capabilities of OpenAI's o3, the state-of-the-art reasoning LLM. To this end, we introduce Tagged Span Annotation (TSA), an annotation scheme designed to more accurately extract span-level information from the LLM. On our refined version of WMT24 ESA dataset, our reference-free method achieves an F1 score of approximately 27 for character-level label prediction, outperforming the reference-based XCOMET-XXL at approximately 17.
UR - https://www.scopus.com/pages/publications/105028911903
U2 - 10.18653/v1/2025.wmt-1.62
DO - 10.18653/v1/2025.wmt-1.62
M3 - Conference contribution
AN - SCOPUS:105028911903
T3 - Conference on Machine Translation - Proceedings
SP - 878
EP - 886
BT - WMT 2025 - 10th Conference on Machine Translation, Proceedings of the Conference
A2 - Haddow, Barry
A2 - Kocmi, Tom
A2 - Koehn, Philipp
A2 - Monz, Christof
PB - Association for Computational Linguistics
T2 - 10th Conference on Machine Translation, WMT 2025
Y2 - 8 November 2025 through 9 November 2025
ER -