You Cannot Feed Two Birds with One Score: the Accuracy-Naturalness Tradeoff in Translation

Add to your list(s) Download to your calendar using vCal

• Gergely Flamich (Imperial College London)
• Thursday 07 August 2025, 10:00-11:00
• Seminar Room 2, Newton Institute.

If you have a question about this talk, please contact nobody.

RCL - Representing, calibrating & leveraging prediction uncertainty from statistics to machine learning

The goal of translation, be it by human or by machine, is, given some textin a source language, to produce text in a target language that simultaneously 1) preserves the meaning of the source text and 2) achieves natural expression in the target language. However, researchers in the machine translation community usually assess translations using a single score in-tended to capture semantic accuracy and the naturalness of the output simultaneously. In this talk, I build on Michaeli and Blau’s the seminal work on distortion-perception theory to propose a general notion of accuracy and naturalness, and show that such single-score summaries do not and cannot give the complete picture of a translation system’s true performance. I also demonstrate the accuracy-naturalness tradeoff by evaluating the performance of the state-of-the-art translation systems on a popular benchmark. These findings help explain well-known empirical phenomena, such as the observation that optimizing translation systems for a specific accuracy metric initially improves the system’s naturalness, while “overfitting” the system to the metric can significantly degrade its naturalness. Finally, I propose a new statistical distance in the spirit of integral probability metrics to measure naturalness in practice and discuss some of its properties.

This talk is part of the Isaac Newton Institute Seminar Series series.

This talk is included in these lists:

• All CMS events
• Featured lists
• INI info aggregator
• Isaac Newton Institute Seminar Series
• School of Physical Sciences
• Seminar Room 2, Newton Institute
• bld31

Note that ex-directory lists are not shown.