Running professional human evaluations of machine translation requires detailed methodology and software tools to streamline the process. The most detailed approach to evaluation today is to label errors and assign weights to critical issues using a variation of DQF/MQM ontology. In this article, we outline Custom.MT’s journey to selecting and implementing a tool for human evaluation of MT. It will be useful for all teams that face a similar challenge.

Measuring machine translation (MT) quality by the number of errors per 1000 words and assigning weights to critical errors focuses engineers on important issues. Neural MT can produce fluent, human-like output but it often hides catastrophic errors.

Here is one example from our work: a machine translation engine produced a smooth translation where it changed “50 mV” to “50 V”. Only one symbol was wrong, but that spelled a 1000x increase in voltage. Automated metrics BLEU, WER, chrF, and TER consider this translation “almost perfect”. In reality, should someone use the translation and apply a 1000x increase in voltage, it could potentially destroy electronic equipment.

To address this type of issue, error classification metrics were developed by Dr. Arle Lommel in 2013 as MQM (Multidimensional Quality Metrics) and continued in 2015 by TAUS as DQF (Dynamic Quality Framework). The metric has been adopted by teams at Google Translate and Microsoft Translator with some simplification and different weight formulas.

Here is our take on the error typology.

Starting with Spreadsheets for DQF/MQM

We began with Excel & Google Docs. At the start of our error classification tasks at Custom.MT, we asked linguists to review machine translation in spreadsheets. Our initial setup was simple, as we used dropdown lists to allow for the selection of error types and severities based on the MQM framework.

An additional page provided instructions to linguists, and we aimed to keep it as concise as possible. Within weeks of deploying this approach, we hit several problems.

1. 1. Working in Excel was boring, and linguists got burned out, especially when comparing outputs from several engines for 5-8 hours a day; 

1. 1. Distributing files to linguists was cumbersome for the PMs, especially with 5-10 language evaluations using 3 linguists per language; 

1. 1. There were risks of introducing errors to the source; 

1. 1. We didn’t track time in Excel, and had no way to measure cognitive effort; 

1. 1. Attaching images for visual references in Excel was cumbersome.

Moreover, linguists without DQF/MQM training tended to report errors in different ways, what was Stylistic for one person turned out Accuracy for another evaluator. As a company specializing in machine translation, we wanted to professionalize this work and started working on a tool to streamline error classification. 

MT Quality Evaluation in Phrase TMS

First, we turned to CAT tools such as Phrase (Memsource) and Smartcat, because linguists often work there and they are very familiar with the UI. Phrase, for example, has an LQA feature that is built for this purpose, a feature that has been adopted by Translators without Borders. 

However, this tool did not stick internally, because couldn’t customize it freely. We wanted to be able to quickly tweak the error typology, track time, highlight parts of the sentences, and provide instructions.