Machine Translation Evaluation

MT Evaluation Portal

Find the best machine translation engine for your content. Three LLMs from different vendors work as an LLM jury and score every segment. Your reviewers check only the close calls.

BOOK A DEMO

A Machine Translation Evaluation Tool Built on Your Content

The MT Evaluation Portal by Custom.MT helps you find the best machine translation engine for your content. Upload your files, and an LLM jury ranks NMT and LLM engines side by side with a score out of 100 and a pass rate before and after human review.

LLM-as-a-Jury

Three LLM judges from three vendors, picked from a curated model pool. The platform computes the verdict from their votes.

Code Checks

Tags, placeholders, symbols, numbers, and units are verified automatically.

Error Spans

Every finding quotes the exact words in the translation it points to.

Targeted Review

Reviewers check only split verdicts on critical categories.

Custom.MT Eval Platform screenshot | Quality Score Matrix — Quality scores by category for every engine in one view.

How LLM-as-a-Jury Evaluation Works

The portal uses LLM-as-a-jury evaluation: three LLMs from different vendors act as judges, and the platform combines their votes into one verdict. Judges always come from outside the engine under test.

Source + translationOne segment from your file, per engine.

Code checksTags, placeholders, symbols, numbers, units.

LLM juryMeaning, terminology, style, grammar.

Judge 1Judge 2Judge 3

Majority vote: 2 of 3

PASSTwo or more judges find no major error

FAILTwo or more judges find a major or critical error

HUMAN REVIEWThe judges split on a critical category

Voting rules

A check counts when at least two of three judges report it.
A segment fails when two judges rate an issue major or critical, even if they name different checks on the same span.
When the judges split on a critical category, such as accuracy or terminology, a reviewer confirms or overturns the verdict.
Each finding must quote an error span that occurs in the translation.

Custom.MT Eval Platform screenshot | AI Jury — Example panel: three judges, majority vote 2 of 3. A model that translates in a run cannot judge that run.

The Machine Translation Evaluation Workflow

From file upload to engine ranking in five steps.

1
Upload your file
Spreadsheet, TMX, or XLIFF.
2
Choose the engines
Each translation column is an engine. Compare translations you already have, add a new one for the same source, or request one from Custom.MT Console through a template.
3
Add your rules
Attach your glossary and style guide, set project rules, and rank your quality priorities.
4
Run the evaluation
Code checks and the LLM jury score every segment.
5
Read the score
Engines ranked by a score out of 100, with the pass rate before and after human review.

Custom.MT Eval Platform screenshot | New evaluation Engines — Pick the engines to compare: your existing translations, an LLM, or Custom.MT.

Custom.MT Eval Platform screenshot | New Evaluation Checks — Set the checks for the run: code checks and the LLM jury.

Custom.MT Eval Platform screenshot | Result panel — Engines ranked side by side with a score out of 100.

Custom.MT Eval Platform screenshot | Engine Difference — Compare engine output side by side and see where translations differ.

Tune the Evaluation to Your Quality Standards

Your brand rules go straight into the judge setup. The jury evaluates each translation against your terminology and voice.

Custom.MT Eval Platform screenshot | Scoring — Rank quality priorities and set points for each error severity.

Glossaries and style guides. Terminology, tone, register, and style rules feed into every judge.
Project rules. Set tone, register, and form of address, and add risk rules, for example "no advice that sounds like a medical diagnosis."
Quality priorities. Rank categories such as accuracy, terminology, and fluency. The top category carries the most weight in the score.
Critical categories. Choose the error types that matter most, for example accuracy and terminology. When the judges split on one of them, a reviewer checks the segment.
Judge selection. Build the panel from a model pool curated by Custom.MT. The team adds new models on request.
Judge prompts. Adjust the judge prompts to your quality criteria.

Who Uses the MT Evaluation Portal

Built for the teams that choose, check, and pay for machine translation.

Localization managers

Run engine comparisons on real content and pick the best machine translation engine for each project, based on a clear score.

Reviewers and linguists

Confirm or overturn the jury on split verdicts in critical categories only.

Vendor and procurement managers

Compare engines and vendors with a clear, shareable report.

What teams use it for

MT engine benchmarking

Compare engines on your own content and choose the best fit for each project and language.

NMT vs. LLM comparison

Test whether LLM translation outperforms your current NMT engine before you switch.

Quality monitoring

Track machine translation quality over time across your production content.

Why Localization Teams Use an MT Evaluation Tool

The best machine translation engine depends on the language, the content, and the month. The portal turns each engine decision into a fast, repeatable test on your own data.

The right engine for every language

Pick the strongest engine for each language pair and content type: UI, marketing, legal, or support.

Ready for every new model

New LLM and NMT versions arrive every few months. Test each release on your content before you switch.

Confident prompt and glossary updates

Changed a prompt, glossary, or style guide? Run the same benchmark again and see the effect in one score.

Quality tracked over time

Repeat benchmarks and follow each engine's score in the Trends view.

Proof that customization pays off

Show IT and management, on your own data, how a customized engine with your glossary and prompts compares with a generic one.

Save expert time and budget

The LLM jury checks every segment, so linguists review only split verdicts. You see the cost of each run before it starts.

How Often the Judges Agree

Measured on pilot runs with an enterprise client. Independent judges do not always agree, which is why every verdict comes from a majority vote.

66%

of segments got the same verdict, PASS or FAIL, from all three judges

77.3%

average agreement between any two judges: in about 3 of 4 segments, two judges reach the same verdict

judges on every panel, each from a different vendor

100

point platform score, with the pass rate before and after human review

Formats and Integrations

The portal runs as a standalone web app next to your TMS and CAT tool.

File formats: spreadsheet, TMX, XLIFF.
REST API: integration set up by the Custom.MT team.
Custom.MT Console: request new translations through a Console template inside the same run.
Your existing translations: upload output from several engines in one file, one column per engine, and compare them directly, with no need to translate again.

Data and Security

Every run keeps a full audit trail.

Sources, translations, and judge answers are stored with the run for later review.
Hosted in the EU (AWS, Frankfurt).
Dedicated hosting in your preferred country or region is available on request.
Your data stays available until you delete it.

Frequently Asked Questions

What is the MT Evaluation Portal?

A tool that helps you find the best machine translation engine for your content. Upload a file and choose the engines. An LLM jury of three judges scores each engine out of 100 and ranks them side by side.

Why use an MT evaluation tool instead of choosing an engine once?

The best engine changes with the language pair, the content type, and each new model release. Prompt and glossary updates shift quality too. The portal makes each comparison fast and repeatable, with a report you can share.

How does LLM-as-a-jury evaluation work?

Three LLMs from different vendors act as judges and review each source-translation pair. A check counts when two of three report it. A segment fails when two judges rate the issue major or critical.

Which models act as judges?

Any model from a pool curated by Custom.MT, with each judge from a different vendor. Best results come from judges that differ from, and outperform, the engines under evaluation. The Custom.MT team adds new models on request.

When does a person review a segment?

Only on split verdicts in critical categories. When the judges split on a critical category, such as accuracy or terminology, a reviewer confirms or overturns the verdict.

Which file formats and integrations are supported?

Spreadsheet, TMX, and XLIFF uploads, REST API integration set up by the Custom.MT team, and translation requests through Custom.MT Console templates. The portal works as a standalone web app.

Can I apply my own glossary and style guide?

Yes. Glossaries and style guides feed terminology, tone, register, and style rules into the judge setup. You also rank your quality priorities, such as accuracy, terminology, and fluency.

How is my data stored?

Sources, translations, and judge answers are stored with the run. Data is hosted in the EU (AWS, Frankfurt), and dedicated hosting in your preferred country or region is available on request. Your data stays available until you delete it.

Which content and languages does it support?

Any content type and any language. Each evaluation is tuned to your project through glossaries, style guides, and project rules.

How do I get trial access?

Book a demo. After a short call with the Custom.MT team, you get trial access to the portal.

See LLM-as-a-Jury Evaluation on Your Own Content

Book a short demo with the Custom.MT team. After the call, you get trial access to the MT Evaluation Portal.