--- title: "HERMeS: Human Evaluation & Ranking of — Vanhorn 2026" description: "Comparing many MT engines and LLMs through human judgment is laborious." authors: ["Rex Vanhorn"] year: 2026 venue: "Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 2)" source: "https://aclanthology.org/2026.eamt-2.8/" licence: "CC BY-ND 4.0 (EAMT)" licence_url: "https://creativecommons.org/licenses/by-nd/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/vanhorn-2026-hermes-human-evaluation-ranking-multiple/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/vanhorn-2026-hermes-human-evaluation-ranking-multiple/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/) / HERMeS: Human Evaluation & Ranking of MultiplE Systems # HERMeS: Human Evaluation & Ranking of MultiplE Systems **Authors:** Rex Vanhorn · **Year:** 2026 · **Venue:** Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 2) · **Licence:** [CC BY-ND 4.0 (EAMT)](https://creativecommons.org/licenses/by-nd/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.eamt-2.8/) · [PDF](https://aclanthology.org/2026.eamt-2.8.pdf) ## Summary Comparing many MT engines and LLMs through human judgment is laborious. Annotators tire, their ratings drift, and existing tools are not always built for side-by-side scoring of a dozen or more anonymized outputs. HERMeS is a lightweight, open-source web tool for this scenario. Each source sentence appears on a card with its candidate translations. Evaluators work in two steps. First, they sort every translation into one of four coarse quality buckets. Second, they give it a numeric direct-assessment score within that bucket's range. The tool can automatically group and reorder items as judgments are made, and evaluators can add notes per sentence. Inputs are randomized and anonymized, and hash checks expose tampering and make it safe to merge annotations collected separately. The app keeps no state and requires no login, so evaluators save their progress as downloadable spreadsheet checkpoints. The author positions HERMeS as a complement to Appraise, not a replacement, and it does not support detailed error categorization. Early qualitative feedback suggests the bucket-first approach may lower perceived effort and improve consistency. No quantitative results are reported. ## Key points - **Two-stage workflow:** coarse ranking into four buckets (Best, Good, OK, Poor; names configurable), then direct assessment on a configurable scale. In the author's study, each bucket covers 7 points (e.g., 1–7 for Poor, 8–14 for OK). - **Data integrity:** sources are randomized, anonymized and hash-verified before evaluation, and the research team de-anonymizes and de-randomizes the returned results. - **Infrastructure:** built with Streamlit, hosted on Streamlit Community Cloud, free to use, with no login and no server-side storage. Work is saved and resumed via spreadsheet checkpoints, and the code is on GitHub. - **Pilot study:** 13 systems × 100 sentences = 1,300 translations, with only preliminary qualitative feedback reported. - **Positioning:** meant to complement Appraise (Federmann, 2012) for large multi-system comparison, not for fine-grained error analysis. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{vanhorn2026hermes, title = {{HERMeS: Human Evaluation \& Ranking of MultiplE Systems}}, author = {Vanhorn, Rex}, booktitle = {Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 2)}, year = {2026}, url = {https://aclanthology.org/2026.eamt-2.8/} } ``` - [← previous HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model](https://md.research.elamur.ai/free/en/papers/langstream-2026-kayzer-hebatron-hebrew-specialized-open-weight/) - [next → High-Order Question Generation in a Multilingual Educational Context](https://md.research.elamur.ai/free/en/papers/ucar-2026-high-order-question-generation-multilingual/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Castilho, S. (2021, April). Towards document-level human MT evaluation: On the issues of annotator agreement, effort and misevaluation. In Proceedings of the workshop on human evaluation of NLP systems (HumEval) (pp. 34– 45). [find](https://scholar.google.com/scholar?q=Castilho%2C+S%2E+%282021%2C+April%29%2E+Towards+document%2Dlevel+human+MT+evaluation%3A+On+the+issues+of+annotator+agreement%2C+effort+and+misevaluation%2E+In+Proceedings+of+the+workshop+on+human+evaluation+of+NLP+system) 2. Federmann, C. (2012). Appraise: an Open-Source Toolkit for Manual Evaluation of MT Output. Prague Bull. Math. Linguistics, 98, 25–36. ——————————————————————— 2 [https://streamlit.io](https://streamlit.io) 3 https://mt-eval.streamlit.app/?type=admin 4 https://github.com/RexVH/St-MT-Evaluation-app [↗](https://streamlit.io)