---
title: "Fine-Grained Reward Optimization for Machine — Ramos 2026"
description: "Most RL training for neural MT scores a translation with one number per sentence."
authors: ["Miguel Moura Ramos", "Tomás Almeida", "Daniel Vareta", "Filipe Azevedo", "Sweta Agrawal", "Patrick Fernandes", "André F. T. Martins"]
year: 2026
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2026.tacl-1.33/"
doi: "10.1162/tacl.a.646"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/ramos-2026-fine-grained-reward-optimization-machine/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/ramos-2026-fine-grained-reward-optimization-machine/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/page/2/) / Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings

# Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings

**Authors:** Miguel Moura Ramos, Tomás Almeida, Daniel Vareta, Filipe Azevedo, [Sweta Agrawal](https://md.research.elamur.ai/free/authors/sweta-agrawal/), [Patrick Fernandes](https://md.research.elamur.ai/free/authors/patrick-fernandes/), [André F. T. Martins](https://md.research.elamur.ai/free/authors/andre-f-t-martins/) · **Year:** 2026 · **Venue:** Transactions of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.tacl-1.33/) · [PDF](https://aclanthology.org/2026.tacl-1.33.pdf) · [DOI 10.1162/tacl.a.646](https://doi.org/10.1162/tacl.a.646) · [OpenAlex](https://openalex.org/W4404400385)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14) · 7 in our library, of the 140673 papers whose bibliographies we read

## Summary
Most RL training for neural MT scores a translation with one number per sentence. This sparse signal does not say where errors occur, so learning is slow and unstable, especially for long inputs. The authors instead derive a reward for each token from XCOMET, a quality estimation model that marks error spans and labels them minor, major or critical. They propose a custom severity map that converts these labels into numeric token rewards, replacing standard MQM weights. The token rewards feed into REINFORCE and PPO.

They test the approach with NLLB (encoder-decoder) and TOWER (LLM-based) systems on EN→DE, EN→FR, DE→EN and FR→EN, using both small and large datasets. Token-level RL (tRL) beats sentence-level RL (sRL) on most automatic metrics. It also wins in human Direct Assessment, with larger gains for longer sentences, and mean rewards rise more steadily during training. The work shows that span-level quality metrics can serve as dense, human-aligned training signals for MT.

## Key points
- **Reward model and mapping:** XCOMET supplies the token-level rewards. All words inside an error span receive the same severity value, set by the new severity map.
- **PPO vs. REINFORCE:** Both algorithms work, but PPO gives the best overall scores. On IWSLT2017 EN→FR, for example, tPPO with XCOMET-MQM reaches 87.17 and 92.17 on two of the reported metrics.
- **Cost of the reward model:** As a tRL reward, XCOMET yields COMET22 of 74.66 versus 72.22 for BLEU (TOWER, WMT24 EN→DE). The trade-off is much higher latency and lower throughput.
- **Human evaluation on WMT24 EN→DE:** Two professional translators rated outputs, with 50 samples per length bin. tRL scored an average DA of 82.6, which is 2.3 points above sRL. Inter-annotator agreement was Pearson r = 0.59 and Spearman ρ = 0.57.
- **Hybrid variant:** A model using sRL for short inputs and tRL for long ones beats sRL but still falls behind pure tRL (Appendix B).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

Topics: [Semantic role labeling](https://md.research.elamur.ai/free/topics/semantic-role-labeling/) · [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{ramos2026fine,
  title = {{Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings}},
  author = {Ramos, Miguel Moura and Almeida, Tomás and Vareta, Daniel and Azevedo, Filipe and Agrawal, Sweta and Fernandes, Patrick and Martins, André F. T.},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.646},
  url = {https://doi.org/10.1162/tacl.a.646}
}
```

- [← previous Findings of the Fifth Shared Task on Multilingual Coreference Resolution: Expanding Datasets for Long-Range Entities](https://md.research.elamur.ai/free/papers/novak-2026-findings-fifth-shared-task-multilingual/)
- [next → First Shared Task on UMR Parsing](https://md.research.elamur.ai/free/papers/tepanek-2026-first-shared-task-umr-parsing/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/page/2/)

## References

2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sweta Agrawal, José G. C. De Souza, Ricardo Rei, António Farinhas, Gonçalo Faria, Patrick Fernandes, Nuno M. Guerreiro, and Andre Martins. 2024. Modeling user preferences with automatic metrics: Creating a high-quality preference dataset for machine translation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14503–14519, Miami, Florida, USA. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.emnlp-main.803 [↗](https://aclanthology.org/2024.emnlp-main.803/)
2. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. 2024. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12248–12267, Bangkok, Thailand. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024 .acl-long.662 [↗](https://doi.org/10.18653/v1/2024)