---
title: "Reinforcement Learning with Semantic Rewards Enables"
description: "When LLMs are adapted to a poorly supported language, they often lose ground on skills they already had. 0.7164 BLEU. 26 references."
authors: ["Zeli Su", "Ziyin Zhang", "Zhou Liu", "Xuexian Song", "Zhankai Xu", "Longfei Zheng", "Xiaolu Zhang", "Rong Fu", "Guixian Xu", "Wentao Zhang"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.880/"
doi: "10.18653/v1/2026.findings-acl.880"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/su-2026-reinforcement-learning-semantic-rewards-enables/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/su-2026-reinforcement-learning-semantic-rewards-enables/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/47/) / Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax

# Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax

**Authors:** [Zeli Su](https://md.research.elamur.ai/free/en/authors/zeli-su/), [Ziyin Zhang](https://md.research.elamur.ai/free/en/authors/ziyin-zhang/), [Zhou Liu](https://md.research.elamur.ai/free/en/authors/zhou-liu/), [Xuexian Song](https://md.research.elamur.ai/free/en/authors/xuexian-song/), Zhankai Xu, Longfei Zheng, [Xiaolu Zhang](https://md.research.elamur.ai/free/en/authors/xiaolu-zhang/), Rong Fu, [Guixian Xu](https://md.research.elamur.ai/free/en/authors/guixian-xu/) and [Wentao Zhang](https://md.research.elamur.ai/free/en/authors/wentao-zhang/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.880/) · [PDF](https://aclanthology.org/2026.findings-acl.880.pdf) · [DOI 10.18653/v1/2026.findings-acl.880](https://doi.org/10.18653/v1/2026.findings-acl.880) · [OpenAlex](https://openalex.org/W7166829037)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
When LLMs are adapted to a poorly supported language, they often lose ground on skills they already had. This is known as the alignment tax. The authors blame supervised fine-tuning (SFT): its token-level likelihood objective forces the model to copy small, skewed datasets, which overwrites pretrained knowledge. They instead frame language expansion as alignment in semantic space. A brief cold-start SFT first teaches the model to write in the target script. Training then continues with GRPO, which scores outputs by embedding similarity to the reference plus a language-consistency check. Any wording that keeps the meaning is rewarded, and policy updates stay constrained. Evaluated on two tasks, translating between Tibetan and Chinese and producing headlines in Tibetan, the method gains Tibetan ability with almost no loss of Chinese competence, and it outscores SFT on the CMRC benchmark. Its outputs share fewer n-grams with references, yet LLM judges prefer them, and they transfer better in few-shot settings. For teams working on low-resource languages, this suggests that careful reward design may be a safer route than additional SFT.

## Key points
- **Two-stage pipeline:** cold-start SFT on 5k instances for two epochs, then a single epoch of GRPO on the remaining training data, using embedding-similarity and language-consistency (LC) rewards.
- **Machine translation:** on the dominant-language CMRC benchmark, the RL model beats the SFT baseline by +5.15 points, indicating much less forgetting. An appendix analysis of OOD token NLL and KL divergence to the base model supports this.
- **Headline generation:** despite lower n-gram overlap, the RL model achieves a +16.1% higher win rate than SFT under LLM-based judges.
- **Reward ablation (semantic similarity, MT):**
  - Embedding + LC: 0.7164
  - BLEU + LC: 0.6375
  - BLEU + Embedding + LC: 0.6175
  - BLEU + Embedding: 0.2312

  Removing LC leads to mixed Tibetan–Chinese output, and adding BLEU brings back SFT-like rigidity.
- **Length reward:** an extra length-constraint reward gave no quality gain and cut CMRC by about 2 points.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations
- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 212 works · 19,544 citations
- [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) — Government body, Beijing, China · 246 works · 24,951 citations
- [Institute of Automation](https://md.research.elamur.ai/free/en/labs/institute-of-automation/) — Research institute, Beijing, China · 87 works · 14,883 citations
- [University of Macau](https://md.research.elamur.ai/free/en/universities/university-of-macau/) — University, Macao, China · 43 works · 2,368 citations
- [Ant Group](https://md.research.elamur.ai/free/en/companies/ant-group/) — Company, Hangzhou, China · 51 works · 1,211 citations
- [Minzu University of China](https://md.research.elamur.ai/free/en/universities/minzu-university-of-china/) — University, Beijing, China · 7 works · 2 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🌱 Low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/) · 287 works

## Cite

Cite as: Zeli Su, Ziyin Zhang, Zhou Liu, Xuexian Song, Zhankai Xu, Longfei Zheng, Xiaolu Zhang, Rong Fu, Guixian Xu and Wentao Zhang. 2026. Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{su2026reinforcement,
  title = {{Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax}},
  author = {Zeli Su, Ziyin Zhang, Zhou Liu, Xuexian Song, Zhankai Xu, Longfei Zheng, Xiaolu Zhang, Rong Fu, Guixian Xu and Wentao Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.880},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.880}
}
```

- [← previous Reinforcement Learning for Latent-Space Thinking in LLMs](https://md.research.elamur.ai/free/en/papers/ozeren-2026-reinforcement-learning-latent-space-thinking/)
- [next → Reinforcement Learning–Guided Adaptive Tuning for Out-of-Distribution Harmful Text Detection](https://md.research.elamur.ai/free/en/papers/xiang-2026-reinforcement-learning-guided-adaptive-tuning/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/47/)

## References

26 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Thales Sales Almeida, Rodrigo Nogueira, and Hélio Pedrini. 2025. Curió-edu 7b: Examining data selection impacts in llm continued pretraining. Preprint, arXiv:2512.12770. — [arXiv:2512.12770](https://arxiv.org/abs/2512.12770) [find](https://scholar.google.com/scholar?q=Thales+Sales+Almeida%2C+Rodrigo+Nogueira%2C+and+H%C3%A9lio+Pedrini%2E+2025%2E+Curi%C3%B3%2Dedu+7b%3A+Examining+data+selection+impacts+in+llm+continued+pretraining%2E+Preprint%2C+arXiv%3A2512%2E12770%2E+%E2%80%94+%5BarXiv%3A2512%2E12770%5D%28https%3A%2F%2Fa)
2. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham Abdullah Alyahya, Sultan AlRashed, Faisal Abdulrahman Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Saad Ami… — [ALLaM: Large Language Models for Arabic and English](https://md.research.elamur.ai/free/en/papers/bari-2024-allam-large-language-models-arabic/)
3. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Paul+F%2E+Christiano%2C+Jan+Leike%2C+Tom+B%2E+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+2017%2E+Deep+reinforcement+learning+from+human+preferences%2E+In+Advances+in+Neural+Information+Processing+Systems)
4. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, and 1 others. 2025. [Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/). Preprint, arXiv:2507.06261.
5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. [Unsupervised cross-lingual representation learning at scale](https://md.research.elamur.ai/free/en/papers/conneau-2020-unsupervised-cross-lingual-representation-learning/). In Proceedings of the 58th Annual Meeting of the Association fo…
6. Zoltan Csaki, Bo Li, Jonathan Lingjie Li, Qiantong Xu, Pian Pawakapan, Leon Zhang, Yun Du, Hengyu Zhao, Changran Hu, and Urmish Thakker. 2024. SambaLingo: Teaching large language models new languages. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pages 1–2… [find](https://scholar.google.com/scholar?q=Zoltan+Csaki%2C+Bo+Li%2C+Jonathan+Lingjie+Li%2C+Qiantong+Xu%2C+Pian+Pawakapan%2C+Leon+Zhang%2C+Yun+Du%2C+Hengyu+Zhao%2C+Changran+Hu%2C+and+Urmish+Thakker%2E+2024%2E+SambaLingo%3A+Teaching+large+language+models+new+languages%2E)
7. Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. 2019. [A span-extraction dataset for Chinese machine reading comprehension](https://md.research.elamur.ai/free/en/papers/cui-2018-span-extraction-dataset-chinese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conf…
8. DeepSeek-AI and 1 others. 2025. [Deepseek-v3.2: Pushing the frontier of open large language models](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-v3-2-pushing/). Preprint, arXiv:2512.02556.
9. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
10. Danni Liu and Jan Niehues. 2025. Conditions for catastrophic forgetting in multilingual translation. In Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025), pages 347–359. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Danni+Liu+and+Jan+Niehues%2E+2025%2E+Conditions+for+catastrophic+forgetting+in+multilingual+translation%2E+In+Proceedings+of+the+5th+Workshop+on+Multilingual+Representation+Learning+%28MRL+2025%29%2C+pages+347%E2%80%9335)
11. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+7th+International+Conference+on+Learning+Representations%2C+ICLR+2019%2C+New+Orleans%2C+LA%2C+USA%2C+May+6%2D9%2C+2019%2E+OpenReview%2En)
12. OpenAI. 2025. Introducing gpt-4.1 in the api. OpenAI Blog. Accessed: 2025-12-25. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025%2E+Introducing+gpt%2D4%2E1+in+the+api%2E+OpenAI+Blog%2E+Accessed%3A+2025%2D12%2D25%2E)
13. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). In Advances in Neural Information Processing Systems.
14. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). arXiv preprint arXiv:2305.18290.
15. Fabian David Schmidt. 2025. Robust and Scalable Cross-Lingual Transfer. Ph.D. thesis, Bayerische Julius-Maximilians-Universitaet Wuerzburg (Germany). [find](https://scholar.google.com/scholar?q=Fabian+David+Schmidt%2E+2025%2E+Robust+and+Scalable+Cross%2DLingual+Transfer%2E+Ph%2ED%2E+thesis%2C+Bayerische+Julius%2DMaximilians%2DUniversitaet+Wuerzburg+%28Germany%29%2E)
16. John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. 2015. Trust region policy optimization. arXiv preprint arXiv:1502.05477. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Sergey+Levine%2C+Philipp+Moritz%2C+Michael+I%2E+Jordan%2C+and+Pieter+Abbeel%2E+2015%2E+Trust+region+policy+optimization%2E+arXiv+preprint+arXiv%3A1502%2E05477%2E)
17. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+2017%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2E)
18. Zhihong Shao and 1 others. 2024. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300.
19. Kimi Team, Yifan Bai, and 1 others. 2025. [Kimi k2: Open agentic intelligence](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k2-open-agentic/). Preprint, arXiv:2507.20534.
20. Jiahong Wu, He Zheng, Bo Zhao, Yixin Li, Baoming Yan, Rui Liang, Wenjia Wang, Shipei Zhou, Guosen Lin, Yanwei Fu, and 1 others. 2019. Large-scale datasets for going deeper in image understanding. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pages 1480–1485. IEEE. [find](https://scholar.google.com/scholar?q=Jiahong+Wu%2C+He+Zheng%2C+Bo+Zhao%2C+Yixin+Li%2C+Baoming+Yan%2C+Rui+Liang%2C+Wenjia+Wang%2C+Shipei+Zhou%2C+Guosen+Lin%2C+Yanwei+Fu%2C+and+1+others%2E+2019%2E+Large%2Dscale+datasets+for+going+deeper+in+image+understanding%2E+In+2)
21. Guixian Xu, Zeli Su, Ziyin Zhang, Jianing Liu, Xu Han, Ting Zhang, and Yushuang Dong. 2025. CMHG: A dataset and benchmark for headline generation of minority languages in China. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 12350–12357, Suzhou, Chin… [find](https://scholar.google.com/scholar?q=Guixian+Xu%2C+Zeli+Su%2C+Ziyin+Zhang%2C+Jianing+Liu%2C+Xu+Han%2C+Ting+Zhang%2C+and+Yushuang+Dong%2E+2025%2E+CMHG%3A+A+dataset+and+benchmark+for+headline+generation+of+minority+languages+in+China%2E+In+Proceedings+of+the+)
22. Atsuki Yamaguchi, Terufumi Morishita, Aline Villavicencio, and Nikolaos Aletras. 2025. [Mitigating catastrophic forgetting in target language adaptation of llms via source-shielded updates](https://md.research.elamur.ai/free/en/papers/yamaguchi-2026-mitigating-catastrophic-forgetting-target-language/). Preprint, arXiv:2512.04844.
23. An Yang and 1 others. 2025a. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). Preprint, arXiv:2505.09388.
24. Menglin Yang, Jialin Chen, Yifei Zhang, Jiahong Liu, Jiasheng Zhang, Qiyao Ma, Harshit Verma, Qianru Zhang, Min Zhou, Irwin King, and Rex Ying. 2025b. Low-rank adaptation for foundation models: A comprehensive review. CoRR, abs/2501.00365. — [arXiv:2501.00365](https://arxiv.org/abs/2501.00365) [find](https://scholar.google.com/scholar?q=Menglin+Yang%2C+Jialin+Chen%2C+Yifei+Zhang%2C+Jiahong+Liu%2C+Jiasheng+Zhang%2C+Qiyao+Ma%2C+Harshit+Verma%2C+Qianru+Zhang%2C+Min+Zhou%2C+Irwin+King%2C+and+Rex+Ying%2E+2025b%2E+Low%2Drank+adaptation+for+foundation+models%3A+A+comp)
25. Ziqing Yang, Zihang Xu, Yiming Cui, Baoxin Wang, Min Lin, Dayong Wu, and Zhigang Chen. 2022. CINO: A Chinese minority pre-trained language model. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3937– 3949, Gyeongju, Republic of Korea. International Committee o… [find](https://scholar.google.com/scholar?q=Ziqing+Yang%2C+Zihang+Xu%2C+Yiming+Cui%2C+Baoxin+Wang%2C+Min+Lin%2C+Dayong+Wu%2C+and+Zhigang+Chen%2E+2022%2E+CINO%3A+A+Chinese+minority+pre%2Dtrained+language+model%2E+In+Proceedings+of+the+29th+International+Conference+on)
26. Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. [Llama beyond english: An empirical study on language capability transfer](https://md.research.elamur.ai/free/en/papers/zhao-2024-llama-english-empirical-study-language/). CoRR, abs/2401.01055.

*Updated 2 November 2026*