---
title: "Mitigating the Alignment Tax of RLHF — paper by Lin et al. (2024)"
description: "Building on this, they propose Heterogeneous Model Averaging (HMA), which tunes a separate… 115 references, cited by 54 papers here."
authors: ["Yong Lin", "Hangyu Lin", "Wei Xiong", "Shizhe Diao", "Jianmeng Liu", "Jipeng Zhang", "Rui Pan", "Haoxiang Wang", "Wenbin Hu", "Hanning Zhang", "Hanze Dong", "Renjie Pi", "Han Zhao", "Nan Jiang", "Heng Ji", "Yuan Yao", "Tong Zhang"]
year: 2024
venue: "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2024.emnlp-main.35/"
doi: "10.18653/v1/2024.emnlp-main.35"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lin-2024-mitigating-alignment-tax-rlhf/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lin-2024-mitigating-alignment-tax-rlhf/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Mitigating the Alignment Tax of RLHF

# Mitigating the Alignment Tax of RLHF

**Authors:** [Yong Lin](https://md.research.elamur.ai/free/en/authors/yong-lin/), [Hangyu Lin](https://md.research.elamur.ai/free/en/authors/hangyu-lin/), [Wei Xiong](https://md.research.elamur.ai/free/en/authors/wei-xiong/), [Shizhe Diao](https://md.research.elamur.ai/free/en/authors/shizhe-diao/), Jianmeng Liu, [Jipeng Zhang](https://md.research.elamur.ai/free/en/authors/jipeng-zhang/), [Rui Pan](https://md.research.elamur.ai/free/en/authors/rui-pan/), [Haoxiang Wang](https://md.research.elamur.ai/free/en/authors/haoxiang-wang/), Wenbin Hu, Hanning Zhang, [Hanze Dong](https://md.research.elamur.ai/free/en/authors/hanze-dong/), Renjie Pi, [Han Zhao](https://md.research.elamur.ai/free/en/authors/han-zhao/), Nan Jiang, [Heng Ji](https://md.research.elamur.ai/free/en/authors/heng-ji/), [Yuan Yao](https://md.research.elamur.ai/free/en/authors/yuan-yao/) and [Tong Zhang](https://md.research.elamur.ai/free/en/authors/tong-zhang/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2024](https://md.research.elamur.ai/free/en/venues/emnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.emnlp-main.35/) · [PDF](https://aclanthology.org/2024.emnlp-main.35.pdf) · [DOI 10.18653/v1/2024.emnlp-main.35](https://doi.org/10.18653/v1/2024.emnlp-main.35) · [OpenAlex](https://openalex.org/W4404782827)

**Citations:** 28 citations (OpenAlex, 2026-10-14) · 54 in our library, of the 141,840 papers whose bibliographies we read

## Summary
RLHF makes LLMs follow human preferences, but it also tends to erode skills learned in pre-training. This loss is called the alignment tax. The authors measured it by aligning OpenLLaMA-3B on the Anthropic helpfulness/harmlessness data with Rejection Sampling Fine-tuning (RSF) and DPO. They then tracked performance on commonsense QA, reading comprehension and translation benchmarks. As reward went up, benchmark scores fell. They compared several remedies: continual-learning regularizers, LoRA, reward penalties and experience replay. The best trade-off came from plain model averaging, which interpolates the weights of the model before and after RLHF. Their theoretical account is that averaging adds feature diversity in layers where the two tasks share features. Experiments support this, since averaging the lower transformer layers helps both objectives. Building on this, they propose Heterogeneous Model Averaging (HMA), which tunes a separate mixing ratio for each block of layers. HMA improves on uniform averaging across RLHF algorithms and also on Mistral-7B-based models. The result is a cheap post-hoc way to keep general abilities while still gaining alignment.

## Key points
- Forgetting was measured on ARC Easy/Challenge, Race, PIQA, SQuAD, DROP and WMT 2014 French→English. Experiments used OpenLLaMA-3B with RSF and DPO.
- Experience replay was tested with a pre-training subset up to four times the size of the RLHF data. It still lost to model averaging on two of three benchmark groups.
- HMA splits the transformer into K parts (K = 3 by default), each with its own averaging ratio. The mean ratio is held fixed, and the per-part ratios are tuned to maximize reward. Larger K (6, 9) tends to overfit slightly.
- An averaging ratio of α = 0.2 is recommended. It reliably reduced forgetting without hurting alignment.
- On AlpacaEval 2.0, judged by GPT4, HMA raised Zephyr-7B-β's win rate from 8.10% to 9.32%. It also improved reading F1 (37.47→38.93) and Fr-En BLEU (36.55→37.23). Zephyr-7B-Gemma also improved, and PairRM evaluations showed the same trends.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Illinois Urbana-Champaign](https://md.research.elamur.ai/free/en/universities/university-of-illinois-urbana-champaign/) — University, Urbana, United States · 108 works · 67,432 citations
- [Princeton University](https://md.research.elamur.ai/free/en/universities/princeton-university/) — University, Princeton, United States · 61 works · 36,471 citations
- [Hong Kong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/hong-kong-university-of-science-and-technology/) — University, Hong Kong, China · 58 works · 23,058 citations
- [Nvidia](https://md.research.elamur.ai/free/en/companies/nvidia/) — Company, Santa Clara, United States · 16 works · 4,820 citations
- [Nvidia (United Kingdom)](https://md.research.elamur.ai/free/en/companies/nvidia-united-kingdom/) — Company, Reading, United Kingdom · 13 works · 4,550 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao and Tong Zhang. 2024. Mitigating the Alignment Tax of RLHF. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lin2024mitigating,
  title = {{Mitigating the Alignment Tax of RLHF}},
  author = {Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao and Tong Zhang},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.35},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.35}
}
```

- [← previous CodeGemma: Open Code Models Based on Gemma](https://md.research.elamur.ai/free/en/papers/zhao-2024-codegemma-open-code-models-based/)
- [next → Chain-of-Dictionary Prompting Elicits Translation in Large Language Models](https://md.research.elamur.ai/free/en/papers/lu-2024-chain-dictionary-prompting-elicits-translation/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

115 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. 2020. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33:15920– 15930. [find](https://scholar.google.com/scholar?q=Pietro+Buzzega%2C+Matteo+Boschini%2C+Angelo+Porrello%2C+Davide+Abati%2C+and+Simone+Calderara%2E+2020%2E+Dark+experience+for+general+continual+learning%3A+a+strong%2C+simple+baseline%2E+Advances+in+neural+information+pr)
2. Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. 2021. New insights on reducing abrupt representation change in online continual learning. arXiv preprint arXiv:2104.05025. — [arXiv:2104.05025](https://arxiv.org/abs/2104.05025) [find](https://scholar.google.com/scholar?q=Lucas+Caccia%2C+Rahaf+Aljundi%2C+Nader+Asadi%2C+Tinne+Tuytelaars%2C+Joelle+Pineau%2C+and+Eugene+Belilovsky%2E+2021%2E+New+insights+on+reducing+abrupt+representation+change+in+online+continual+learning%2E+arXiv+prepri)
3. Lucas Caccia, Eugene Belilovsky, Massimo Caccia, and Joelle Pineau. 2020. Online learned continual compression with adaptive quantization modules. In International Conference on Machine Learning, pages 1240–1250. PMLR. [find](https://scholar.google.com/scholar?q=Lucas+Caccia%2C+Eugene+Belilovsky%2C+Massimo+Caccia%2C+and+Joelle+Pineau%2E+2020%2E+Online+learned+continual+compression+with+adaptive+quantization+modules%2E+In+International+Conference+on+Machine+Learning%2C+page)
4. Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. 2024. Evolutionary optimization of model merging recipes. arXiv preprint arXiv:2403.13187. — [arXiv:2403.13187](https://arxiv.org/abs/2403.13187) [find](https://scholar.google.com/scholar?q=Takuya+Akiba%2C+Makoto+Shing%2C+Yujin+Tang%2C+Qi+Sun%2C+and+David+Ha%2E+2024%2E+Evolutionary+optimization+of+model+merging+recipes%2E+arXiv+preprint+arXiv%3A2403%2E13187%2E+%E2%80%94+%5BarXiv%3A2403%2E13187%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403)
5. Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. 2018. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pages 139–154. [find](https://scholar.google.com/scholar?q=Rahaf+Aljundi%2C+Francesca+Babiloni%2C+Mohamed+Elhoseiny%2C+Marcus+Rohrbach%2C+and+Tinne+Tuytelaars%2E+2018%2E+Memory+aware+synapses%3A+Learning+what+%28not%29+to+forget%2E+In+Proceedings+of+the+European+conference+on+co)
6. Zeyuan Allen-Zhu and Yuanzhi Li. 2020. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816. — [arXiv:2012.09816](https://arxiv.org/abs/2012.09816) [find](https://scholar.google.com/scholar?q=Zeyuan+Allen%2DZhu+and+Yuanzhi+Li%2E+2020%2E+Towards+understanding+ensemble%2C+knowledge+distillation+and+self%2Ddistillation+in+deep+learning%2E+arXiv+preprint+arXiv%3A2012%2E09816%2E+%E2%80%94+%5BarXiv%3A2012%2E09816%5D%28https%3A%2F%2Farxi)
7. Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca Roelofs. 2021. The evolution of out-of-distribution robustness throughout fine-tuning. arXiv preprint arXiv:2106.15831. — [arXiv:2106.15831](https://arxiv.org/abs/2106.15831) [find](https://scholar.google.com/scholar?q=Anders+Andreassen%2C+Yasaman+Bahri%2C+Behnam+Neyshabur%2C+and+Rebecca+Roelofs%2E+2021%2E+The+evolution+of+out%2Dof%2Ddistribution+robustness+throughout+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2106%2E15831%2E+%E2%80%94+%5BarXiv%3A2106%2E15)
8. Anthropic. 2023. Introducing claude. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2023%2E+Introducing+claude%2E)
9. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861. [find](https://scholar.google.com/scholar?q=Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Andy+Jones%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Nova+DasSarma%2C+et+al%2E+2021%2E+A+general+language+assistant+as+a+laboratory+for+alignm)
10. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. 2023. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217. [find](https://scholar.google.com/scholar?q=Stephen+Casper%2C+Xander+Davies%2C+Claudia+Shi%2C+Thomas+Krendl+Gilbert%2C+J%C3%A9r%C3%A9my+Scheurer%2C+Javier+Rando%2C+Rachel+Freedman%2C+Tomasz+Korbak%2C+David+Lindner%2C+Pedro+Freire%2C+et+al%2E+2023%2E+Open+problems+and+fundamenta)
11. Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. 2021a. Co2l: Contrastive continual learning. In Proceedings of the IEEE/CVF International conference on computer vision, pages 9516–9525. [find](https://scholar.google.com/scholar?q=Hyuntak+Cha%2C+Jaeho+Lee%2C+and+Jinwoo+Shin%2E+2021a%2E+Co2l%3A+Contrastive+continual+learning%2E+In+Proceedings+of+the+IEEE%2FCVF+International+conference+on+computer+vision%2C+pages+9516%E2%80%939525%2E)
12. Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. 2021b. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405–22418. [find](https://scholar.google.com/scholar?q=Junbum+Cha%2C+Sanghyuk+Chun%2C+Kyungjae+Lee%2C+Han%2DCheol+Cho%2C+Seunghyun+Park%2C+Yunsung+Lee%2C+and+Sungrae+Park%2E+2021b%2E+Swad%3A+Domain+generalization+by+seeking+flat+minima%2E+Advances+in+Neural+Information+Process)
13. Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2018. Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812.00420. [find](https://scholar.google.com/scholar?q=Arslan+Chaudhry%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Marcus+Rohrbach%2C+and+Mohamed+Elhoseiny%2E+2018%2E+Efficient+lifelong+learning+with+a%2Dgem%2E+arXiv+preprint+arXiv%3A1812%2E00420%2E)
14. Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. 2023. [A general theoretical paradigm to understand learning from human preferences](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/). arXiv preprint arXiv:2310.12036.
15. Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, and Claire Cui. 2023. Lifelong language pretraining with distribution-specialized experts. In International Conference on Machine Learning, pages 5383–5395. PMLR. [find](https://scholar.google.com/scholar?q=Wuyang+Chen%2C+Yanqi+Zhou%2C+Nan+Du%2C+Yanping+Huang%2C+James+Laudon%2C+Zhifeng+Chen%2C+and+Claire+Cui%2E+2023%2E+Lifelong+language+pretraining+with+distribution%2Dspecialized+experts%2E+In+International+Conference+on+Ma)
16. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+2022%2E+Training+a+helpful+and+harmless+assistant+with+rein)
17. Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. 2019. On the weaknesses of reinforcement learning for neural machine translation. arXiv preprint arXiv:1907.01752. — [arXiv:1907.01752](https://arxiv.org/abs/1907.01752) [find](https://scholar.google.com/scholar?q=Leshem+Choshen%2C+Lior+Fox%2C+Zohar+Aizenbud%2C+and+Omri+Abend%2E+2019%2E+On+the+weaknesses+of+reinforcement+learning+for+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1907%2E01752%2E+%E2%80%94+%5BarXiv%3A1907%2E01752%5D%28https%3A)
18. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.
19. Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. 2014. [Findings of the 2014 workshop on statistical machine translation](https://md.research.elamur.ai/free/en/papers/bojar-2014-findings-2014-workshop-statistical-machine/). In Proceedings of the ninth workshop on statistical machine tr…
20. Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345. [find](https://scholar.google.com/scholar?q=Ralph+Allan+Bradley+and+Milton+E+Terry%2E+1952%2E+Rank+analysis+of+incomplete+block+designs%3A+I%2E+the+method+of+paired+comparisons%2E+Biometrika%2C+39%283%2F4%29%3A324%E2%80%93345%2E)
21. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
22. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240):1–113.
23. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+2017%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2C+30%2E)
24. Xu Chu, Yujie Jin, Wenwu Zhu, Yasha Wang, Xin Wang, Shanghang Zhang, and Hong Mei. 2022. Dna: Domain generalization with diversified neural averaging. In International Conference on Machine Learning, pages 4010–4034. PMLR. [find](https://scholar.google.com/scholar?q=Xu+Chu%2C+Yujie+Jin%2C+Wenwu+Zhu%2C+Yasha+Wang%2C+Xin+Wang%2C+Shanghang+Zhang%2C+and+Hong+Mei%2E+2022%2E+Dna%3A+Domain+generalization+with+diversified+neural+averaging%2E+In+International+Conference+on+Machine+Learning%2C+)
25. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. [find](https://scholar.google.com/scholar?q=Peter+Clark%2C+Isaac+Cowhey%2C+Oren+Etzioni%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+Carissa+Schoenick%2C+and+Oyvind+Tafjord%2E+2018%2E+Think+you+have+solved+question+answering%3F+try+arc%2C+the+ai2+reasoning+challenge%2E+arX)
26. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
27. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023a. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023a%2E+Mis)
28. Shizhe Diao, Rui Pan, Hanze Dong, Ka Shun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. 2023. Lmflow: An extensible toolkit for finetuning and inference of large foundation models. arXiv preprint arXiv:2306.12420. [find](https://scholar.google.com/scholar?q=Shizhe+Diao%2C+Rui+Pan%2C+Hanze+Dong%2C+Ka+Shun+Shum%2C+Jipeng+Zhang%2C+Wei+Xiong%2C+and+Tong+Zhang%2E+2023%2E+Lmflow%3A+An+extensible+toolkit+for+finetuning+and+inference+of+large+foundation+models%2E+arXiv+preprint+arX)
29. Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023. [Raft: Reward ranked finetuning for generative foundation model alignment](https://md.research.elamur.ai/free/en/papers/dong-2023-raft-reward-ranked-finetuning-generative/). arXiv preprint arXiv:2304.06767.
30. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. [Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/). arXiv preprint arXiv:1903.00161.
31. Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. 2020. Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprint arXiv:2005.12729. [find](https://scholar.google.com/scholar?q=Logan+Engstrom%2C+Andrew+Ilyas%2C+Shibani+Santurkar%2C+Dimitris+Tsipras%2C+Firdaus+Janoos%2C+Larry+Rudolph%2C+and+Aleksander+Madry%2E+2020%2E+Implementation+matters+in+deep+policy+gradients%3A+A+case+study+on+ppo+and+t)
32. Leo Gao, John Schulman, and Jacob Hilton. 2023. [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). In International Conference on Machine Learning, pages 10835–10866. PMLR.
33. Xinyang Geng and Hao Liu. 2023. Openllama: An open reproduction of llama. [find](https://scholar.google.com/scholar?q=Xinyang+Geng+and+Hao+Liu%2E+2023%2E+Openllama%3A+An+open+reproduction+of+llama%2E)
34. Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023b. [Llm-blender: Ensembling large language models with pairwise ranking and generative fusion](https://md.research.elamur.ai/free/en/papers/jiang-2023-llm-blender-ensembling-large-language/). arXiv preprint arXiv:2306.02561.
35. Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. 2021. Lifelong pretraining: Continually adapting language models to emerging corpora. arXiv preprint arXiv:2110.08534. [find](https://scholar.google.com/scholar?q=Xisen+Jin%2C+Dejiao+Zhang%2C+Henghui+Zhu%2C+Wei+Xiao%2C+Shang%2DWen+Li%2C+Xiaokai+Wei%2C+Andrew+Arnold%2C+and+Xiang+Ren%2E+2021%2E+Lifelong+pretraining%3A+Continually+adapting+language+models+to+emerging+corpora%2E+arXiv+pre)
36. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521… [find](https://scholar.google.com/scholar?q=James+Kirkpatrick%2C+Razvan+Pascanu%2C+Neil+Rabinowitz%2C+Joel+Veness%2C+Guillaume+Desjardins%2C+Andrei+A+Rusu%2C+Kieran+Milan%2C+John+Quan%2C+Tiago+Ramalho%2C+Agnieszka+Grabska%2DBarwinska%2C+et+al%2E+2017%2E+Overcoming+catas)
37. Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. 2022. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10054. — [arXiv:2202.10054](https://arxiv.org/abs/2202.10054) [find](https://scholar.google.com/scholar?q=Ananya+Kumar%2C+Aditi+Raghunathan%2C+Robbie+Jones%2C+Tengyu+Ma%2C+and+Percy+Liang%2E+2022%2E+Fine%2Dtuning+can+distort+pretrained+features+and+underperform+out%2Dof%2Ddistribution%2E+arXiv+preprint+arXiv%3A2202%2E10054%2E+%E2%80%94+%5Ba)
38. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. [Race: Large-scale reading comprehension dataset from examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). arXiv preprint arXiv:1704.04683.
39. Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. 2022a. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pages 1702–1712. PMLR. [find](https://scholar.google.com/scholar?q=Seunghyun+Lee%2C+Younggyo+Seo%2C+Kimin+Lee%2C+Pieter+Abbeel%2C+and+Jinwoo+Shin%2E+2022a%2E+Offline%2Dto%2Donline+reinforcement+learning+via+balanced+replay+and+pessimistic+q%2Densemble%2E+In+Conference+on+Robot+Learning%2C)
40. Zheng Gong, Kun Zhou, Wayne Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen. 2022. Continual pre-training of language models for math problem understanding with syntax-aware memory network. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa… [find](https://scholar.google.com/scholar?q=Zheng+Gong%2C+Kun+Zhou%2C+Wayne+Xin+Zhao%2C+Jing+Sha%2C+Shijin+Wang%2C+and+Ji%2DRong+Wen%2E+2022%2E+Continual+pre%2Dtraining+of+language+models+for+math+problem+understanding+with+syntax%2Daware+memory+network%2E+In+Procee)
41. Yoonho Lee, Annie S. Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. 2022b. Surgical fine-tuning improves adaptation to distribution shifts. ArXiv, abs/2210.11466. — [arXiv:2210.11466](https://arxiv.org/abs/2210.11466) [find](https://scholar.google.com/scholar?q=Yoonho+Lee%2C+Annie+S%2E+Chen%2C+Fahim+Tajwar%2C+Ananya+Kumar%2C+Huaxiu+Yao%2C+Percy+Liang%2C+and+Chelsea+Finn%2E+2022b%2E+Surgical+fine%2Dtuning+improves+adaptation+to+distribution+shifts%2E+ArXiv%2C+abs%2F2210%2E11466%2E+%E2%80%94+%5BarXi)
42. Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. 2022. Finetune like you pretrain: Improved finetuning of zero-shot vision models. arXiv preprint arXiv:2212.00638. — [arXiv:2212.00638](https://arxiv.org/abs/2212.00638) [find](https://scholar.google.com/scholar?q=Sachin+Goyal%2C+Ananya+Kumar%2C+Sankalp+Garg%2C+Zico+Kolter%2C+and+Aditi+Raghunathan%2E+2022%2E+Finetune+like+you+pretrain%3A+Improved+finetuning+of+zero%2Dshot+vision+models%2E+arXiv+preprint+arXiv%3A2212%2E00638%2E+%E2%80%94+%5BarXi)
43. Shengzhi Li, Rongyu Lin, and Shichao Pei. 2024. Multimodal preference alignment remedies regression of visual instruction tuning on language model. arXiv preprint arXiv:2402.10884. — [arXiv:2402.10884](https://arxiv.org/abs/2402.10884) [find](https://scholar.google.com/scholar?q=Shengzhi+Li%2C+Rongyu+Lin%2C+and+Shichao+Pei%2E+2024%2E+Multimodal+preference+alignment+remedies+regression+of+visual+instruction+tuning+on+language+model%2E+arXiv+preprint+arXiv%3A2402%2E10884%2E+%E2%80%94+%5BarXiv%3A2402%2E10884)
44. Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. 2023. [Reinforced self-training (rest) for language modeling](https://md.research.elamur.ai/free/en/papers/gulcehre-2023-reinforced-self-training-rest-language/). arXiv preprint arXiv:2308.08998.
45. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/](https://github.com/) tatsu-lab/alpaca_eval. [↗](https://github.com/)
46. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+2016%2E+Deep+residual+learning+for+image+recognition%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pages+770%E2%80%937)
47. Yong Lin, Hanze Dong, Hao Wang, and Tong Zhang. 2022a. Bayesian invariant risk minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16021–16030. [find](https://scholar.google.com/scholar?q=Yong+Lin%2C+Hanze+Dong%2C+Hao+Wang%2C+and+Tong+Zhang%2E+2022a%2E+Bayesian+invariant+risk+minimization%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pages+16021%E2%80%9316030%2E)
48. J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). ArXiv, abs/2106.09685.
49. Yong Lin, Lu Tan, Yifan Hao, Honam Wong, Hanze Dong, Weizhong Zhang, Yujiu Yang, and Tong Zhang. 2023. Spurious feature diversification improves out-of-distribution generalization. arXiv preprint arXiv:2309.17230. — [arXiv:2309.17230](https://arxiv.org/abs/2309.17230) [find](https://scholar.google.com/scholar?q=Yong+Lin%2C+Lu+Tan%2C+Yifan+Hao%2C+Honam+Wong%2C+Hanze+Dong%2C+Weizhong+Zhang%2C+Yujiu+Yang%2C+and+Tong+Zhang%2E+2023%2E+Spurious+feature+diversification+improves+out%2Dof%2Ddistribution+generalization%2E+arXiv+preprint+arXi)
50. Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. 2021. Continual learning for text classification with information disentanglement based regularization. arXiv preprint arXiv:2104.05489. [find](https://scholar.google.com/scholar?q=Yufan+Huang%2C+Yanzhe+Zhang%2C+Jiaao+Chen%2C+Xuezhi+Wang%2C+and+Diyi+Yang%2E+2021%2E+Continual+learning+for+text+classification+with+information+disentanglement+based+regularization%2E+arXiv+preprint+arXiv%3A2104%2E054)
51. Yong Lin, Shengyu Zhu, Lu Tan, and Peng Cui. 2022b. Zin: When and how to learn invariance without environment partition? Advances in Neural Information Processing Systems, 35:24529–24542. [find](https://scholar.google.com/scholar?q=Yong+Lin%2C+Shengyu+Zhu%2C+Lu+Tan%2C+and+Peng+Cui%2E+2022b%2E+Zin%3A+When+and+how+to+learn+invariance+without+environment+partition%3F+Advances+in+Neural+Information+Processing+Systems%2C+35%3A24529%E2%80%9324542%2E)
52. Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. 2023. [Statistical rejection sampling improves preference optimization](https://md.research.elamur.ai/free/en/papers/liu-2023-statistical-rejection-sampling-improves-preference/). arXiv preprint arXiv:2309.06657.
53. Zihan Liu, Genta Indra Winata, and Pascale Fung. 2021. Continual mixed-language pre-training for extremely low-resource neural machine translation. arXiv preprint arXiv:2105.03953. [find](https://scholar.google.com/scholar?q=Zihan+Liu%2C+Genta+Indra+Winata%2C+and+Pascale+Fung%2E+2021%2E+Continual+mixed%2Dlanguage+pre%2Dtraining+for+extremely+low%2Dresource+neural+machine+translation%2E+arXiv+preprint+arXiv%3A2105%2E03953%2E)
54. Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eunjoon Cho, and Zhiguang Wang. 2020. Continual learning in task-oriented dialogue systems. arXiv preprint arXiv:2012.15504. [find](https://scholar.google.com/scholar?q=Andrea+Madotto%2C+Zhaojiang+Lin%2C+Zhenpeng+Zhou%2C+Seungwhan+Moon%2C+Paul+Crook%2C+Bing+Liu%2C+Zhou+Yu%2C+Eunjoon+Cho%2C+and+Zhiguang+Wang%2E+2020%2E+Continual+learning+in+task%2Doriented+dialogue+systems%2E+arXiv+preprint+)
55. James L McClelland, Bruce L McNaughton, and Randall C O’Reilly. 1995. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419. [find](https://scholar.google.com/scholar?q=James+L+McClelland%2C+Bruce+L+McNaughton%2C+and+Randall+C+O%E2%80%99Reilly%2E+1995%2E+Why+there+are+complementary+learning+systems+in+the+hippocampus+and+neocortex%3A+insights+from+the+successes+and+failures+of+connect)
56. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. [Webgpt: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/). arXiv preprint arXiv:2112.09332.
57. Michael Noukhovitch, Samuel Lavoie, Florian Strub, and Aaron Courville. 2023. Language model alignment with elastic reset. arXiv preprint arXiv:2312.07551. — [arXiv:2312.07551](https://arxiv.org/abs/2312.07551) [find](https://scholar.google.com/scholar?q=Michael+Noukhovitch%2C+Samuel+Lavoie%2C+Florian+Strub%2C+and+Aaron+Courville%2E+2023%2E+Language+model+alignment+with+elastic+reset%2E+arXiv+preprint+arXiv%3A2312%2E07551%2E+%E2%80%94+%5BarXiv%3A2312%2E07551%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
58. Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2022. [Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization…](https://md.research.elamur.ai/free/en/papers/ramamurthy-2022-reinforcement-learning-natural-language-processing/)
59. Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. 2024. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Alexandre+Rame%2C+Guillaume+Couairon%2C+Corentin+Dancette%2C+Jean%2DBaptiste+Gaya%2C+Mustafa+Shukor%2C+Laure+Soulier%2C+and+Matthieu+Cord%2E+2024%2E+Rewarded+soups%3A+towards+pareto%2Doptimal+alignment+by+interpolating+wei)
60. Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. 2024. Warm: On the benefits of weight averaged reward models. arXiv preprint arXiv:2401.12187. — [arXiv:2401.12187](https://arxiv.org/abs/2401.12187) [find](https://scholar.google.com/scholar?q=Alexandre+Ram%C3%A9%2C+Nino+Vieillard%2C+L%C3%A9onard+Hussenot%2C+Robert+Dadashi%2C+Geoffrey+Cideron%2C+Olivier+Bachem%2C+and+Johan+Ferret%2E+2024%2E+Warm%3A+On+the+benefits+of+weight+averaged+reward+models%2E+arXiv+preprint+arXiv)
61. Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. 2023. Progressive prompts: Continual learning for language models. arXiv preprint arXiv:2301.12314. [find](https://scholar.google.com/scholar?q=Anastasia+Razdaibiedina%2C+Yuning+Mao%2C+Rui+Hou%2C+Madian+Khabsa%2C+Mike+Lewis%2C+and+Amjad+Almahairi%2E+2023%2E+Progressive+prompts%3A+Continual+learning+for+language+models%2E+arXiv+preprint+arXiv%3A2301%2E12314%2E)
62. Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010. [find](https://scholar.google.com/scholar?q=Sylvestre%2DAlvise+Rebuffi%2C+Alexander+Kolesnikov%2C+Georg+Sperl%2C+and+Christoph+H+Lampert%2E+2017%2E+icarl%3A+Incremental+classifier+and+representation+learning%2E+In+Proceedings+of+the+IEEE+conference+on+Computer)
63. Michael Noukhovitch, Samuel Lavoie, Florian Strub, and Aaron C Courville. 2024. Language model alignment with elastic reset. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Michael+Noukhovitch%2C+Samuel+Lavoie%2C+Florian+Strub%2C+and+Aaron+C+Courville%2E+2024%2E+Language+model+alignment+with+elastic+reset%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2E)
64. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. [Know what you don’t know: Unanswerable questions for squad](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). arXiv preprint arXiv:1806.03822.
65. ArXiv, Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
66. Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, Christoph H Lampert, and icarl. Incremental classifier and representation learning. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 5533–5542. [find](https://scholar.google.com/scholar?q=Sylvestre%2DAlvise+Rebuffi%2C+Alexander+Kolesnikov%2C+Georg+Sperl%2C+Christoph+H+Lampert%2C+and+icarl%2E+Incremental+classifier+and+representation+learning%2E+In+Conference+on+Computer+Vision+and+Pattern+Recognitio)
67. Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. 2018. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910. [find](https://scholar.google.com/scholar?q=Matthew+Riemer%2C+Ignacio+Cases%2C+Robert+Ajemian%2C+Miao+Liu%2C+Irina+Rish%2C+Yuhai+Tu%2C+and+Gerald+Tesauro%2E+2018%2E+Learning+to+learn+without+forgetting+by+maximizing+transfer+and+minimizing+interference%2E+arXiv+)
68. Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. 2023. Task-specific skill localization in fine-tuned language models. arXiv preprint arXiv:2302.06600. — [arXiv:2302.06600](https://arxiv.org/abs/2302.06600) [find](https://scholar.google.com/scholar?q=Abhishek+Panigrahi%2C+Nikunj+Saunshi%2C+Haoyu+Zhao%2C+and+Sanjeev+Arora%2E+2023%2E+Task%2Dspecific+skill+localization+in+fine%2Dtuned+language+models%2E+arXiv+preprint+arXiv%3A2302%2E06600%2E+%E2%80%94+%5BarXiv%3A2302%2E06600%5D%28https%3A%2F%2Fa)
69. Hippolyt Ritter, Aleksandar Botev, and David Barber. 2018. Online structured laplace approximations for overcoming catastrophic forgetting. Advances in Neural Information Processing Systems, 31. [find](https://scholar.google.com/scholar?q=Hippolyt+Ritter%2C+Aleksandar+Botev%2C+and+David+Barber%2E+2018%2E+Online+structured+laplace+approximations+for+overcoming+catastrophic+forgetting%2E+Advances+in+Neural+Information+Processing+Systems%2C+31%2E)
70. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
71. Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. [Multitask prompted training enables zero-shot task generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/). arXiv preprint arXiv:2110.08207.
72. Yujia Qin, Jiajie Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022. Elle: Efficient lifelong pre-training for emerging data. arXiv preprint arXiv:2203.06311. [find](https://scholar.google.com/scholar?q=Yujia+Qin%2C+Jiajie+Zhang%2C+Yankai+Lin%2C+Zhiyuan+Liu%2C+Peng+Li%2C+Maosong+Sun%2C+and+Jie+Zhou%2E+2022%2E+Elle%3A+Efficient+lifelong+pre%2Dtraining+for+emerging+data%2E+arXiv+preprint+arXiv%3A2203%2E06311%2E)
73. Sunny Sanyal, Atula Tejaswi Neerkaje, Jean Kaddour, Abhishek Kumar, et al. 2023. Early weight averaging meets high learning rates for llm pre-training. In Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ NeurIPS 2023). [find](https://scholar.google.com/scholar?q=Sunny+Sanyal%2C+Atula+Tejaswi+Neerkaje%2C+Jean+Kaddour%2C+Abhishek+Kumar%2C+et+al%2E+2023%2E+Early+weight+averaging+meets+high+learning+rates+for+llm+pre%2Dtraining%2E+In+Workshop+on+Advancing+Neural+Network+Training)
74. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In International Conference on Machine Learning, pages 8748–8763. PM…
75. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). arXiv preprint arXiv:2305.18290.
76. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+2022%2E+Bloom%3A+A+176b%2Dparameter)
77. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+2017%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2E)
78. Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. 2018. Progress & compress: A scalable framework for continual learning. In International conference on machine learning, pages 4528–4537. PMLR. [find](https://scholar.google.com/scholar?q=Jonathan+Schwarz%2C+Wojciech+Czarnecki%2C+Jelena+Luketina%2C+Agnieszka+Grabska%2DBarwinska%2C+Yee+Whye+Teh%2C+Razvan+Pascanu%2C+and+Raia+Hadsell%2E+2018%2E+Progress+%26+compress%3A+A+scalable+framework+for+continual+learni)
79. Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. 2017. Continual learning with deep generative replay. Advances in neural information processing systems, 30. [find](https://scholar.google.com/scholar?q=Hanul+Shin%2C+Jung+Kwon+Lee%2C+Jaehong+Kim%2C+and+Jiwon+Kim%2E+2017%2E+Continual+learning+with+deep+generative+replay%2E+Advances+in+neural+information+processing+systems%2C+30%2E)
80. Karen Simonyan and Andrew Zisserman. 2014. [Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/). arXiv preprint arXiv:1409.1556.
81. Ziang Song, Tianle Cai, Jason D Lee, and Weijie J Su. 2023. Reward collapse in aligning large language models. arXiv preprint arXiv:2305.17608. — [arXiv:2305.17608](https://arxiv.org/abs/2305.17608) [find](https://scholar.google.com/scholar?q=Ziang+Song%2C+Tianle+Cai%2C+Jason+D+Lee%2C+and+Weijie+J+Su%2E+2023%2E+Reward+collapse+in+aligning+large+language+models%2E+arXiv+preprint+arXiv%3A2305%2E17608%2E+%E2%80%94+%5BarXiv%3A2305%2E17608%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E17608%29)
82. Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. 2019. Lamol: Language modeling for lifelong language learning. arXiv preprint arXiv:1909.03329. [find](https://scholar.google.com/scholar?q=Fan%2DKeng+Sun%2C+Cheng%2DHao+Ho%2C+and+Hung%2DYi+Lee%2E+2019%2E+Lamol%3A+Language+modeling+for+lifelong+language+learning%2E+arXiv+preprint+arXiv%3A1909%2E03329%2E)
83. Xiaoyu Tan, LIN Yong, Shengyu Zhu, Chao Qu, Xihe Qiu, Xu Yinghui, Peng Cui, and Yuan Qi. 2023. Provably invariant learning without domain information. [find](https://scholar.google.com/scholar?q=Xiaoyu+Tan%2C+LIN+Yong%2C+Shengyu+Zhu%2C+Chao+Qu%2C+Xihe+Qiu%2C+Xu+Yinghui%2C+Peng+Cui%2C+and+Yuan+Qi%2E+2023%2E+Provably+invariant+learning+without+domain+information%2E)
84. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. [Galactica: A large language model for science](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/). arXiv preprint arXiv:2211.09085.
85. Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. 2022b. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec… [find](https://scholar.google.com/scholar?q=Mitchell+Wortsman%2C+Gabriel+Ilharco%2C+Jong+Wook+Kim%2C+Mike+Li%2C+Simon+Kornblith%2C+Rebecca+Roelofs%2C+Raphael+Gontijo+Lopes%2C+Hannaneh+Hajishirzi%2C+Ali+Farhadi%2C+Hongseok+Namkoong%2C+et+al%2E+2022b%2E+Robust+fine%2Dtuni)
86. Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. 2021. Robust fine-tuning of zero-shot models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7949–7961. [find](https://scholar.google.com/scholar?q=Mitchell+Wortsman%2C+Gabriel+Ilharco%2C+Mike+Li%2C+Jong+Wook+Kim%2C+Hannaneh+Hajishirzi%2C+Ali+Farhadi%2C+Hongseok+Namkoong%2C+and+Ludwig+Schmidt%2E+2021%2E+Robust+fine%2Dtuning+of+zero%2Dshot+models%2E+2022+IEEE%2FCVF+Confere)
87. Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021a. [Recursively summarizing books with human feedback](https://md.research.elamur.ai/free/en/papers/wu-2021-recursively-summarizing-books-human-feedback/). arXiv preprint arXiv:2109.10862.
88. Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi, and Gholamreza Haffari. 2021b. Pretrained language model in continual learning: A comparative study. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tongtong+Wu%2C+Massimo+Caccia%2C+Zhuang+Li%2C+Yuan%2DFang+Li%2C+Guilin+Qi%2C+and+Gholamreza+Haffari%2E+2021b%2E+Pretrained+language+model+in+continual+learning%3A+A+comparative+study%2E+In+International+Conference+on+Lea)
89. Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi, and Gholamreza Haffari. 2022. Pretrained language model in continual learning: A comparative study. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tongtong+Wu%2C+Massimo+Caccia%2C+Zhuang+Li%2C+Yuan%2DFang+Li%2C+Guilin+Qi%2C+and+Gholamreza+Haffari%2E+2022%2E+Pretrained+language+model+in+continual+learning%3A+A+comparative+study%2E+In+International+Conference+on+Lear)
90. Wei Xiong, Hanze Dong, Chen Ye, Han Zhong, Nan Jiang, and Tong Zhang. 2023. Gibbs sampling from human feedback: A provable kl- constrained framework for rlhf. [find](https://scholar.google.com/scholar?q=Wei+Xiong%2C+Hanze+Dong%2C+Chen+Ye%2C+Han+Zhong%2C+Nan+Jiang%2C+and+Tong+Zhang%2E+2023%2E+Gibbs+sampling+from+human+feedback%3A+A+provable+kl%2D+constrained+framework+for+rlhf%2E)
91. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
92. LI Xuhong, Yves Grandvalet, and Franck Davoine. 2018. Explicit inductive bias for transfer learning with convolutional networks. In International Conference on Machine Learning, pages 2825–2834. PMLR. [find](https://scholar.google.com/scholar?q=LI+Xuhong%2C+Yves+Grandvalet%2C+and+Franck+Davoine%2E+2018%2E+Explicit+inductive+bias+for+transfer+learning+with+convolutional+networks%2E+In+International+Conference+on+Machine+Learning%2C+pages+2825%E2%80%932834%2E+PMLR%2E)
93. Haoqin Tu, Bingchen Zhao, Chen Wei, and Cihang Xie. 2023. Sight beyond text: Multi-modal training enhances llms in truthfulness and ethics. arXiv preprint arXiv:2309.07120. — [arXiv:2309.07120](https://arxiv.org/abs/2309.07120) [find](https://scholar.google.com/scholar?q=Haoqin+Tu%2C+Bingchen+Zhao%2C+Chen+Wei%2C+and+Cihang+Xie%2E+2023%2E+Sight+beyond+text%3A+Multi%2Dmodal+training+enhances+llms+in+truthfulness+and+ethics%2E+arXiv+preprint+arXiv%3A2309%2E07120%2E+%E2%80%94+%5BarXiv%3A2309%2E07120%5D%28https%3A)
94. Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. 2023. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint arXiv:2310.02575. [find](https://scholar.google.com/scholar?q=Enneng+Yang%2C+Zhenyi+Wang%2C+Li+Shen%2C+Shiwei+Liu%2C+Guibing+Guo%2C+Xingwei+Wang%2C+and+Dacheng+Tao%2E+2023%2E+Adamerging%3A+Adaptive+model+merging+for+multi%2Dtask+learning%2E+arXiv+preprint+arXiv%3A2310%2E02575%2E)
95. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. 2023. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944. [find](https://scholar.google.com/scholar?q=Lewis+Tunstall%2C+Edward+Beeching%2C+Nathan+Lambert%2C+Nazneen+Rajani%2C+Kashif+Rasul%2C+Younes+Belkada%2C+Shengyi+Huang%2C+Leandro+von+Werra%2C+Cl%C3%A9mentine+Fourrier%2C+Nathan+Habib%2C+et+al%2E+2023%2E+Zephyr%3A+Direct+distilla)
96. Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. 2015. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579. — [arXiv:1506.06579](https://arxiv.org/abs/1506.06579) [find](https://scholar.google.com/scholar?q=Jason+Yosinski%2C+Jeff+Clune%2C+Anh+Nguyen%2C+Thomas+Fuchs%2C+and+Hod+Lipson%2E+2015%2E+Understanding+neural+networks+through+deep+visualization%2E+arXiv+preprint+arXiv%3A1506%2E06579%2E+%E2%80%94+%5BarXiv%3A1506%2E06579%5D%28https%3A%2F%2Farxi)
97. Jeffrey S Vitter. 1985. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 11(1):37–57. [find](https://scholar.google.com/scholar?q=Jeffrey+S+Vitter%2E+1985%2E+Random+sampling+with+a+reservoir%2E+ACM+Transactions+on+Mathematical+Software+%28TOMS%29%2C+11%281%29%3A37%E2%80%9357%2E)
98. Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. 2023a. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. arXiv preprint arXiv:2309.16240. — [arXiv:2309.16240](https://arxiv.org/abs/2309.16240) [find](https://scholar.google.com/scholar?q=Chaoqi+Wang%2C+Yibo+Jiang%2C+Chenghao+Yang%2C+Han+Liu%2C+and+Yuxin+Chen%2E+2023a%2E+Beyond+reverse+kl%3A+Generalizing+direct+preference+optimization+with+diverse+divergence+constraints%2E+arXiv+preprint+arXiv%3A2309%2E16)
99. Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. 2023b. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487. [find](https://scholar.google.com/scholar?q=Liyuan+Wang%2C+Xingxing+Zhang%2C+Hang+Su%2C+and+Jun+Zhu%2E+2023b%2E+A+comprehensive+survey+of+continual+learning%3A+Theory%2C+method+and+application%2E+arXiv+preprint+arXiv%3A2302%2E00487%2E)
100. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2022%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+pre)
101. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022a. [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://md.research.elamur.ai/free/en/papers/wortsman-2022-model-soups-averaging-weights-multiple/). In I…
102. Pengfei Yu and Heng Ji. 2023. Self information update for large language models through mitigating exposure bias. In arxiv. [find](https://scholar.google.com/scholar?q=Pengfei+Yu+and+Heng+Ji%2E+2023%2E+Self+information+update+for+large+language+models+through+mitigating+exposure+bias%2E+In+arxiv%2E)
103. Pengfei Yu, Heng Ji, and Premkumar Natarajan. 2021. Lifelong event detection with knowledge transfer. In Proc. [find](https://scholar.google.com/scholar?q=Pengfei+Yu%2C+Heng+Ji%2C+and+Premkumar+Natarajan%2E+2021%2E+Lifelong+event+detection+with+knowledge+transfer%2E+In+Proc%2E)
104. The 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP2021). [find](https://scholar.google.com/scholar?q=The+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP2021%29%2E)
105. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. [Rrhf: Rank responses to align language models with human feedback without tears](https://md.research.elamur.ai/free/en/papers/yuan-2023-rrhf-rank-responses-align-language/). arXiv preprint arXiv:2304.05302.
106. Matthew D Zeiler and Rob Fergus. 2014. [Visualizing and understanding convolutional networks](https://md.research.elamur.ai/free/en/papers/zeiler-2013-visualizing-understanding-convolutional-networks/). In Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 818–833. Springer.
107. Michael Zhang and Christopher Ré. 2022. Contrastive adapters for foundation model group robustness. arXiv preprint arXiv:2207.07180. — [arXiv:2207.07180](https://arxiv.org/abs/2207.07180) [find](https://scholar.google.com/scholar?q=Michael+Zhang+and+Christopher+R%C3%A9%2E+2022%2E+Contrastive+adapters+for+foundation+model+group+robustness%2E+arXiv+preprint+arXiv%3A2207%2E07180%2E+%E2%80%94+%5BarXiv%3A2207%2E07180%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2207%2E07180%29)
108. Tong Zhang. 2023. Mathematical Analysis of Machine Learning Algorithms. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Tong+Zhang%2E+2023%2E+Mathematical+Analysis+of+Machine+Learning+Algorithms%2E+Cambridge+University+Press%2E)
109. Yanzhe Zhang, Xuezhi Wang, and Diyi Yang. 2022. Continual sequence generation with adaptive compositional modules. arXiv preprint arXiv:2203.10652. [find](https://scholar.google.com/scholar?q=Yanzhe+Zhang%2C+Xuezhi+Wang%2C+and+Diyi+Yang%2E+2022%2E+Continual+sequence+generation+with+adaptive+compositional+modules%2E+arXiv+preprint+arXiv%3A2203%2E10652%2E)
110. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. 2023. [Slic-hf: Sequence likelihood calibration with human feedback](https://md.research.elamur.ai/free/en/papers/zhao-2023-slic-hf-sequence-likelihood-calibration/). arXiv preprint arXiv:2305.10425.
111. Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. 2023. [Secrets of rlhf in large language models part i: Ppo](https://md.research.elamur.ai/free/en/papers/zheng-2023-secrets-rlhf-large-language-models/). arXiv preprint arXiv:2307.04964.
112. Xiao Zhou, Yong Lin, Renjie Pi, Weizhong Zhang, Renzhe Xu, Peng Cui, and Tong Zhang. 2022a. Model agnostic sample reweighting for out-of-distribution learning. In International Conference on Machine Learning, pages 27203– 27221. PMLR. [find](https://scholar.google.com/scholar?q=Xiao+Zhou%2C+Yong+Lin%2C+Renjie+Pi%2C+Weizhong+Zhang%2C+Renzhe+Xu%2C+Peng+Cui%2C+and+Tong+Zhang%2E+2022a%2E+Model+agnostic+sample+reweighting+for+out%2Dof%2Ddistribution+learning%2E+In+International+Conference+on+Machine+L)
113. Xiao Zhou, Yong Lin, Weizhong Zhang, and Tong Zhang. 2022b. Sparse invariant risk minimization. In International Conference on Machine Learning, pages 27222– 27244. PMLR. [find](https://scholar.google.com/scholar?q=Xiao+Zhou%2C+Yong+Lin%2C+Weizhong+Zhang%2C+and+Tong+Zhang%2E+2022b%2E+Sparse+invariant+risk+minimization%2E+In+International+Conference+on+Machine+Learning%2C+pages+27222%E2%80%93+27244%2E+PMLR%2E)
114. Banghua Zhu, Jiantao Jiao, and Michael I Jordan. 2023. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. arXiv preprint arXiv:2301.11270. — [arXiv:2301.11270](https://arxiv.org/abs/2301.11270) [find](https://scholar.google.com/scholar?q=Banghua+Zhu%2C+Jiantao+Jiao%2C+and+Michael+I+Jordan%2E+2023%2E+Principled+reinforcement+learning+with+human+feedback+from+pairwise+or+k%2Dwise+comparisons%2E+arXiv+preprint+arXiv%3A2301%2E11270%2E+%E2%80%94+%5BarXiv%3A2301%2E11270%5D%28)
115. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. [find](https://scholar.google.com/scholar?q=Daniel+M+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+2019%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+ar)

*Updated 24 October 2026*