---
title: "Unintended Impacts of LLM Alignment on Global Representation"
description: "Before chat LLMs are released, they are aligned to human preferences through supervised… 80 references, cited by 46 papers here."
authors: ["Michael J Ryan", "William Held", "Diyi Yang"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.853/"
doi: "10.18653/v1/2024.acl-long.853"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ryan-2024-unintended-impacts-llm-alignment-global/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ryan-2024-unintended-impacts-llm-alignment-global/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / Unintended Impacts of LLM Alignment on Global Representation

# Unintended Impacts of LLM Alignment on Global Representation

**Authors:** [Michael J Ryan](https://md.research.elamur.ai/free/en/authors/michael-j-ryan/), [William Held](https://md.research.elamur.ai/free/en/authors/william-held/) and [Diyi Yang](https://md.research.elamur.ai/free/en/authors/diyi-yang/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.853/) · [PDF](https://aclanthology.org/2024.acl-long.853.pdf) · [DOI 10.18653/v1/2024.acl-long.853](https://doi.org/10.18653/v1/2024.acl-long.853) · [OpenAlex](https://openalex.org/W4402671512)

**Citations:** 11 citations (OpenAlex, 2026-10-14) · [46](https://md.research.elamur.ai/free/en/papers/ryan-2024-unintended-impacts-llm-alignment-global/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Before chat LLMs are released, they are aligned to human preferences through supervised fine-tuning (SFT) and preference tuning (RLHF/PPO or DPO). Standard evaluations of alignment measure reasoning, knowledge and truthfulness. They rarely check whether the gains reach users around the world equally. The authors ask whose preferences alignment actually serves. They compare checkpoints from each stage (base, SFT, preference-tuned) of 9 open models built on Llama 2 7B and Mistral v0.1 7B, along three axes:

- **English dialects:** intent prediction in US, Indian and Nigerian English conversations.
- **Multilingualism:** extractive QA and reading comprehension.
- **Opinions:** views held by countries (GlobalOpinions survey) and views about countries (a new AskReddit-based set).

Results are mixed. Alignment boosts overall dialect performance but widens the gap between dialects. It unexpectedly improves several non-English languages. It shifts model opinions toward those of the US. These effects come from developer choices such as SFT data mixture, preference prompts and annotator pools. The authors therefore argue for documenting these choices and designing preference tuning more equitably.

## Key points
- **Models:** Llama 2 Chat, Tülu 2 DPO, Starling LM and Zephyr-7B-beta, plus their SFT and base checkpoints. The Llama 2 SFT checkpoint is unreleased.
- **Dialect gap:** The disparity between English dialects grows from about 1% before alignment to as much as 17.1% after.
- **Multilingual gains:** Only 13.1% of Tülu SFT data is non-English. Even so, Tülu improves in 6 of 9 languages on extractive QA and in all 9 on reading comprehension.
- **Opinion shift:** Every alignment procedure tested makes answers closer to US opinions than to those of China, Jordan or Nigeria.
- **AskReddit dataset and Starling RM bias:**
  - The new dataset contains 554 opinionated questions about countries.
  - On average, the Starling reward model scores 99.4% of other countries below the USA.
  - This bias barely carries over to Starling LM: country rankings across 185 countries cluster by base model family.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,052 citations
- [Georgia Institute of Technology](https://md.research.elamur.ai/free/en/universities/georgia-institute-of-technology/) — University, Atlanta, United States · 152 works · 27,763 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 626 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Michael J Ryan, William Held and Diyi Yang. 2024. Unintended Impacts of LLM Alignment on Global Representation. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ryan2024unintended,
  title = {{Unintended Impacts of LLM Alignment on Global Representation}},
  author = {Michael J Ryan, William Held and Diyi Yang},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.853},
  url = {https://doi.org/10.18653/v1/2024.acl-long.853}
}
```

- [← previous Understanding and Mitigating Language Confusion in LLMs](https://md.research.elamur.ai/free/en/papers/marchisio-2024-understanding-mitigating-language-confusion-llms/)
- [next → Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking](https://md.research.elamur.ai/free/en/papers/fung-2024-massively-multi-cultural-knowledge-acquisition/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

80 references, 41 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. 2024. [Yi: Open foundation models by 01.ai](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/).
2. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609. [↗](https://arxiv.org/abs/2309.16609)
3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. [↗](https://doi.org/10.48550/arxiv.2204.05862)
4. Michiel A. Bakker, Martin J Chadwick, Hannah Sheahan, Michael Henry Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matthew Botvinick, and Christopher Summerfield. 2022. Fine-tuning language models to find agreement among humans with diverse preferences. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Michiel+A%2E+Bakker%2C+Martin+J+Chadwick%2C+Hannah+Sheahan%2C+Michael+Henry+Tessler%2C+Lucy+Campbell%2DGillingham%2C+Jan+Balaguer%2C+Nat+McAleese%2C+Amelia+Glaese%2C+John+Aslanides%2C+Matthew+Botvinick%2C+and+Christopher+Sum)
5. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023. [The belebele benchmark: a parallel reading comprehension dataset in 122 language variants](https://md.research.elamur.ai/free/en/papers/bandarkar-2023-belebele-benchmark-parallel-reading-comprehension/). arXiv preprint arXiv:2308.16884. [↗](https://arxiv.org/abs/2308.16884)
6. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. [Language (technology) is power: A critical survey of “bias” in NLP](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454– 5476, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.485)
7. Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang, and Percy Liang. 2023. The foundation model transparency index. [↗](https://doi.org/10.48550/arxiv.2310.12941)
8. Megan Brenan. 2023. Canada, britain favored most in u.s.; russia, n. korea least. [find](https://scholar.google.com/scholar?q=Megan+Brenan%2E+2023%2E+Canada%2C+britain+favored+most+in+u%2Es%2E%3B+russia%2C+n%2E+korea+least%2E)
9. Yang Trista Cao, Anna Sotnikova, Jieyu Zhao, Linda X. Zou, Rachel Rudinger, and Hal Daume III au2. 2023. Multilingual large language models leak human stereotypes across language boundaries. [find](https://scholar.google.com/scholar?q=Yang+Trista+Cao%2C+Anna+Sotnikova%2C+Jieyu+Zhao%2C+Linda+X%2E+Zou%2C+Rachel+Rudinger%2C+and+Hal+Daume+III+au2%2E+2023%2E+Multilingual+large+language+models+leak+human+stereotypes+across+language+boundaries%2E)
10. Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. [https://github.com/sahil280114/codealpaca](https://github.com/sahil280114/codealpaca). [↗](https://github.com/sahil280114/codealpaca)
11. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/).
12. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. [Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/). Transactions of the Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2003.05002)
13. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). [↗](https://doi.org/10.48550/arxiv.1803.05457)
14. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168. [↗](https://arxiv.org/abs/2110.14168)
15. Together Computer. 2023. [Redpajama: an open dataset for training large language models](https://md.research.elamur.ai/free/en/papers/weber-2024-redpajama-open-dataset-training-large/).
16. Luigi Daniele and Suphavadeeprasit. 2023. Amplify-instruct: Synthetically generated diverse multi-turn conversations for effecient llm training. arXiv preprint arXiv:(comming soon). [find](https://scholar.google.com/scholar?q=Luigi+Daniele+and+Suphavadeeprasit%2E+2023%2E+Amplify%2Dinstruct%3A+Synthetically+generated+diverse+multi%2Dturn+conversations+for+effecient+llm+training%2E+arXiv+preprint+arXiv%3A%28comming+soon%29%2E)
17. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm.int8(): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339. [↗](https://arxiv.org/abs/2208.07339)
18. Harnoor Dhingra, Preetiha Jayashanker, Sayali Moghe, and Emma Strubell. 2023. Queer people are people first: Deconstructing sexual identity stereotypes in large language models. [↗](https://doi.org/10.48550/arxiv.2307.00101)
19. Esin Durmus, Karina Nyugen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2023. [Towards measuring the representation of subjective global opinions in language models](https://md.research.elamur.ai/free/en/papers/durmus-2023-towards-measuring-representation-subjective-global/). [↗](https://doi.org/10.48550/arxiv.2306.16388)
20. Jacob Eisenstein, Vinodkumar Prabhakaran, Clara Rivera, Dora Demszky, and Devyani Sharma. 2023. Md3: The multi-dialect dataset of dialogues. In InterSpeech. [find](https://scholar.google.com/scholar?q=Jacob+Eisenstein%2C+Vinodkumar+Prabhakaran%2C+Clara+Rivera%2C+Dora+Demszky%2C+and+Devyani+Sharma%2E+2023%2E+Md3%3A+The+multi%2Ddialect+dataset+of+dialogues%2E+In+InterSpeech%2E)
21. Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2023. Gpts are gpts: An early look at the labor market impact potential of large language models. arXiv preprint arXiv:2303.10130. [↗](https://arxiv.org/abs/2303.10130)
22. Emilio Ferrara. 2023. Should chatgpt be biased? challenges and risks of bias in large language models. First Monday. [↗](https://doi.org/10.5210/fm.v28i11.13346)
23. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. [↗](https://arxiv.org/abs/2101.00027)
24. Thomas Krendl Gilbert, Nathan Lambert, Sarah Dean, Tom Zick, and Aaron Snoswell. 2023. Reward reports for reinforcement learning. [find](https://scholar.google.com/scholar?q=Thomas+Krendl+Gilbert%2C+Nathan+Lambert%2C+Sarah+Dean%2C+Tom+Zick%2C+and+Aaron+Snoswell%2E+2023%2E+Reward+reports+for+reinforcement+learning%2E)
25. Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. 2023. [The political ideology of conversational ai: Converging evidence on chatgpt’s pro-environmental, left-libertarian orientation](https://md.research.elamur.ai/free/en/papers/hartmann-2023-political-ideology-conversational-ai-converging/). [↗](https://doi.org/10.25592/uhhfdm.11732)
26. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2009.03300)
27. Tom Hosking, Phil Blunsom, and Max Bartolo. 2023. Human feedback is not gold standard. [↗](https://doi.org/10.48550/arxiv.2309.16349)
28. Jing Huang and Diyi Yang. 2023. [Culturally aware natural language inference](https://md.research.elamur.ai/free/en/papers/huang-2023-culturally-aware-natural-language-inference/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7591–7609, Singapore. Association for Computational Linguistics. [↗](https://doi.org/10.48448/6etb-3283)
29. Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. [Camels in a changing climate: Enhancing lm adaptation with tulu 2](https://md.research.elamur.ai/free/en/papers/ivison-2023-camels-changing-climate-enhancing-lm/). [↗](https://doi.org/10.48550/arxiv.2311.10702)
30. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
31. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759. [↗](https://arxiv.org/abs/1607.01759)
32. Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2023. [Understanding the effects of rlhf on llm generalisation and diversity](https://md.research.elamur.ai/free/en/papers/kirk-2023-understanding-effects-rlhf-llm-generalisation/). [↗](https://doi.org/10.48550/arxiv.2310.06452)
33. Hadas Kotek, Rikker Dockum, and David Sun. 2023. Gender bias and stereotypes in large language models. In Proceedings of The ACM Collective Intelligence Conference, CI ’23, page 12–24, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Hadas+Kotek%2C+Rikker+Dockum%2C+and+David+Sun%2E+2023%2E+Gender+bias+and+stereotypes+in+large+language+models%2E+In+Proceedings+of+The+ACM+Collective+Intelligence+Conference%2C+CI+%E2%80%9923%2C+page+12%E2%80%9324%2C+New+York%2C+NY%2C+U)
34. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. 2023. [Openassistant conversations – democratizing large language model alignment](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/). [↗](https://doi.org/10.48550/arxiv.2304.07327)
35. Nathan Lambert, Thomas Krendl Gilbert, and Tom Zick. 2023. The history and risks of reinforcement learning and human feedback. [↗](https://doi.org/10.48550/arxiv.2310.13595)
36. Zongxia Li, Ishani Mondal, Yijun Liang, Huy Nghiem, and Jordan Boyd-Graber. 2024. Cfmatch: Aligning automated answer equivalence evaluation with expert judgments for open-domain question answering. [↗](https://doi.org/10.48550/arxiv.2401.13170)
37. Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023. Openorca: An open dataset of gpt augmented flan reasoning traces. [https://https://huggingface.co/Open-Orca/OpenOrca](https://https://huggingface.co/Open-Orca/OpenOrca). [↗](https://https://huggingface.co/Open-Orca/OpenOrca)
38. Andreas Liesenfeld, Alianda Lopez, and Mark Dingemanse. 2023. Opening up chatgpt: Tracking openness, transparency, and accountability in instruction-tuned text generators. In Proceedings of the 5th International Conference on Conversational User Interfaces, CUI ’23, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Andreas+Liesenfeld%2C+Alianda+Lopez%2C+and+Mark+Dingemanse%2E+2023%2E+Opening+up+chatgpt%3A+Tracking+openness%2C+transparency%2C+and+accountability+in+instruction%2Dtuned+text+generators%2E+In+Proceedings+of+the+5th+In)
39. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. [TruthfulQA: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics.
40. Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, and Tong Zhang. 2024. [Mitigating the alignment tax of rlhf](https://md.research.elamur.ai/free/en/papers/lin-2024-mitigating-alignment-tax-rlhf/). [↗](https://doi.org/10.48550/arxiv.2309.06256)
41. Gabrielle Kaili-May Liu. 2023. Perspectives on the social impacts of reinforcement learning with human feedback. [↗](https://doi.org/10.48550/arxiv.2303.02891)
42. Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, Xinyi Wu, Enrico Shippole, Kurt Bollacker, Tongshuang Wu, Luis Villa, Sandy Pentland, and Sara Hooker. 2023. The data provenance initiative: A large scale audit of dataset licensing & attribution in ai. [↗](https://doi.org/10.48550/arxiv.2310.16787)
43. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220–229. [find](https://scholar.google.com/scholar?q=Margaret+Mitchell%2C+Simone+Wu%2C+Andrew+Zaldivar%2C+Parker+Barnes%2C+Lucy+Vasserman%2C+Ben+Hutchinson%2C+Elena+Spitzer%2C+Inioluwa+Deborah+Raji%2C+and+Timnit+Gebru%2E+2019%2E+Model+cards+for+model+reporting%2E+In+Proceedi)
44. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). [↗](https://doi.org/10.52202/075280-2338)
45. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. [StereoSet: Measuring stereotypical bias in pretrained language models](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5356–5371, Online. Association for Computational Linguistics.
46. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv e-prints. [↗](https://doi.org/10.48550/arxiv.1910.10683)
47. Shuyo Nakatani. 2010. Language detection library for java. [find](https://scholar.google.com/scholar?q=Shuyo+Nakatani%2E+2010%2E+Language+detection+library+for+java%2E)
48. Tarek Naous, Michael J. Ryan, Alan Ritter, and Wei Xu. 2023. [Having beer after prayer? measuring cultural bias in large language models](https://md.research.elamur.ai/free/en/papers/naous-2023-having-beer-after-prayer-measuring/).
49. Gabriel Nicholas and Aliya Bhatia. 2023. [Lost in translation: Large language models in non-english content analysis](https://md.research.elamur.ai/free/en/papers/nicholas-2023-lost-translation-large-language-models/). [↗](https://doi.org/10.48550/arxiv.2306.07377)
50. OpenAI. 2023a. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
51. OpenAI. 2023b. Openai devday: Opening keynote. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023b%2E+Openai+devday%3A+Opening+keynote%2E)
52. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). [↗](https://doi.org/10.52202/068431-2011)
53. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. [Instruction tuning with gpt-4](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/). [↗](https://doi.org/10.48550/arxiv.2304.03277)
54. Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. 2023. [Discovering language model behaviors with model-written evaluations](https://md.research.elamur.ai/free/en/papers/perez-2022-discovering-language-model-behaviors-model/). In Findings of the Association for Computational Linguistics: ACL 2023, pages 13387–13434, Toronto, Canada. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2212.09251)
55. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2E)
56. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. [Whose opinions do language models reflect?](https://md.research.elamur.ai/free/en/papers/santurkar-2023-whose-opinions-language-models-reflect/) [↗](https://doi.org/10.48550/arxiv.2303.17548)
57. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [↗](https://arxiv.org/abs/1707.06347)
58. Omar Shaikh, Kristina Gligorić, Ashna Khetan, Matthias Gerstgrasser, Diyi Yang, and Dan Jurafsky. 2023. Grounding or guesswork? large language models are presumptive grounders. arXiv preprint arXiv:2311.09144. [↗](https://arxiv.org/abs/2311.09144)
59. Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2023. [A long way to go: Investigating length correlations in rlhf](https://md.research.elamur.ai/free/en/papers/singhal-2023-long-way-go-investigating-length/). [↗](https://doi.org/10.48550/arxiv.2310.03716)
60. Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. 2024. [A roadmap to pluralistic alignment](https://md.research.elamur.ai/free/en/papers/sorensen-2024-roadmap-pluralistic-alignment/). [↗](https://doi.org/10.48550/arxiv.2402.05070)
61. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. [Challenging BIG-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). In Findings of the Association for Computational Linguistics: ACL 2023, pages 13003–13051, Toronto, Canada. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2210.09261)
62. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca). [↗](https://github.com/tatsu-lab/stanford_alpaca)
63. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. [↗](https://doi.org/10.48550/arxiv.2207.04672)
64. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/).
65. Christoph Treude and Hideaki Hata. 2023. She elicits requirements and he tests: Software engineering gender bias in large language models. [find](https://scholar.google.com/scholar?q=Christoph+Treude+and+Hideaki+Hata%2E+2023%2E+She+elicits+requirements+and+he+tests%3A+Software+engineering+gender+bias+in+large+language+models%2E)
66. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alexander M. Rush, and Thomas Wolf. 2023a. The alignment handbook. [https://github.com/](https://github.com/) huggingface/alignment-handbook. [↗](https://github.com/)
67. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2023b. Zephyr: Direct distillation of lm alignment. [↗](https://doi.org/10.48550/arxiv.2310.16944)
68. Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, and Nanyun Peng. 2023. ["kelly is a warm person, joseph is a role model": Gender biases in llm-generated reference letters](https://md.research.elamur.ai/free/en/papers/wan-2023-kelly-is-warm-person-joseph/).
69. Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023. [Openchat: Advancing open-source language models with mixed-quality data](https://md.research.elamur.ai/free/en/papers/wang-2023-openchat-advancing-open-source-language/). arXiv preprint arXiv:2309.11235. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations. [↗](https://arxiv.org/abs/2309.11235)
70. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. [Wizardlm: Empowering large language models to follow complex instructions](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/). arXiv preprint arXiv:2304.12244. [↗](https://arxiv.org/abs/2304.12244)
71. Zheng-Xin Yong, Cristina Menghini, and Stephen H. Bach. 2023. [Low-resource languages jailbreak gpt-4](https://md.research.elamur.ai/free/en/papers/yong-2023-low-resource-languages-jailbreak-gpt/). [↗](https://doi.org/10.48550/arxiv.2310.02446)
72. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284. [↗](https://arxiv.org/abs/2309.12284)
73. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023. [Mammoth: Building math generalist models through hybrid instruction tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/). arXiv preprint arXiv:2309.05653. [↗](https://arxiv.org/abs/2309.05653)
74. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p19-1472)
75. Ge Zhang, Yemin Shi, Ruibo Liu, Ruibin Yuan, Yizhi Li, Siwei Dong, Yu Shu, Zhaoqun Li, Zekun Wang, Chenghua Lin, Wenhao Huang, and Jie Fu. 2023. Chinese open instruction generalist: A preliminary release. [↗](https://doi.org/10.48550/arxiv.2304.07987)
76. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging LLM-as-a-judge with MT-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
77. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. [Agieval: A human-centric benchmark for evaluating foundation models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/).
78. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. LIMA: Less is more for alignment. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srini+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+LILI+YU%2C+Susan+Zhang%2C+Gargi+Ghosh%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Omer+Levy%2E+2023%2E+LIMA%3A+Less)
79. Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. 2023. Starling-7b: Improving llm helpfulness & harmlessness with rlaif. [find](https://scholar.google.com/scholar?q=Banghua+Zhu%2C+Evan+Frick%2C+Tianhao+Wu%2C+Hanlin+Zhu%2C+and+Jiantao+Jiao%2E+2023%2E+Starling%2D7b%3A+Improving+llm+helpfulness+%26+harmlessness+with+rlaif%2E)
80. Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. [Multi-VALUE: A framework for cross-dialectal English NLP](https://md.research.elamur.ai/free/en/papers/ziems-2023-multi-value-framework-cross-dialectal/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 744–768, Toronto, Canada. Association for Computational Linguistics.

*Updated 2 November 2026*