---
title: "When Benchmarks are Targets — paper by Alzahrani et al. (2024)"
description: "The authors, from NCAI (SDAIA, Riyadh), test whether LLM leaderboards built on multiple-choice… 45 references, cited by 72 papers here."
authors: ["Norah Alzahrani", "Hisham Abdullah Alyahya", "Yazeed Alnumay", "Sultan Alrashed", "Shaykhah Alsubaie", "Yusef Almushaykeh", "Faisal Mirza", "Nouf Alotaibi", "Nora Altwairesh", "Areeb Alowisheq", "M Saiful Bari", "Haidar Khan"]
year: 2024
venue: "arXiv:2402.01781"
source: "https://arxiv.org/abs/2402.01781"
doi: "10.48550/arxiv.2402.01781"
arxiv: "2402.01781"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/alzahrani-2024-benchmarks-targets-revealing-sensitivity-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/alzahrani-2024-benchmarks-targets-revealing-sensitivity-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards

# When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards

**Authors:** [Norah Alzahrani](https://md.research.elamur.ai/free/en/authors/norah-alzahrani/), [Hisham Abdullah Alyahya](https://md.research.elamur.ai/free/en/authors/hisham-abdullah-alyahya/), [Yazeed Alnumay](https://md.research.elamur.ai/free/en/authors/yazeed-alnumay/), [Sultan Alrashed](https://md.research.elamur.ai/free/en/authors/sultan-alrashed/), [Shaykhah Alsubaie](https://md.research.elamur.ai/free/en/authors/shaykhah-z-alsubaie/), Yusef Almushaykeh, [Faisal Mirza](https://md.research.elamur.ai/free/en/authors/faisal-abdulrahman-mirza/), [Nouf Alotaibi](https://md.research.elamur.ai/free/en/authors/nouf-alotaibi/), [Nora Altwairesh](https://md.research.elamur.ai/free/en/authors/nora-altwairesh/), [Areeb Alowisheq](https://md.research.elamur.ai/free/en/authors/areeb-alowisheq/), [M Saiful Bari](https://md.research.elamur.ai/free/en/authors/m-saiful-bari/) and [Haidar Khan](https://md.research.elamur.ai/free/en/authors/haidar-khan/) · **Year:** 2024 · **Venue:** arXiv:2402.01781 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.01781) · [PDF](https://arxiv.org/pdf/2402.01781) · [DOI 10.48550/arxiv.2402.01781](https://doi.org/10.48550/arxiv.2402.01781) · [arXiv 2402.01781](https://arxiv.org/abs/2402.01781) · [OpenAlex](https://openalex.org/W4391591687)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [72](https://md.research.elamur.ai/free/en/papers/alzahrani-2024-benchmarks-targets-revealing-sensitivity-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors, from NCAI (SDAIA, Riyadh), test whether LLM leaderboards built on multiple-choice question (MCQ) benchmarks can be trusted. Their main subject is MMLU, and they also mention ARC and CSQA. The researchers systematically make small changes to the benchmarks that should not affect a model's knowledge. These changes fall into three groups. The first concerns the format and order of the answer choices: shuffling them, fixing the correct answer at one position, and replacing the letters A–D with common or rare symbols. The second covers changes to the prompt and the scoring method, for example scoring by the choice symbol versus cloze scoring. The third manipulates in-context knowledge, including hidden patterns in the few-shot examples. The authors compared the open models Llama-2, Mistral, Yi and Phi-2 and measured how well the rankings agreed with Kendall's τ. Even cosmetic changes moved models by as many as eight places on the leaderboard, and small and medium-sized models moved the most. The authors attribute this to model biases toward answer position, choice symbol and scoring method. They conclude that an expensive model should not be chosen on its leaderboard rank alone, and that hybrid scoring and more robust evaluation schemes are a better choice.

## Key points
- The models compared were Llama2 (7b, 13b, 70b and their chat versions), Mistral-7b and Mistral-7b-instruct, Yi-6b, Yi-34b and Phi-2. On MMLU their order shifted by up to 8 positions. Deviation from the original ranking was measured with Kendall's τ.
- The variants tested were two sets of replacement symbols (common ones, `$ & # @`, and rare ones such as `œ § ü` and a Cyrillic letter), fixing the correct answer at position A/B/C/D, and cloze scoring in place of choosing by letter.
- When all answers in the 5-shot examples are at the same position, accuracy drops for every model. For example, Yi-6B falls from 63.22 to 36.33 (−26.89 points). This shows that models pick up even unintended patterns in the context.
- Some model families depend too heavily on question format, which may point to benchmark leakage into the training data. At the same time, some of the changes did not affect the ranking.
- The authors recommend a hybrid scoring method, though they acknowledge that it is not fully robust to such changes either. The code is available in a fork of `lm-evaluation-harness` on GitHub (National-Center-for-AI-Saudi-Arabia).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Norah Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yusef Almushaykeh, Faisal Mirza, Nouf Alotaibi, Nora Altwairesh, Areeb Alowisheq, M Saiful Bari and Haidar Khan. 2024. When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards. arXiv:2402.01781.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{alzahrani2024when,
  title = {{When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards}},
  author = {Norah Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yusef Almushaykeh, Faisal Mirza, Nouf Alotaibi, Nora Altwairesh, Areeb Alowisheq, M Saiful Bari and Haidar Khan},
  year = {2024},
  doi = {10.48550/arxiv.2402.01781},
  eprint = {2402.01781},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.01781}
}
```

- [← previous Self-Taught Evaluators](https://md.research.elamur.ai/free/en/papers/wang-2024-self-taught-evaluators/)
- [next → Benchmarking Benchmark Leakage in Large Language Models](https://md.research.elamur.ai/free/en/papers/xu-2024-benchmarking-benchmark-leakage-large-language/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

45 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Se… — [PaLM 2 Technical Report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
2. Anthropic. 2023. Anthropic. model card and evaluations for claude models. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2023%2E+Anthropic%2E+model+card+and+evaluations+for+claude+models%2E)
3. Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. 2023. Open llm leaderboard. [https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard). [↗](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
4. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023. [A survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/). arXiv preprint arXiv:2307.03109.
5. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311.
6. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018a. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). ArXiv, abs/1803.05457.
7. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018b. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457.
8. OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. [https://github.com/open-compass/opencompass](https://github.com/open-compass/opencompass). [↗](https://github.com/open-compass/opencompass)
9. Google Deepmind. 2023. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/).
10. Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. 2021. The benchmark lottery. [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani%2C+Yi+Tay%2C+Alexey+A%2E+Gritsenko%2C+Zhe+Zhao%2C+Neil+Houlsby%2C+Fernando+Diaz%2C+Donald+Metzler%2C+and+Oriol+Vinyals%2E+2021%2E+The+benchmark+lottery%2E)
11. Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2023. Benchmark probing: Investigating data leakage in large language models. In NeurIPS 2023 Workshop on Backdoors in Deep Learning-The Good, the Bad, and the Ugly. [find](https://scholar.google.com/scholar?q=Chunyuan+Deng%2C+Yilun+Zhao%2C+Xiangru+Tang%2C+Mark+Gerstein%2C+and+Arman+Cohan%2E+2023%2E+Benchmark+probing%3A+Investigating+data+leakage+in+large+language+models%2E+In+NeurIPS+2023+Workshop+on+Backdoors+in+Deep+Lea)
12. Kawin Ethayarajh and Dan Jurafsky. 2021. Utility is in the eye of the user: A critique of nlp leaderboards. [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh+and+Dan+Jurafsky%2E+2021%2E+Utility+is+in+the+eye+of+the+user%3A+A+critique+of+nlp+leaderboards%2E)
13. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
14. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations.
15. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
16. Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81–93. [find](https://scholar.google.com/scholar?q=Maurice+G+Kendall%2E+1938%2E+A+new+measure+of+rank+correlation%2E+Biometrika%2C+30%281%2F2%29%3A81%E2%80%9393%2E)
17. Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Xiangji Huang. 2023. [A systematic study and comprehensive evaluation of chatgpt on benchmark datasets](https://md.research.elamur.ai/free/en/papers/laskar-2023-systematic-study-comprehensive-evaluation-chatgpt/). arXiv preprint arXiv:2305.18486.
18. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+Benjamin+Newman%2C+Binhang+Yuan%2C+Bobby+Yan%2C+Ce+Zhang%2C+C)
19. Séverin Lions, Carlos Monsalve, Pablo Dartnell, María Inés Godoy, Nora Córdova, Daniela Jiménez, María Paz Blanco, Gabriel Ortega, and Julie Lemarié. 2021. The position of distractors in multiple-choice test items: The strongest precede the weakest. In Frontiers in Education, volume 6, page 731763.… [find](https://scholar.google.com/scholar?q=S%C3%A9verin+Lions%2C+Carlos+Monsalve%2C+Pablo+Dartnell%2C+Mar%C3%ADa+In%C3%A9s+Godoy%2C+Nora+C%C3%B3rdova%2C+Daniela+Jim%C3%A9nez%2C+Mar%C3%ADa+Paz+Blanco%2C+Gabriel+Ortega%2C+and+Julie+Lemari%C3%A9%2E+2021%2E+The+position+of+distractors+in+multiple%2Dchoi)
20. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. [Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p…
21. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Ab…
22. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generalization via natural language crowdsourcing instructions. [find](https://scholar.google.com/scholar?q=Swaroop+Mishra%2C+Daniel+Khashabi%2C+Chitta+Baral%2C+and+Hannaneh+Hajishirzi%2E+2022%2E+Cross%2Dtask+generalization+via+natural+language+crowdsourcing+instructions%2E)
23. Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt llm evaluation. [find](https://scholar.google.com/scholar?q=Moran+Mizrahi%2C+Guy+Kaplan%2C+Dan+Malkin%2C+Rotem+Dror%2C+Dafna+Shahaf%2C+and+Gabriel+Stanovsky%2E+2024%2E+State+of+what+art%3F+a+call+for+multi%2Dprompt+llm+evaluation%2E)
24. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. [Adversarial nli: A new benchmark for natural language understanding](https://md.research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/). arXiv preprint arXiv:1910.14599.
25. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
26. Pouya Pezeshkpour and Estevam Hruschka. 2023. [Large language models sensitivity to the order of options in multiple-choice questions](https://md.research.elamur.ai/free/en/papers/pezeshkpour-2023-large-language-models-sensitivity-order/).
27. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21(140):1–67.
28. Joshua Robinson, Christopher Michael Rytting, and David Wingate. 2023. [Leveraging large language models for multiple choice question answering](https://md.research.elamur.ai/free/en/papers/robinson-2022-leveraging-large-language-models-multiple/).
29. Amrita Saha, Vardaan Pahuja, Mitesh M. Khapra, Karthik Sankaranarayanan, and Sarath Chandar. 2018. Complex sequential question answering: Towards learning to converse over linked question answer pairs with a knowledge graph. [find](https://scholar.google.com/scholar?q=Amrita+Saha%2C+Vardaan+Pahuja%2C+Mitesh+M%2E+Khapra%2C+Karthik+Sankaranarayanan%2C+and+Sarath+Chandar%2E+2018%2E+Complex+sequential+question+answering%3A+Towards+learning+to+converse+over+linked+question+answer+pairs)
30. Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. [Multitask prompted training enables zero-shot task generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/). arXiv preprint arXiv:2110.08207.
31. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. [find](https://scholar.google.com/scholar?q=Melanie+Sclar%2C+Yejin+Choi%2C+Yulia+Tsvetkov%2C+and+Alane+Suhr%2E+2023%2E+Quantifying+language+models%E2%80%99+sensitivity+to+spurious+features+in+prompt+design+or%3A+How+i+learned+to+start+worrying+about+prompt+formatt)
32. Chenhui Shen, Liying Cheng, Yang You, and Lidong Bing. 2023. Are large language models good evaluators for abstractive summarization? arXiv preprint arXiv:2305.13091. [find](https://scholar.google.com/scholar?q=Chenhui+Shen%2C+Liying+Cheng%2C+Yang+You%2C+and+Lidong+Bing%2E+2023%2E+Are+large+language+models+good+evaluators+for+abstractive+summarization%3F+arXiv+preprint+arXiv%3A2305%2E13091%2E)
33. Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, et al. 2022. [Alexatm 20b: Few-shot learning using a large-scale multilingual seq2seq model](https://md.research.elamur.ai/free/en/papers/soltan-2022-alexatm-20b-few-shot-learning/). arXiv preprint arXiv:2208.01448.
34. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. [Challenging big-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). arXiv preprint arXiv:2210.09261.
35. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
36. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). Advances in neural information processing systems, 32.
37. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). arXiv preprint arXiv:1804.07461.
38. Yike Wang, Shangbin Feng, Heng Wang, Weijia Shi, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. 2023. [Resolving knowledge conflicts in large language models](https://md.research.elamur.ai/free/en/papers/wang-2023-resolving-knowledge-conflicts-large-language/).
39. Lucas Weber, Elia Bruni, and Dieuwke Hupkes. 2023. [Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learning](https://md.research.elamur.ai/free/en/papers/weber-2023-mind-instructions-holistic-evaluation-consistency/). In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 294–313, Singapore. Association for Computational…
40. Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. 2023. [Larger language models do in-context learning differently](https://md.research.elamur.ai/free/en/papers/wei-2023-larger-language-models-context-learning/).
41. Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2023. [Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts](https://md.research.elamur.ai/free/en/papers/xie-2023-adaptive-chameleon-stubborn-sloth-revealing/).
42. Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang goo Lee, and Taeuk Kim. 2022. [Ground-truth labels matter: A deeper look into input-label demonstrations](https://md.research.elamur.ai/free/en/papers/yoo-2022-ground-truth-labels-matter-deeper/).
43. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. [Calibrate before use: Improving few-shot performance of language models](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/). In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697–12706. PMLR.
44. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023. Large language models are not robust multiple choice selectors. arXiv e-prints, pages arXiv–2309. [find](https://scholar.google.com/scholar?q=Chujie+Zheng%2C+Hao+Zhou%2C+Fandong+Meng%2C+Jie+Zhou%2C+and+Minlie+Huang%2E+2023%2E+Large+language+models+are+not+robust+multiple+choice+selectors%2E+arXiv+e%2Dprints%2C+pages+arXiv%E2%80%932309%2E)
45. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. [Agieval: A human-centric benchmark for evaluating foundation models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/). arXiv preprint arXiv:2304.06364.

*Updated 2 November 2026*