---
title: "ICLEval — paper by Chen et al. (2025) — summary and ideas"
description: "To keep models from relying on memorized facts, the authors replace entities with hash strings and… 62 references, cited by 7 papers here."
authors: ["Wentong Chen", "Yankai Lin", "ZhenHao Zhou", "HongYun Huang", "YanTao Jia", "Zhao Cao", "Ji-Rong Wen"]
year: 2025
venue: "Proceedings of the 31st International Conference on Computational Linguistics"
source: "https://aclanthology.org/2025.coling-main.693/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2025-icleval-evaluating-context-learning-ability/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2025-icleval-evaluating-context-learning-ability/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/10/) / ICLEval: Evaluating In-Context Learning Ability of Large Language Models

# ICLEval: Evaluating In-Context Learning Ability of Large Language Models

**Authors:** [Wentong Chen](https://md.research.elamur.ai/free/en/authors/wentong-chen/), [Yankai Lin](https://md.research.elamur.ai/free/en/authors/yankai-lin/), [ZhenHao Zhou](https://md.research.elamur.ai/free/en/authors/zhenhao-zhou/), HongYun Huang, YanTao Jia, [Zhao Cao](https://md.research.elamur.ai/free/en/authors/zhao-cao/) and [Ji-Rong Wen](https://md.research.elamur.ai/free/en/authors/ji-rong-wen/) · **Year:** 2025 · **Venue:** [Proceedings of the 31st International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2025](https://md.research.elamur.ai/free/en/venues/coling/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.coling-main.693/) · [PDF](https://aclanthology.org/2025.coling-main.693.pdf) · [OpenAlex](https://openalex.org/W4399992032)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/chen-2025-icleval-evaluating-context-learning-ability/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most LLM benchmarks mix in-context learning (ICL) with language skill and world knowledge, so a low score does not show which ability is weak. ICLEval addresses this by isolating ICL into two components. The first is exact copying: finding a matching prefix and reproducing what follows it, in both free text and key-value "dictionary" contexts. The second is rule learning: inferring format, order, statistical and list-mapping rules from n-shot examples.

To keep models from relying on memorized facts, the authors replace entities with hash strings and design tasks that need little linguistic or factual knowledge. They tested models from 1.1B to 65B parameters, along with intermediate pretraining checkpoints. Their main findings are:
- Every model shows some ICL ability.
- Larger models tend to do better, but size alone does not decide performance.
- Copying in particular appears very early in pretraining and then levels off.

An error analysis traces failures to tokenization and other model-internal factors. The benchmark therefore gives a cleaner way to compare and study how LLMs acquire ICL.

## Key points
- **Benchmark size:** ICLEval contains 12 tasks with 2,040 test samples, scored mostly by exact match. The code and data are at github.com/RUCBM/ICLEval.
- **Copying tasks:** String Completion (100 samples) and Dictionary Search (190). **Rule-learning tasks** include Format Check, Format Cloning and Format Conversion (e.g., Markdown table to XML), Order Check and Order Adjustment, Duplication Check, De-Duplication, Count & Navigation, and Relation Analysis.
- **Numbers' List Mapping:** This task (250 samples, one per mapping rule) reuses the "list_functions" task from BIG-Bench.
- **Small models can compete:** TinyLlama-1.1B scores 0.89 on unstructured copying, close to 7B models such as ChatGLM3-6B (0.90) and Qwen-7B (0.89). LLaMA3-8B scores only 0.57 on the same task.
- **Sources of failure:** The authors identify four factors: weak discrimination between similar contexts, inherent model preferences that override the context, limited capacity to track many attention points, and the tokenizer. Tokenizer effects take three forms, which the authors call token fracture, token insert and token replace.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Renmin University of China](https://md.research.elamur.ai/free/en/universities/renmin-university-of-china/) — University, Beijing, China · 114 works · 16,118 citations
- [Huawei Technologies](https://md.research.elamur.ai/free/en/companies/huawei-technologies/) — Company, Shenzhen, China · 75 works · 12,975 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Wentong Chen, Yankai Lin, ZhenHao Zhou, HongYun Huang, YanTao Jia, Zhao Cao and Ji-Rong Wen. 2025. ICLEval: Evaluating In-Context Learning Ability of Large Language Models. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chen2025icleval,
  title = {{ICLEval: Evaluating In-Context Learning Ability of Large Language Models}},
  author = {Wentong Chen, Yankai Lin, ZhenHao Zhou, HongYun Huang, YanTao Jia, Zhao Cao and Ji-Rong Wen},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.693/}
}
```

- [← previous GUM-SAGE: A Novel Dataset and Approach for Graded Entity Salience Prediction](https://md.research.elamur.ai/free/en/papers/lin-2025-gum-sage-novel-dataset-approach/)
- [next → Internal Chain-of-Thought: Empirical Evidence for Layer-wise Subtask Scheduling in LLMs](https://md.research.elamur.ai/free/en/papers/yang-2025-internal-chain-thought-empirical-evidence/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/10/)

## References

62 references, 44 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2022. [What learning algorithm is in-context learning? investigations with linear models](https://md.research.elamur.ai/free/en/papers/akyurek-2022-learning-algorithm-context-learning-investigations/). arXiv preprint arXiv:2211.15661.
2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732.
3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609.
4. Baichuan. 2023. [Baichuan 2: Open large-scale language models](https://md.research.elamur.ai/free/en/papers/yang-2023-baichuan-2-open-large-scale/). arXiv preprint arXiv:2309.10305.
5. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. [Semantic parsing on freebase from question-answer pairs](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/). In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544.
6. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.
7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
8. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712.
9. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021a. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
10. Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. 2021b. Meta-learning via language model in-context tuning. arXiv preprint arXiv:2110.07814. [find](https://scholar.google.com/scholar?q=Yanda+Chen%2C+Ruiqi+Zhong%2C+Sheng+Zha%2C+George+Karypis%2C+and+He+He%2E+2021b%2E+Meta%2Dlearning+via+language+model+in%2Dcontext+tuning%2E+arXiv+preprint+arXiv%3A2110%2E07814%2E)
11. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457.
12. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
13. Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. 2023. Active prompting with chain-of-thought for large language models. arXiv preprint arXiv:2302.12246. [find](https://scholar.google.com/scholar?q=Shizhe+Diao%2C+Pengcheng+Wang%2C+Yong+Lin%2C+and+Tong+Zhang%2E+2023%2E+Active+prompting+with+chain%2Dof%2Dthought+for+large+language+models%2E+arXiv+preprint+arXiv%3A2302%2E12246%2E)
14. Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. 2022. Successive prompting for decomposing complex questions. arXiv preprint arXiv:2212.04092. [find](https://scholar.google.com/scholar?q=Dheeru+Dua%2C+Shivanshu+Gupta%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+2022%2E+Successive+prompting+for+decomposing+complex+questions%2E+arXiv+preprint+arXiv%3A2212%2E04092%2E)
15. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022. [Complexity-based prompting for multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2022-complexity-based-prompting-multi-step/). arXiv preprint arXiv:2210.00720.
16. Chi Han, Ziqi Wang, Han Zhao, and Heng Ji. 2023. [In-context learning of large language models explained as kernel regression](https://md.research.elamur.ai/free/en/papers/han-2023-understanding-emergent-context-learning-kernel/). arXiv preprint arXiv:2305.12766.
17. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+et+al%2E+2022%2E+Holistic+evaluation+of+language+models%2E+)
18. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). arXiv preprint arXiv:2109.07958.
19. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300.
20. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146. [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Dani+Yogatama%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+2017%2E+Program+induction+by+rationale+generation%3A+Learning+to+solve+and+explain+algebraic+word+problems%2E+arXiv+preprint+arXiv%3A1705%2E04146%2E)
21. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874.
22. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023a. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). arXiv preprint arXiv:2307.03172.
23. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. 2023. [C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/). arXiv preprint arXiv:2305.08322.
24. Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Roberto Iriondo, Cun Mu, Z… [find](https://scholar.google.com/scholar?q=Zhengzhong+Liu%2C+Aurick+Qiao%2C+Willie+Neiswanger%2C+Hongyi+Wang%2C+Bowen+Tan%2C+Tianhua+Tao%2C+Junbo+Li%2C+Yuqi+Wang%2C+Suqi+Sun%2C+Omkar+Pangarkar%2C+Richard+Fan%2C+Yi+Gu%2C+Victor+Miller%2C+Yonghao+Zhuang%2C+Guowei+He%2C+Haona)
25. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada. Association for Computationa…
26. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). Advances in neural information processing systems, 35:22199– 22213.
27. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. [Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistic…
28. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. [Race: Large-scale reading comprehension dataset from examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). arXiv preprint arXiv:1704.04683.
29. Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. [Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning](https://md.research.elamur.ai/free/en/papers/lai-2023-chatgpt-beyond-english-comprehensive-evaluation/). arXiv preprint arXiv:2304.05613.
30. Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. 2023. Transformers as algorithms: Generalization and stability in in-context learning. [find](https://scholar.google.com/scholar?q=Yingcong+Li%2C+Muhammed+Emrullah+Ildiz%2C+Dimitris+Papailiopoulos%2C+and+Samet+Oymak%2E+2023%2E+Transformers+as+algorithms%3A+Generalization+and+stability+in+in%2Dcontext+learning%2E)
31. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. [Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/). arXiv preprint arXiv:2104.08786.
32. Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. [Faithful chain-of-thought reasoning](https://md.research.elamur.ai/free/en/papers/lyu-2023-faithful-chain-thought-reasoning/). arXiv preprint arXiv:2301.13379.
33. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+arXiv+preprint+arXiv%3A1809%2E02789%2E)
34. Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2021. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Hannaneh+Hajishirzi%2E+2021%2E+Metaicl%3A+Learning+to+learn+in+context%2E+arXiv+preprint+arXiv%3A2110%2E15943%2E)
35. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) arXiv preprint arXiv:2202.12837.
36. Feng Nie, Meixi Chen, Zhirui Zhang, and Xu Cheng. 2022. Improving few-shot performance of language models via nearest neighbor calibration. arXiv preprint arXiv:2212.02216. — [arXiv:2212.02216](https://arxiv.org/abs/2212.02216) [find](https://scholar.google.com/scholar?q=Feng+Nie%2C+Meixi+Chen%2C+Zhirui+Zhang%2C+and+Xu+Cheng%2E+2022%2E+Improving+few%2Dshot+performance+of+language+models+via+nearest+neighbor+calibration%2E+arXiv+preprint+arXiv%3A2212%2E02216%2E+%E2%80%94+%5BarXiv%3A2212%2E02216%5D%28https%3A)
37. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. 2022. [In-context learning and induction heads](https://md.research.elamur.ai/free/en/papers/olsson-2022-context-learning-induction-heads/). arXiv preprint arXiv:2209.11895.
38. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
39. Tianduo Wang Peiyuan Zhang, Guangtao Zeng and Wei Lu. 2023. Tinyllama. [find](https://scholar.google.com/scholar?q=Tianduo+Wang+Peiyuan+Zhang%2C+Guangtao+Zeng+and+Wei+Lu%2E+2023%2E+Tinyllama%2E)
40. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023a. [Is chatgpt a general-purpose natural language processing task solver?](https://md.research.elamur.ai/free/en/papers/qin-2023-chatgpt-general-purpose-natural-language/) arXiv preprint arXiv:2302.06476.
41. Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. 2023b. [Tool learning with foundation models](https://md.research.elamur.ai/free/en/papers/qin-2023-tool-learning-foundation-models/). arXiv preprint arXiv:2304.08354.
42. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. [Know what you don’t know: Unanswerable questions for squad](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). arXiv preprint arXiv:1806.03822.
43. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. [Coqa: A conversational question answering challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/). Transactions of the Association for Computational Linguistics, 7:249–266.
44. Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2022. [Distilling multi-step reasoning capabilities of large language models into smaller models via semantic decompositions](https://md.research.elamur.ai/free/en/papers/shridhar-2023-distilling-reasoning-capabilities-into-smaller/). arXiv preprint arXiv:2212.00193.
45. KaShun Shum, Shizhe Diao, and Tong Zhang. 2023. [Automatic prompt augmentation and selection with chain-of-thought from labeled data](https://md.research.elamur.ai/free/en/papers/shum-2023-automatic-prompt-augmentation-selection-chain/). arXiv preprint arXiv:2302.12822.
46. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). arXiv preprint arXiv:2206.04615.
47. InternLM Team. 2023. Internlm: A multilingual language model with progressively enhanced capabilities. [https://github.com/InternLM/](https://github.com/InternLM/) InternLM. [↗](https://github.com/InternLM/)
48. Arun James Thirunavukarasu, Refaat Hassan, Shathar Mahmood, Rohan Sanghera, Kara Barzangi, Mo-hanned El Mukashfi, and Sachin Shah. 2023. Trialling a large language model (chatgpt) in general practice with the applied knowledge test: observational study demonstrating opportunities and limitations in… [find](https://scholar.google.com/scholar?q=Arun+James+Thirunavukarasu%2C+Refaat+Hassan%2C+Shathar+Mahmood%2C+Rohan+Sanghera%2C+Kara+Barzangi%2C+Mo%2Dhanned+El+Mukashfi%2C+and+Sachin+Shah%2E+2023%2E+Trialling+a+large+language+model+%28chatgpt%29+in+general+practice+)
49. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
50. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
51. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). arXiv preprint arXiv:1804.07461.
52. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171.
53. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837.
54. Zhenyu Wu, YaoXiang Wang, Jiacheng Ye, Jiangtao Feng, Jingjing Xu, Yu Qiao, and Zhiyong Wu. 2023. Openicl: An open-source framework for in-context learning. arXiv preprint arXiv:2303.02913. [find](https://scholar.google.com/scholar?q=Zhenyu+Wu%2C+YaoXiang+Wang%2C+Jiacheng+Ye%2C+Jiangtao+Feng%2C+Jingjing+Xu%2C+Yu+Qiao%2C+and+Zhiyong+Wu%2E+2023%2E+Openicl%3A+An+open%2Dsource+framework+for+in%2Dcontext+learning%2E+arXiv+preprint+arXiv%3A2303%2E02913%2E)
55. Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. [An explanation of in-context learning as implicit bayesian inference](https://md.research.elamur.ai/free/en/papers/xie-2021-explanation-context-learning-implicit-bayesian/). arXiv preprint arXiv:2111.02080.
56. Jiaxi Yang, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. 2023. Iterative forward tuning boosts in-context learning in language models. arXiv preprint arXiv:2305.13016. [find](https://scholar.google.com/scholar?q=Jiaxi+Yang%2C+Binyuan+Hui%2C+Min+Yang%2C+Binhua+Li%2C+Fei+Huang%2C+and+Yongbin+Li%2E+2023%2E+Iterative+forward+tuning+boosts+in%2Dcontext+learning+in+language+models%2E+arXiv+preprint+arXiv%3A2305%2E13016%2E)
57. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+2023%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+arXiv+preprint+ar)
58. Xi Ye, Srinivasan Iyer, Asli Celikyilmaz, Ves Stoyanov, Greg Durrett, and Ramakanth Pasunuru. 2022. [Complementary explanations for effective in-context learning](https://md.research.elamur.ai/free/en/papers/ye-2023-complementary-explanations-effective-context-learning/). arXiv preprint arXiv:2211.13892.
59. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. 2023. How well do large language models perform in arithmetic tasks? arXiv preprint arXiv:2304.02015. [find](https://scholar.google.com/scholar?q=Zheng+Yuan%2C+Hongyi+Yuan%2C+Chuanqi+Tan%2C+Wei+Wang%2C+and+Songfang+Huang%2E+2023%2E+How+well+do+large+language+models+perform+in+arithmetic+tasks%3F+arXiv+preprint+arXiv%3A2304%2E02015%2E)
60. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2E)
61. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. [Agieval: A human-centric benchmark for evaluating foundation models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/). arXiv preprint arXiv:2304.06364.
62. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. 2022. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). arXiv preprint arXiv:2205.10625.

*Updated 1 November 2026*