---
title: "What In-Context Learning “Learns” In-Context: Disentangling"
description: "The paper asks what language models actually do when they solve tasks from a few in-context… 28 references, cited by 71 papers here."
authors: ["Jane Pan", "Tianyu Gao", "Howard Chen", "Danqi Chen"]
year: 2023
venue: "Findings of the Association for Computational Linguistics: ACL 2023"
source: "https://aclanthology.org/2023.findings-acl.527/"
doi: "10.18653/v1/2023.findings-acl.527"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pan-2023-what-context-learning-learns-context/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pan-2023-what-context-learning-learns-context/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning

# What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning

**Authors:** Jane Pan, [Tianyu Gao](https://md.research.elamur.ai/free/en/authors/tianyu-gao/), [Howard Chen](https://md.research.elamur.ai/free/en/authors/howard-chen/) and [Danqi Chen](https://md.research.elamur.ai/free/en/authors/danqi-chen/) · **Year:** 2023 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2023](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2023](https://md.research.elamur.ai/free/en/venues/findings/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.findings-acl.527/) · [PDF](https://aclanthology.org/2023.findings-acl.527.pdf) · [DOI 10.18653/v1/2023.findings-acl.527](https://doi.org/10.18653/v1/2023.findings-acl.527) · [OpenAlex](https://openalex.org/W4385571671)

**Citations:** 29 citations (OpenAlex, 2026-10-14) · [71](https://md.research.elamur.ai/free/en/papers/pan-2023-what-context-learning-learns-context/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks what language models actually do when they solve tasks from a few in-context examples. Do they simply identify a task they already know from pre-training, or do they acquire new input-label associations from the prompt? The authors separate these into two abilities. **Task recognition (TR)** means identifying the task and applying prior knowledge. **Task learning (TL)** means picking up a mapping never seen during pre-training. They measure each under three conditions. GOLD uses correct labels. RANDOM uses shuffled labels, which isolates TR. ABSTRACT uses meaningless symbols remapped per prompt with minimal templates, which isolates TL. Experiments span 16 classification datasets and three model families (GPT-3, LLaMA, OPT), varying both model size and number of demonstrations. TR already works in small models but stays flat as models or prompts grow. TL emerges only at scale and keeps improving with more examples. The authors argue that future ICL studies should report which regime they are probing, since the two behave so differently.

## Key points
- Models tested: GPT-3 (ada 350M, babbage 1.3B, curie 6.7B, davinci 175B), LLaMA (7B–65B), and OPT (350M–66B). The datasets cover sentiment, toxicity, NLI/paraphrase, and topic/stance tasks.
- With the largest models and more than 16 demonstrations, ABSTRACT surpasses RANDOM and approaches GOLD. The gap between GOLD and RANDOM widens with scale, which nuances the earlier findings of Min et al. (2022).
- Abstract label types were compared. Numbers and letters consistently beat symbols such as "@" or "#", probably because they are more common in pre-training text.
- TL scales less on harder tasks: NLI shows a flatter ABSTRACT curve than sentiment analysis.
- Code is released at github.com/princeton-nlp/WhatICLLearns. The study is limited to classification tasks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Princeton University](https://md.research.elamur.ai/free/en/universities/princeton-university/) — University, Princeton, United States · 61 works · 36,471 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Jane Pan, Tianyu Gao, Howard Chen and Danqi Chen. 2023. What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning. Findings of the Association for Computational Linguistics: ACL 2023.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pan2023what,
  title = {{What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning}},
  author = {Jane Pan, Tianyu Gao, Howard Chen and Danqi Chen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.527},
  url = {https://doi.org/10.18653/v1/2023.findings-acl.527}
}
```

- [← previous SemEval-2023 Task 11: Learning with Disagreements (LeWiDi)](https://md.research.elamur.ai/free/en/papers/leonardelli-2023-semeval-2023-task-11-learning/)
- [next → Inverse Scaling: When Bigger Isn't Better](https://md.research.elamur.ai/free/en/papers/mckenzie-2023-inverse-scaling-bigger-isn-t/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

28 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2023. [What learning algorithm is in-context learning? investigations with linear models](https://md.research.elamur.ai/free/en/papers/akyurek-2022-learning-algorithm-context-learning-investigations/). In International Conference on Learning Representations (ICLR).
2. Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, and Dan Roth. 2022. Rethinking the role of scale for in-context learning: An interpretability-based case study at 66 billion scale. arXiv preprint arXiv:2212.09095. [find](https://scholar.google.com/scholar?q=Hritik+Bansal%2C+Karthik+Gopalakrishnan%2C+Saket+Dingliwal%2C+Sravan+Bodapati%2C+Katrin+Kirchhoff%2C+and+Dan+Roth%2E+2022%2E+Rethinking+the+role+of+scale+for+in%2Dcontext+learning%3A+An+interpretability%2Dbased+case+stud)
3. Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. [SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter](https://md.research.elamur.ai/free/en/papers/basile-2019-semeval-2019-task-5-multilingual/). In Proceedings of the 13th International Wor…
4. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Empirical Methods in Natural Language Processing (EMNLP), pages 632–642. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Empirical+Methods+in+Natural+Language+Processi)
5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. 2022. Data distributional properties drive emergent in-context learning in transformers. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 18878… [find](https://scholar.google.com/scholar?q=Stephanie+Chan%2C+Adam+Santoro%2C+Andrew+Lampinen%2C+Jane+Wang%2C+Aaditya+Singh%2C+Pierre+Richemond%2C+James+McClelland%2C+and+Felix+Hill%2E+2022%2E+Data+distributional+properties+drive+emergent+in%2Dcontext+learning+in+)
7. Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. 2023. Why can GPT learn in-context? language models implicitly perform gradient descent as meta-optimizers. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models. [find](https://scholar.google.com/scholar?q=Damai+Dai%2C+Yutao+Sun%2C+Li+Dong%2C+Yaru+Hao%2C+Shuming+Ma%2C+Zhifang+Sui%2C+and+Furu+Wei%2E+2023%2E+Why+can+GPT+learn+in%2Dcontext%3F+language+models+implicitly+perform+gradient+descent+as+meta%2Doptimizers%2E+In+ICLR+2023)
8. William B. Dolan and Chris Brockett. 2005. [Automatically constructing a corpus of sentential paraphrases](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/). In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
9. Louis Kirsch, James Harrison, Jascha Sohl-Dickstein, and Luke Metz. 2022. General-purpose in-context learning by meta-learning transformers. arXiv preprint arXiv:2212.04458. — [arXiv:2212.04458](https://arxiv.org/abs/2212.04458) [find](https://scholar.google.com/scholar?q=Louis+Kirsch%2C+James+Harrison%2C+Jascha+Sohl%2DDickstein%2C+and+Luke+Metz%2E+2022%2E+General%2Dpurpose+in%2Dcontext+learning+by+meta%2Dlearning+transformers%2E+arXiv+preprint+arXiv%3A2212%2E04458%2E+%E2%80%94+%5BarXiv%3A2212%2E04458%5D%28https)
10. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+winograd+schema+challenge%2E+In+Thirteenth+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2E)
11. P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. 2014. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology. [find](https://scholar.google.com/scholar?q=P%2E+Malo%2C+A%2E+Sinha%2C+P%2E+Korhonen%2C+J%2E+Wallenius%2C+and+P%2E+Takala%2E+2014%2E+Good+debt+or+bad+debt%3A+Detecting+semantic+orientations+in+economic+texts%2E+Journal+of+the+Association+for+Information+Science+and+Tech)
12. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. [A SICK cure for the evaluation of compositional distributional semantic models](https://md.research.elamur.ai/free/en/papers/marelli-2014-sick-cure-evaluation-compositional-distributional/). In International Conference on Language Resources and Evaluation (LREC), pages 216–223.
13. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) In Empirical Methods in Natural Language Processing (EMNLP), pages 11048–11064.
14. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. [SemEval-2018 task 1: Affect in tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2018-semeval-2018-task-1-affect/). In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 1–17. Association for Computational Linguistics.
15. Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. 2020. Ethos: an online hate speech detection dataset. arXiv preprint arXiv:2006.08328. [find](https://scholar.google.com/scholar?q=Ioannis+Mollas%2C+Zoe+Chrysopoulou%2C+Stamatis+Karlos%2C+and+Grigorios+Tsoumakas%2E+2020%2E+Ethos%3A+an+online+hate+speech+detection+dataset%2E+arXiv+preprint+arXiv%3A2006%2E08328%2E)
16. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. 2022. In-context learning and induction heads. arXiv preprint arXiv:2209.11895. [find](https://scholar.google.com/scholar?q=Catherine+Olsson%2C+Nelson+Elhage%2C+Neel+Nanda%2C+Nicholas+Joseph%2C+Nova+DasSarma%2C+Tom+Henighan%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+et+al%2E+2022%2E+In%2Dcontext+learning+and+induction+heads%2E+arXiv+p)
17. Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. [Impact of pretraining term frequencies on few-shot numerical reasoning](https://md.research.elamur.ai/free/en/papers/razeghi-2022-impact-pretraining-term-frequencies-few/). In Findings of Empirical Methods in Natural Language Processing (EMNLP), pages 840–854. Association for Computational Linguistics.
18. Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018. [CARER: Contextualized affect representations for emotion recognition](https://md.research.elamur.ai/free/en/papers/saravia-2018-carer-contextualized-affect-representations-emotion/). In Empirical Methods in Natural Language Processing (EMNLP), pages 3687–3697.
19. Emily Sheng and David Uthus. 2020. Investigating societal biases in a poetry composition system. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 93–106. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Emily+Sheng+and+David+Uthus%2E+2020%2E+Investigating+societal+biases+in+a+poetry+composition+system%2E+In+Proceedings+of+the+Second+Workshop+on+Gender+Bias+in+Natural+Language+Processing%2C+pages+93%E2%80%93106%2E+Asso)
20. Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. 2023. [Larger language models do in-context learning differently](https://md.research.elamur.ai/free/en/papers/wei-2023-larger-language-models-context-learning/). arXiv preprint arXiv:2303.03846.
21. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
22. Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. [An explanation of in-context learning as implicit bayesian inference](https://md.research.elamur.ai/free/en/papers/xie-2021-explanation-context-learning-implicit-bayesian/). In International Conference on Learning Representations (ICLR).
23. Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, and Taeuk Kim. 2022. [Ground-truth labels matter: A deeper look into input-label demonstrations](https://md.research.elamur.ai/free/en/papers/yoo-2022-ground-truth-labels-matter-deeper/). In Empirical Methods in Natural Language Processing (EMNLP), pages 2422–2437.
24. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. [OPT: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068.
25. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Empirical Methods in Natural Language Processing (EMNLP), pages 1631–1642. [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D%2E+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In)
26. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv pr…
27. Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. 2022. [Transformers learn in-context by gradient descent](https://md.research.elamur.ai/free/en/papers/oswald-2022-transformers-learn-context-gradient-descent/). arXiv preprint arXiv:2212.07677.
28. Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In Association for Computing Machinery Special Interest Group in Information Retrieval (ACM SIGIR), pages 200– 207. [find](https://scholar.google.com/scholar?q=Ellen+M+Voorhees+and+Dawn+M+Tice%2E+2000%2E+Building+a+question+answering+test+collection%2E+In+Association+for+Computing+Machinery+Special+Interest+Group+in+Information+Retrieval+%28ACM+SIGIR%29%2C+pages+200%E2%80%93+20)

*Updated 24 October 2026*