---
title: "In-Context Learning Creates Task Vectors — Hendel 2023"
description: "The paper asks how large language models carry out in-context learning (ICL) internally, and whether the process fits the classic learning-theory view in…"
authors: ["Roee Hendel", "Mor Geva", "Amir Globerson"]
year: 2023
venue: "Findings of the Association for Computational Linguistics: EMNLP 2023"
source: "https://aclanthology.org/2023.findings-emnlp.624/"
doi: "10.18653/v1/2023.findings-emnlp.624"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hendel-2023-context-learning-creates-task-vectors/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hendel-2023-context-learning-creates-task-vectors/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) / In-Context Learning Creates Task Vectors

# In-Context Learning Creates Task Vectors

**Authors:** Roee Hendel, [Mor Geva](https://md.research.elamur.ai/free/en/authors/mor-geva/), [Amir Globerson](https://md.research.elamur.ai/free/en/authors/amir-globerson/) · **Year:** 2023 · **Venue:** Findings of the Association for Computational Linguistics: EMNLP 2023 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.findings-emnlp.624/) · [PDF](https://aclanthology.org/2023.findings-emnlp.624.pdf) · [DOI 10.18653/v1/2023.findings-emnlp.624](https://doi.org/10.18653/v1/2023.findings-emnlp.624) · [OpenAlex](https://openalex.org/W4389518997)

**Citations:** 42 citations (openalex-snapshot · 2026-10-14) · [100](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper asks how large language models carry out in-context learning (ICL) internally, and whether the process fits the classic learning-theory view in which training data selects one function from a hypothesis class. The authors propose a two-stage decomposition. First, a "learning" stage turns the demonstrations S into a single query-independent activation, the task vector θ, read at the "→" token at some intermediate layer L. Second, an "application" stage runs the model on the query alone, with θ patched in at that layer. To keep θ independent of the real query, it is computed using a dummy query. They test this on 18 single-token tasks spanning algorithmic, translation, linguistic and factual-knowledge categories, using seven open models. The patched procedure recovers most of ordinary ICL accuracy, and the task vectors turn out to be stable and semantically interpretable. This suggests ICL effectively compresses the demonstrations into one vector that steers the transformer, which may help with efficient task adaptation.

## Key points
- Models tested: LLaMA 7B, 13B and 30B; GPT-J 6B; and Pythia 2.8B, 6.9B and 12B. Code is available at github.com/roeehendel/icl_task_vectors.
- The best split layer L falls at a similar intermediate depth in every model, regardless of model size or layer count.
- The task-vector procedure keeps about 80–90% of regular ICL accuracy. Running the query without any demonstrations reaches only about 10–20%.
- With LLaMA 7B, 50 task vectors per task were built from different S and dummy queries. In a t-SNE plot they form a distinct cluster for each task. When a vector for task B is injected into a prompt demonstrating task A, the model mostly follows task B.
- Projecting θ onto the vocabulary surfaces tokens that describe the task, even though these tokens never appear in the prompt. For example, French→English yields "English" and "translate". The authors note several limitations: the tasks are simple, outputs are single tokens, and the paper does not explain how θ is formed or how the model uses it.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

100 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 98 not yet.

- [Many-Shot In-Context Learning](https://md.research.elamur.ai/free/en/papers/agarwal-2024-many-shot-context-learning/)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://md.research.elamur.ai/free/en/papers/park-2023-linear-representation-hypothesis-geometry-large/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{hendel2023context,
  title = {{In-Context Learning Creates Task Vectors}},
  author = {Hendel, Roee and Geva, Mor and Globerson, Amir},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.624},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.624}
}
```

- [← previous In-context Examples Selection for Machine Translation](https://md.research.elamur.ai/free/en/papers/agrawal-2023-context-examples-selection-machine-translation/)
- [next → IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://md.research.elamur.ai/free/en/papers/gala-2023-indictrans2-towards-high-quality-accessible/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/)

## References

15 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35. [find](https://scholar.google.com/scholar?q=Pengfei+Liu%2C+Weizhe+Yuan%2C+Jinlan+Fu%2C+Zhengbao+Jiang%2C+Hiroaki+Hayashi%2C+and+Graham+Neubig%2E+2023%2E+Pre%2Dtrain%2C+prompt%2C+and+predict%3A+A+systematic+survey+of+prompting+methods+in+natural+language+processing%2E+) [our card](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/)
2. Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2022. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661. [find](https://scholar.google.com/scholar?q=Ekin+Aky%C3%BCrek%2C+Dale+Schuurmans%2C+Jacob+Andreas%2C+Tengyu+Ma%2C+and+Denny+Zhou%2E+2022%2E+What+learning+algorithm+is+in%2Dcontext+learning%3F+investigations+with+linear+models%2E+arXiv+preprint+arXiv%3A2211%2E15661%2E) [our card](https://md.research.elamur.ai/free/en/papers/akyurek-2022-learning-algorithm-context-learning-investigations/)
3. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+2022%2E+Locating+and+editing+factual+associations+in+gpt%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A17359%E2%80%9317372%2E)
4. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373. [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+et+al%2E+2023%2E+Pythia%3A+A+suite)
5. Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2023. Language models implement simple word2vec-style vector arithmetic. arXiv preprint arXiv:2305.16130. [find](https://scholar.google.com/scholar?q=Jack+Merullo%2C+Carsten+Eickhoff%2C+and+Ellie+Pavlick%2E+2023%2E+Language+models+implement+simple+word2vec%2Dstyle+vector+arithmetic%2E+arXiv+preprint+arXiv%3A2305%2E16130%2E)
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+2020%2E+Language+models+are+few%2Dshot+lear)
7. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. 2022. In-context learning and induction heads. arXiv preprint arXiv:2209.11895. [find](https://scholar.google.com/scholar?q=Catherine+Olsson%2C+Nelson+Elhage%2C+Neel+Nanda%2C+Nicholas+Joseph%2C+Nova+DasSarma%2C+Tom+Henighan%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+et+al%2E+2022%2E+In%2Dcontext+learning+and+induction+heads%2E+arXiv+p)
8. Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. 2022. Data distributional properties drive emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 35:18878–18891. [find](https://scholar.google.com/scholar?q=Stephanie+Chan%2C+Adam+Santoro%2C+Andrew+Lampinen%2C+Jane+Wang%2C+Aaditya+Singh%2C+Pierre+Richemond%2C+James+McClelland%2C+and+Felix+Hill%2E+2022%2E+Data+distributional+properties+drive+emergent+in%2Dcontext+learning+in+)
9. Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2022. Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212.10559. [find](https://scholar.google.com/scholar?q=Damai+Dai%2C+Yutao+Sun%2C+Li+Dong%2C+Yaru+Hao%2C+Zhifang+Sui%2C+and+Furu+Wei%2E+2022%2E+Why+can+gpt+learn+in%2Dcontext%3F+language+models+secretly+perform+gradient+descent+as+meta+optimizers%2E+arXiv+preprint+arXiv%3A2212%2E) [our card](https://md.research.elamur.ai/free/en/papers/dai-2023-why-can-gpt-learn-context/)
10. Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2022. Analyzing transformers in embedding space. arXiv preprint arXiv:2209.02535. nostalgebraist. 2020. interpreting gpt: the logit lens. LessWrong. [find](https://scholar.google.com/scholar?q=Guy+Dar%2C+Mor+Geva%2C+Ankit+Gupta%2C+and+Jonathan+Berant%2E+2022%2E+Analyzing+transformers+in+embedding+space%2E+arXiv+preprint+arXiv%3A2209%2E02535%2E+nostalgebraist%2E+2020%2E+interpreting+gpt%3A+the+logit+lens%2E+LessWrong) [our card](https://md.research.elamur.ai/free/en/papers/dar-2023-analyzing-transformers-embedding-space/)
11. Shai Shalev-Shwartz and Shai Ben-David. 2014. Understanding machine learning: From theory to algorithms. Cambridge university press. [find](https://scholar.google.com/scholar?q=Shai+Shalev%2DShwartz+and+Shai+Ben%2DDavid%2E+2014%2E+Understanding+machine+learning%3A+From+theory+to+algorithms%2E+Cambridge+university+press%2E)
12. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timoth%C3%A9e+Lacroix%2C+Baptiste+Rozi%C3%A8re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+et+al%2E+2023%2E+Llama%3A+Open+and+efficient+f)
13. Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. 2022. Transformers learn in-context by gradient descent. arXiv preprint arXiv:2212.07677. [find](https://scholar.google.com/scholar?q=Johannes+von+Oswald%2C+Eyvind+Niklasson%2C+Ettore+Randazzo%2C+Jo%C3%A3o+Sacramento%2C+Alexander+Mordvintsev%2C+Andrey+Zhmoginov%2C+and+Max+Vladymyrov%2E+2022%2E+Transformers+learn+in%2Dcontext+by+gradient+descent%2E+arXiv+pre) [our card](https://md.research.elamur.ai/free/en/papers/oswald-2022-transformers-learn-context-gradient-descent/)
14. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingoflolz/](https://github.com/kingoflolz/) mesh-transformer-jax. [↗](https://github.com/kingoflolz/)
15. Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Aditi+Raghunathan%2C+Percy+Liang%2C+and+Tengyu+Ma%2E+An+explanation+of+in%2Dcontext+learning+as+implicit+bayesian+inference%2E+In+International+Conference+on+Learning+Representations%2E) [our card](https://md.research.elamur.ai/free/en/papers/xie-2021-explanation-context-learning-implicit-bayesian/)