---
title: "Soft Head Selection for Injecting ICL-Derived Task Embeddings"
description: "The authors propose SITE, which works in three steps: It averages last-token attention-head outputs over several few-shot… 68 references."
authors: ["Jungwon Park", "Jimyeong Kim", "Changin Choi", "Wonjong Rhee"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1355/"
doi: "10.18653/v1/2026.findings-acl.1355"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/park-2026-soft-head-selection-injecting-icl/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/park-2026-soft-head-selection-injecting-icl/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/43/) / Soft Head Selection for Injecting ICL-Derived Task Embeddings

# Soft Head Selection for Injecting ICL-Derived Task Embeddings

**Authors:** Jungwon Park, Jimyeong Kim, Changin Choi and Wonjong Rhee · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1355/) · [PDF](https://aclanthology.org/2026.findings-acl.1355.pdf) · [DOI 10.18653/v1/2026.findings-acl.1355](https://doi.org/10.18653/v1/2026.findings-acl.1355) · [OpenAlex](https://openalex.org/W4417271153)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary

The paper addresses a weakness of ICL-driven embedding-based adaptation. In this approach, task information is taken from a model's internal activations under few-shot prompts and later re-inserted so the model can solve the task from a zero-shot prompt. Earlier methods (FV, TV, MTV, I2CL) have rarely beaten plain few-shot ICL and are seldom compared against PEFT. The authors propose SITE, which works in three steps:

1. It averages last-token attention-head outputs over several few-shot prompts to build per-head task embeddings.
2. It learns one soft selection weight in [0, 1] per head. Each weight blends the head's original activation with the stored embedding. The LLM stays frozen, and the weights are trained by gradient descent on next-token cross-entropy for a few hundred steps.
3. At inference it reuses both the embeddings and the weights.

Across open-ended generation, reasoning and NLU tasks on four benchmarks and 12 LLMs, SITE outperforms earlier embedding-based methods and few-shot ICL. It comes close to PEFT quality while training far fewer parameters. Patching experiments indicate that important heads depend on the task, and that similar tasks rely on overlapping heads.

## Key points

- **Models:** 12 LLMs from 4 model families and 3 variants, ranging from 4B to 70B parameters. Code is published at github.com/SNU-DRL/Soft_Injection.
- **Parameters and storage:** only L × H scalars are trained, which is 1024 for Llama-3.1-8B. Storing these weights plus the task embeddings takes roughly 0.5 MB in float32.
- **Runtime:** on AG_News with Llama-3.1-8B (single A6000), 10,000 test prompts took 172.0 min. This compares with 248.4 min for 10-shot ICL, 156.7 min for zero-shot, 411.2 min for FV, 328.8 min for MTV and 178.7 min for I2CL.
- **Many-shot comparison:** with 100 shots, ICL catches up with SITE on AG_News and Conll2003_Organization. It does not catch up on Choose_Middle_Of_5 or Alphabetically_Last_5.
- **Limitations:**
  - SITE needs some labeled data, though 30–50 examples usually suffice.
  - It requires access to internal activations, so it works only with open-weight models.
  - It was not tested on multi-step reasoning or long-context generation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Seoul National University](https://md.research.elamur.ai/free/en/universities/seoul-national-university/) — University, Seoul, South Korea · 41 works · 45,671 citations
- [Samsung (South Korea)](https://md.research.elamur.ai/free/en/companies/samsung-south-korea/) — Company, Seoul, South Korea · 7 works · 1,214 citations
- [Daegu Gyeongbuk Institute of Science and Technology](https://md.research.elamur.ai/free/en/universities/daegu-gyeongbuk-institute-of-science-and-technology/) — University, Daegu, South Korea · 2 works

Also on the byline: [Samsung Electronics](https://ror.org/020m7t761).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Jungwon Park, Jimyeong Kim, Changin Choi and Wonjong Rhee. 2026. Soft Head Selection for Injecting ICL-Derived Task Embeddings. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{park2026soft,
  title = {{Soft Head Selection for Injecting ICL-Derived Task Embeddings}},
  author = {Jungwon Park, Jimyeong Kim, Changin Choi and Wonjong Rhee},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1355},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1355}
}
```

- [← previous Social-RAG: A Retrieval-Augmented Generation Pipeline for Computational Social Science Research on Telegram](https://md.research.elamur.ai/free/en/papers/nascimento-2026-social-rag-retrieval-augmented-generation/)
- [next → Soft Orthogonal Low-Rank Adaptation for Knowledge Sharing in Large Language Model Continual Learning](https://md.research.elamur.ai/free/en/papers/wang-2026-soft-orthogonal-low-rank-adaptation/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/43/)

## References

68 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rishabh Agarwal, Avi Singh, Lei Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, and 1 others. 2024. [Many-shot in-context learning](https://md.research.elamur.ai/free/en/papers/agarwal-2024-many-shot-context-learning/). Advances in Neural Information Processing Systems, 37:76930– 76966.
2. Leonard Bereska and Efstratios Gavves. 2024. Mechanistic interpretability for ai safety–a review. arXiv preprint arXiv:2404.14082. [find](https://scholar.google.com/scholar?q=Leonard+Bereska+and+Efstratios+Gavves%2E+2024%2E+Mechanistic+interpretability+for+ai+safety%E2%80%93a+review%2E+arXiv+preprint+arXiv%3A2404%2E14082%2E)
3. Amanda Bertsch, Maor Ivgi, Emily Xiao, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. 2025. [In-context learning with long-context models: An in-depth exploration](https://md.research.elamur.ai/free/en/papers/bertsch-2025-context-learning-long-context-models/). In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lingu…
4. Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/). In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 632– 642.
5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
6. Wang Cai, Hsiu-Yuan Huang, Zhixiang Wang, and Yunfang Wu. 2025. Beyond demonstrations: Dynamic vector construction from latent representations. arXiv preprint arXiv:2505.20318. [find](https://scholar.google.com/scholar?q=Wang+Cai%2C+Hsiu%2DYuan+Huang%2C+Zhixiang+Wang%2C+and+Yunfang+Wu%2E+2025%2E+Beyond+demonstrations%3A+Dynamic+vector+construction+from+latent+representations%2E+arXiv+preprint+arXiv%3A2505%2E20318%2E)
7. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. [Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/). arX…
8. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. arXiv preprint arXiv:1710.04087. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Guillaume+Lample%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Ludovic+Denoyer%2C+and+Herv%C3%A9+J%C3%A9gou%2E+2017%2E+Word+translation+without+parallel+data%2E+arXiv+preprint+arXiv%3A1710%2E04087%2E)
9. Amit Elhelo and Mor Geva. 2024. Inferring functionality of attention heads from their parameters. arXiv preprint arXiv:2412.11965. [find](https://scholar.google.com/scholar?q=Amit+Elhelo+and+Mor+Geva%2E+2024%2E+Inferring+functionality+of+attention+heads+from+their+parameters%2E+arXiv+preprint+arXiv%3A2412%2E11965%2E)
10. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
11. Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2021. Self-attention attribution: Interpreting information interactions inside transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12963–12971. [find](https://scholar.google.com/scholar?q=Yaru+Hao%2C+Li+Dong%2C+Furu+Wei%2C+and+Ke+Xu%2E+2021%2E+Self%2Dattention+attribution%3A+Interpreting+information+interactions+inside+transformer%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+vo)
12. Roee Hendel, Mor Geva, and Amir Globerson. 2023. [In-context learning creates task vectors](https://md.research.elamur.ai/free/en/papers/hendel-2023-context-learning-creates-task-vectors/). arXiv preprint arXiv:2310.15916.
13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300.
14. Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. 2023. Linearity of relation decoding in transformer language models. arXiv preprint arXiv:2308.09124. [find](https://scholar.google.com/scholar?q=Evan+Hernandez%2C+Arnab+Sen+Sharma%2C+Tal+Haklay%2C+Kevin+Meng%2C+Martin+Wattenberg%2C+Jacob+Andreas%2C+Yonatan+Belinkov%2C+and+David+Bau%2E+2023%2E+Linearity+of+relation+decoding+in+transformer+language+models%2E+arXiv+)
15. Or Honovich, Uri Shaham, Samuel R Bowman, and Omer Levy. 2022. Instruction induction: From few examples to natural language task descriptions. arXiv preprint arXiv:2205.10782. [find](https://scholar.google.com/scholar?q=Or+Honovich%2C+Uri+Shaham%2C+Samuel+R+Bowman%2C+and+Omer+Levy%2E+2022%2E+Instruction+induction%3A+From+few+examples+to+natural+language+task+descriptions%2E+arXiv+preprint+arXiv%3A2205%2E10782%2E)
16. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+nlp%2E+In+)
17. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). ICLR, 1(2):3.
18. Brandon Huang, Chancharik Mitra, Leonid Karlinsky, Assaf Arbelle, Trevor Darrell, and Roei Herzig. 2024. Multimodal task vectors enable many-shot multimodal in-context learning. Advances in Neural Information Processing Systems, 37:22124–22153. [find](https://scholar.google.com/scholar?q=Brandon+Huang%2C+Chancharik+Mitra%2C+Leonid+Karlinsky%2C+Assaf+Arbelle%2C+Trevor+Darrell%2C+and+Roei+Herzig%2E+2024%2E+Multimodal+task+vectors+enable+many%2Dshot+multimodal+in%2Dcontext+learning%2E+Advances+in+Neural+Inf)
19. Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. 2023. Interpreting clip’s image representation via text-based decomposition. arXiv preprint arXiv:2310.05916. [find](https://scholar.google.com/scholar?q=Yossi+Gandelsman%2C+Alexei+A+Efros%2C+and+Jacob+Steinhardt%2E+2023%2E+Interpreting+clip%E2%80%99s+image+representation+via+text%2Dbased+decomposition%2E+arXiv+preprint+arXiv%3A2310%2E05916%2E)
20. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
21. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. [Making the v in vqa matter: Elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913.
22. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, and 1 others. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+and+1+others%2E+2024%2E+Mi)
23. Joonseong Kang, Soojeong Lee, Subeen Park, Sumin Park, Taero Kim, Jihee Kim, Ryunyi Lee, and Kyungwoo Song. 2025. Adaptive task vectors for large language models. arXiv preprint arXiv:2506.03426. — [arXiv:2506.03426](https://arxiv.org/abs/2506.03426) [find](https://scholar.google.com/scholar?q=Joonseong+Kang%2C+Soojeong+Lee%2C+Subeen+Park%2C+Sumin+Park%2C+Taero+Kim%2C+Jihee+Kim%2C+Ryunyi+Lee%2C+and+Kyungwoo+Song%2E+2025%2E+Adaptive+task+vectors+for+large+language+models%2E+arXiv+preprint+arXiv%3A2506%2E03426%2E+%E2%80%94+%5Ba)
24. Diederik P Kingma. 2014. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980.
25. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. [Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p…
26. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). arXiv preprint arXiv:2104.08691.
27. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+2022%2E+Locating+and+editing+factual+associations+in+gpt%2E+Advances+in+neural+information+processing+systems%2C+35%3A17359%E2%80%9317372%2E)
28. Jiaqian Li, Yanshu Li, Ligong Han, Ruixiang Tang, and Wenya Wang. 2025a. Towards generalizable implicit in-context learning with attention routing. arXiv preprint arXiv:2509.22854. — [arXiv:2509.22854](https://arxiv.org/abs/2509.22854) [find](https://scholar.google.com/scholar?q=Jiaqian+Li%2C+Yanshu+Li%2C+Ligong+Han%2C+Ruixiang+Tang%2C+and+Wenya+Wang%2E+2025a%2E+Towards+generalizable+implicit+in%2Dcontext+learning+with+attention+routing%2E+arXiv+preprint+arXiv%3A2509%2E22854%2E+%E2%80%94+%5BarXiv%3A2509%2E22854)
29. Mihai Nadas, Laura Diosan, and Andreea Tomescu. 2025. Synthetic data generation using large language models: Advances in text and code. arXiv preprint arXiv:2503.14023. — [arXiv:2503.14023](https://arxiv.org/abs/2503.14023) [find](https://scholar.google.com/scholar?q=Mihai+Nadas%2C+Laura+Diosan%2C+and+Andreea+Tomescu%2E+2025%2E+Synthetic+data+generation+using+large+language+models%3A+Advances+in+text+and+code%2E+arXiv+preprint+arXiv%3A2503%2E14023%2E+%E2%80%94+%5BarXiv%3A2503%2E14023%5D%28https%3A%2F%2Far)
30. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. [Inference-time intervention: Eliciting truthful answers from a language model](https://md.research.elamur.ai/free/en/papers/li-2023-inference-time-intervention-eliciting-truthful/). Advances in Neural Information Processing Systems, 36:41451–41530.
31. Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. 2017. Distinguishing antonyms and synonyms in a pattern-based neural network. arXiv preprint arXiv:1701.02962. [find](https://scholar.google.com/scholar?q=Kim+Anh+Nguyen%2C+Sabine+Schulte+im+Walde%2C+and+Ngoc+Thang+Vu%2E+2017%2E+Distinguishing+antonyms+and+synonyms+in+a+pattern%2Dbased+neural+network%2E+arXiv+preprint+arXiv%3A1701%2E02962%2E)
32. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190. [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+arXiv+preprint+arXiv%3A2101%2E00190%2E)
33. Yanshu Li, Yi Cao, Hongyang He, Qisen Cheng, Xiang Fu, Xi Xiao, Tianyang Wang, and Ruixiang Tang. 2025b. M 2 iv: Towards efficient and fine-grained multimodal in-context learning via representation engineering. In Second Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Yanshu+Li%2C+Yi+Cao%2C+Hongyang+He%2C+Qisen+Cheng%2C+Xiang+Fu%2C+Xi+Xiao%2C+Tianyang+Wang%2C+and+Ruixiang+Tang%2E+2025b%2E+M+2+iv%3A+Towards+efficient+and+fine%2Dgrained+multimodal+in%2Dcontext+learning+via+representation+en)
34. Zhuowei Li, Zihao Xu, Ligong Han, Yunhe Gao, Song Wen, Di Liu, Hao Wang, and Dimitris N Metaxas. 2024. Implicit in-context learning. arXiv preprint arXiv:2405.14660. — [arXiv:2405.14660](https://arxiv.org/abs/2405.14660) [find](https://scholar.google.com/scholar?q=Zhuowei+Li%2C+Zihao+Xu%2C+Ligong+Han%2C+Yunhe+Gao%2C+Song+Wen%2C+Di+Liu%2C+Hao+Wang%2C+and+Dimitris+N+Metaxas%2E+2024%2E+Implicit+in%2Dcontext+learning%2E+arXiv+preprint+arXiv%3A2405%2E14660%2E+%E2%80%94+%5BarXiv%3A2405%2E14660%5D%28https%3A%2F%2Farxiv)
35. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022a. [Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/). Advances in Neural Information Processing Systems, 35:1950–1965.
36. Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. 2022b. [What makes good in-context examples for gpt-3?](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/) In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd workshop on knowledge extraction and integration for deep learning architectures, pages 1…
37. Yiting Liu and Zhi-Hong Deng. 2025. Iterative vectors: In-context gradient steering without backpropagation. In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Yiting+Liu+and+Zhi%2DHong+Deng%2E+2025%2E+Iterative+vectors%3A+In%2Dcontext+gradient+steering+without+backpropagation%2E+In+Forty%2Dsecond+International+Conference+on+Machine+Learning%2E)
38. Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. 2024. [On llms-driven synthetic data generation, curation, and evaluation: A survey](https://md.research.elamur.ai/free/en/papers/long-2024-llms-driven-synthetic-data-generation/). arXiv preprint arXiv:2406.15126.
39. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. [Adversarial nli: A new benchmark for natural language understanding](https://md.research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/). In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 4885–4901.
40. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, and 1 others. 2022. [In-context learning and induction heads](https://md.research.elamur.ai/free/en/papers/olsson-2022-context-learning-induction-heads/). arXiv preprint arXiv:2209.11895.
41. OpenAI. 2025. Introducing gpt-5. [https://openai.com/index/introducing-gpt-5/](https://openai.com/index/introducing-gpt-5/). Accessed: 2025-09-11. [↗](https://openai.com/index/introducing-gpt-5/)
42. Jungwon Park, Jungmin Ko, Dongnam Byun, Jangwon Suh, and Wonjong Rhee. 2024. Cross-attention head position patterns can align with human visual concepts in text-to-image generative models. arXiv preprint arXiv:2412.02237. — [arXiv:2412.02237](https://arxiv.org/abs/2412.02237) [find](https://scholar.google.com/scholar?q=Jungwon+Park%2C+Jungmin+Ko%2C+Dongnam+Byun%2C+Jangwon+Suh%2C+and+Wonjong+Rhee%2E+2024%2E+Cross%2Dattention+head+position+patterns+can+align+with+human+visual+concepts+in+text%2Dto%2Dimage+generative+models%2E+arXiv+prepr)
43. Yingzhe Peng, Xinting Hu, Jiawei Peng, Xin Geng, Xu Yang, and 1 others. 2024. Live: Learnable in-context vector for visual question answering. Advances in Neural Information Processing Systems, 37:9773–9800. [find](https://scholar.google.com/scholar?q=Yingzhe+Peng%2C+Xinting+Hu%2C+Jiawei+Peng%2C+Xin+Geng%2C+Xu+Yang%2C+and+1+others%2E+2024%2E+Live%3A+Learnable+in%2Dcontext+vector+for+visual+question+answering%2E+Advances+in+Neural+Information+Processing+Systems%2C+37%3A977)
44. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5:606–624. [find](https://scholar.google.com/scholar?q=Reiner+Pope%2C+Sholto+Douglas%2C+Aakanksha+Chowdhery%2C+Jacob+Devlin%2C+James+Bradbury%2C+Jonathan+Heek%2C+Kefan+Xiao%2C+Shivani+Agrawal%2C+and+Jeff+Dean%2E+2023%2E+Efficiently+scaling+transformer+inference%2E+Proceedings+)
45. Baturay Saglam, Xinyang Hu, Zhuoran Yang, Dionysis Kalogerias, and Amin Karbasi. 2025. Learning task representations from in-context learning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 6634–6663. [find](https://scholar.google.com/scholar?q=Baturay+Saglam%2C+Xinyang+Hu%2C+Zhuoran+Yang%2C+Dionysis+Kalogerias%2C+and+Amin+Karbasi%2E+2025%2E+Learning+task+representations+from+in%2Dcontext+learning%2E+In+Findings+of+the+Association+for+Computational+Linguist)
46. Erik F Sang and Fien De Meulder. 2003. [Introduction to the conll-2003 shared task: Language-independent named entity recognition](https://md.research.elamur.ai/free/en/papers/sang-2003-introduction-conll-2003-shared-task/). arXiv preprint cs/0306050.
47. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. [Recursive deep models for semantic compositionality over a sentiment treebank](https://md.research.elamur.ai/free/en/papers/socher-2013-recursive-deep-models-semantic-compositionality/). In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631…
48. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and 1 others. 2023. [Challenging big-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). In Findings of the Association for Computational Linguistics: AC…
49. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. [Commonsenseqa: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). arXiv preprint arXiv:1811.00937.
50. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, and 1 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Aishwarya+Kamath%2C+Johan+Ferret%2C+Shreya+Pathak%2C+Nino+Vieillard%2C+Ramona+Merhej%2C+Sarah+Perrin%2C+Tatiana+Matejovicova%2C+Alexandre+Ram%C3%A9%2C+Morgane+Rivi%C3%A8re%2C+and+1+others%2E+2025%2E+Gemma+3+technical+rep)
51. Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. 2023. Function vectors in large language models. arXiv preprint arXiv:2310.15213. [find](https://scholar.google.com/scholar?q=Eric+Todd%2C+Millicent+L+Li%2C+Arnab+Sen+Sharma%2C+Aaron+Mueller%2C+Byron+C+Wallace%2C+and+David+Bau%2E+2023%2E+Function+vectors+in+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E15213%2E)
52. Futing Wang, Jianhao Yan, Yue Zhang, and Tao Lin. 2024a. Elicit: Llm augmentation via external in-context capability. arXiv preprint arXiv:2410.09343. — [arXiv:2410.09343](https://arxiv.org/abs/2410.09343) [find](https://scholar.google.com/scholar?q=Futing+Wang%2C+Jianhao+Yan%2C+Yue+Zhang%2C+and+Tao+Lin%2E+2024a%2E+Elicit%3A+Llm+augmentation+via+external+in%2Dcontext+capability%2E+arXiv+preprint+arXiv%3A2410%2E09343%2E+%E2%80%94+%5BarXiv%3A2410%2E09343%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2410%2E0)
53. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024b. [Mmlu-pro: A more robust and challenging multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/). Advances in Neural Information Processing Systems, 37:95266–9529…
54. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long pa…
55. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229–256. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams%2E+1992%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+learning%2C+8%283%29%3A229%E2%80%93256%2E)
56. Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. 2024. [Retrieval head mechanistically explains long-context factuality](https://md.research.elamur.ai/free/en/papers/wu-2024-retrieval-head-mechanistically-explains-long/). arXiv preprint arXiv:2404.15574.
57. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
58. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2023. [Large language models as optimizers](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-optimizers/). In The Twelfth International Conference on Learning Representations.
59. Haolin Yang, Hakaze Cho, Kaize Ding, and Naoya Inoue. 2025b. Task vectors, learned not extracted: Performance gains and mechanistic insight. arXiv preprint arXiv:2509.24169. — [arXiv:2509.24169](https://arxiv.org/abs/2509.24169) [find](https://scholar.google.com/scholar?q=Haolin+Yang%2C+Hakaze+Cho%2C+Kaize+Ding%2C+and+Naoya+Inoue%2E+2025b%2E+Task+vectors%2C+learned+not+extracted%3A+Performance+gains+and+mechanistic+insight%2E+arXiv+preprint+arXiv%3A2509%2E24169%2E+%E2%80%94+%5BarXiv%3A2509%2E24169%5D%28https)
60. Asaf Yehudai, Boaz Carmeli, Yosi Mass, Ofir Arviv, Nathaniel Mills, Assaf Toledo, Eyal Shnarch, and Leshem Choshen. 2024. Genie: Achieving human parity in content-grounded datasets generation. arXiv preprint arXiv:2401.14367. — [arXiv:2401.14367](https://arxiv.org/abs/2401.14367) [find](https://scholar.google.com/scholar?q=Asaf+Yehudai%2C+Boaz+Carmeli%2C+Yosi+Mass%2C+Ofir+Arviv%2C+Nathaniel+Mills%2C+Assaf+Toledo%2C+Eyal+Shnarch%2C+and+Leshem+Choshen%2E+2024%2E+Genie%3A+Achieving+human+parity+in+content%2Dgrounded+datasets+generation%2E+arXiv+p)
61. Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. [Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models](https://md.research.elamur.ai/free/en/papers/ben-zaken-2021-bitfit-simple-parameter-efficient-fine/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–9.
62. Fred Zhang and Neel Nanda. 2023. Towards best practices of activation patching in language models: Metrics and methods. arXiv preprint arXiv:2309.16042. [find](https://scholar.google.com/scholar?q=Fred+Zhang+and+Neel+Nanda%2E+2023%2E+Towards+best+practices+of+activation+patching+in+language+models%3A+Metrics+and+methods%2E+arXiv+preprint+arXiv%3A2309%2E16042%2E)
63. Kaiyi Zhang, Ang Lv, Yuhan Chen, Hansen Ha, Tao Xu, and Rui Yan. 2024. Batch-icl: Effective, efficient, and order-agnostic in-context learning. arXiv preprint arXiv:2401.06469. [find](https://scholar.google.com/scholar?q=Kaiyi+Zhang%2C+Ang+Lv%2C+Yuhan+Chen%2C+Hansen+Ha%2C+Tao+Xu%2C+and+Rui+Yan%2E+2024%2E+Batch%2Dicl%3A+Effective%2C+efficient%2C+and+order%2Dagnostic+in%2Dcontext+learning%2E+arXiv+preprint+arXiv%3A2401%2E06469%2E)
64. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. [Character-level convolutional networks for text classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/). Advances in neural information processing systems, 28.
65. Xiaoqing Zhang, Ang Lv, Yuhan Liu, Flood Sung, Wei Liu, Jian Luan, Shuo Shang, Xiuying Chen, and Rui Yan. 2025. More is not always better? enhancing many-shot in-context learning with differentiated and reweighting objectives. arXiv preprint arXiv:2501.04070. [find](https://scholar.google.com/scholar?q=Xiaoqing+Zhang%2C+Ang+Lv%2C+Yuhan+Liu%2C+Flood+Sung%2C+Wei+Liu%2C+Jian+Luan%2C+Shuo+Shang%2C+Xiuying+Chen%2C+and+Rui+Yan%2E+2025%2E+More+is+not+always+better%3F+enhancing+many%2Dshot+in%2Dcontext+learning+with+differentiated+a)
66. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. [Large language models are human-level prompt engineers](https://md.research.elamur.ai/free/en/papers/zhou-2022-large-language-models-human-level/). In The eleventh international conference on learning representations.
67. Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, Kun Wang, Yang Liu, Junfeng Fang, and Yongbin Li. 2024. On the role of attention heads in large language model safety. arXiv preprint arXiv:2410.13708. — [arXiv:2410.13708](https://arxiv.org/abs/2410.13708) [find](https://scholar.google.com/scholar?q=Zhenhong+Zhou%2C+Haiyang+Yu%2C+Xinghua+Zhang%2C+Rongwu+Xu%2C+Fei+Huang%2C+Kun+Wang%2C+Yang+Liu%2C+Junfeng+Fang%2C+and+Yongbin+Li%2E+2024%2E+On+the+role+of+attention+heads+in+large+language+model+safety%2E+arXiv+preprint+ar)
68. Kaijian Zou, Muhammad Khalifa, and Lu Wang. 2024. Retrieval or global context understanding? [on many-shot in-context learning for long-context evaluation](https://md.research.elamur.ai/free/en/papers/zou-2025-many-shot-context-learning-long/). arXiv preprint arXiv:2411.07130.

*Updated 1 November 2026*